Leveraging AI to Accelerate Experience Mapping
Part 5 of the AI vs. Human: A User Research Showdown series
Summary
There are many new platforms claiming to accelerate or even replace your typical user research with the use of AI. We put these claims to the test in a head-to-head study comparing the effectiveness of AI tools and human UX researchers throughout the generative research process. As part of this test, both human researchers and AI tools (and combinations of the two) created experience maps to help a fictional client understand and empathize with their target audience’s current state experience planning family trips.
We found that AI sped up the process of experience mapping by defining the phases and attributes for the maps, but that humans were ultimately needed to generate nuanced, detailed insights based on study data. Once this information was in place, however, AI tools were useful for brainstorming business opportunities to address customer needs and pain points.
Study Recap
In the rapidly evolving world of user experience research, the emergence of novel artificial intelligence tools presents a new frontier. As we stand on the cusp of potentially transformative changes, the question arises: How might AI impact traditionally human-led endeavors like generative UX research?
At Brilliant Experience, we performed an empirical study to directly compare the effectiveness of AI models and human researchers in conducting qualitative interviews. Our study centered on parents of young children planning international travel, examined under four distinct conditions, each varying in AI and human involvement. Across these scenarios, the researchers (or AI tools) produced standard deliverables: a slide deck report of key insights, a set of personas, and corresponding experience maps for each persona. We evaluated each approach not only for output quality but also for efficiency and depth of understanding.
Read more about our method and goals in our Executive Summary.
This Edition: Experience Mapping
In this post, we’ll continue to explore our findings from the synthesis phase, focusing this time on our experience creating experience maps in each condition.
Experience maps are visual representations that capture the broader context of a user’s interaction with a type of product or service and help teams uncover nuanced insights and opportunities for improvement. Unlike journey maps, which focus on the flow of interacting with a specific product or service, experience maps aim to provide a holistic view of the entire ecosystem in which the experience occurs.
In our study, we mapped the experience of planning international family travel (as opposed to mapping the journey of, say, booking a hotel through Expedia).
It is important to mention that similar to our experience with personas, we did not find an AI tool suitable for our needs for the visual design aspect of experience map creation (some tools and Figma plug-ins exist but tended to have simplistic designs that weren’t sufficiently customizable). Therefore, a human researcher developed the experience map template used across all conditions. Our comparison focuses specifically on the content within the maps.
Choosing an AI Tool to Create Experience Maps
There are three main types of tools you can use to support your experience mapping: general AI tools like ChatGPT, specialized research synthesis tools designed specifically for analyzing qualitative research data, and ideation tools with features meant specifically for creating journey or experience maps.
Using General AI Tools for Experience Map Creation
General AI tools like ChatGPT can be used to generate experience maps in a similar way as they might be used to create personas: you can ask these tools to identify an appropriate structure and build out the content for experience maps given a set of transcripts or interview notes.
As with all data synthesis tasks, it is essential to ensure that the AI tool you are using maintains data privacy within the confines of the conversation and does not use your data to train its model. For participant privacy reasons, we recommend only sharing research data with AI systems that guarantee data confidentiality.
Using Research Synthesis Tools for Experience Map Creation
Some tools like Research Studio have specific features designed to draft journey maps that can be extended for creating experience maps, while others like Next have custom AI features that can help you work with your data in a number of ways, including generating experience maps from it.
Important features to consider when selecting a research synthesis tool for experience map creation are:
Allowed File Types: Some tools allow both videos and transcripts, while others accept only one or the other.
Functionality: Some tools have one-click journey/experience map creation, while others come with more custom AI functionality. Either might be appropriate depending on your goals.
Pricing: As we mentioned in the Data Synthesis edition, tools specifically for research synthesis can be expensive, and often require an enterprise license to access the full feature set.
Using Ideation Tools for Experience Map Creation
Ideation tools like QoQo and UX Pilot can also generate journey or experience maps, but are best used at the beginning of the research process to develop hypotheses about the user’s experience ahead of actual data collection. They draw on their existing world knowledge to draft a journey given a description of the target audience and the scenario. This output is likely to include many relevant insights, but should be verified and expanded upon with primary data collection whenever possible.
AI vs. Human Comparison: Experience Mapping
Experience maps are common deliverables coming out of a generative interview study, and can be extremely useful for visualizing the context a product interaction may occur in and identifying opportunities to address customer needs.
While AI was a valuable partner during our persona creation process, it fell short for some aspects of experience mapping. As we outline below, we found that humans are needed to produce meaningful and nuanced insights for experience maps, but AI can be helpful for identifying actionable business opportunities based on those insights.
AI Win: Defining Phases and Attributes
Experience maps are generally organized chronologically by phase or stage, with the primary content outlining various attributes corresponding to each phase - for example, user actions, pain points, and opportunities. The specific phases and attributes are dependent on the topic and objectives of the research as well as the findings, so structuring an experience map appropriately typically requires research expertise.
We were pleasantly surprised to discover that AI tools (general AI tools, research synthesis tools, and ideation tools) were useful at helping to define the structure of an experience map given a set of transcripts and description of the research objectives. In our case, both the phases and attributes defined by AI were remarkably similar to those generated by humans.
AI Win: Brainstorming Opportunities
AI shines when faced with creative tasks, particularly those involving brainstorming numerous possibilities. For this reason, we found general AI tools to be valuable partners when generating business opportunities to address the user needs described in our experience maps. We provided the tool with the content from the maps (goals, emotions, thoughts, actions, resources, and pain points) which it then used to come up with relevant opportunities for each phase. While not all AI-generated opportunities are guaranteed to be feasible or successful, the ones we encountered were nearly always relevant to the content at hand and expanded on what researchers alone would have otherwise provided.
Human Win: Drafting Experience Map Content
Unfortunately, AI struggled to draft accurate, insightful content to build out the experience maps. Similar to our experience generating key insights from the data, we found that AI tools drew too much on world knowledge instead of on the research data itself, resulting in generic and irrelevant content. Research-specific AI tools that limited the model’s use of general world knowledge were slightly better at drafting content, but the added input of a human researcher improved AI’s performance (see ratings from independent reviewers below).
Human Win: Adding Emotional Context
One aspect of nuance that AI frequently missed when generating experience map content was the emotional context of the individual’s actions and pain points. General AI tools provided descriptions of the emotions a user might be experiencing at a particular phase, but these were often inconsistent with what we heard in the interviews or too simplistic.
This was in part a function of the type of data the AI tools were provided - video transcripts, which may not always capture subtle emotional content.
AI Honorable Mention: Refining Experience Map Content
General AI tools were, unsurprisingly, also great at refining experience map copy drafted by a human researcher, ensuring that the writing was concise and felt unified throughout.
When to Involve AI in Experience Mapping
Given our findings, our experience mapping process has evolved to incorporate AI in the following ways:
Use a general AI tool to define appropriate phases and attributes for the experience map given a set of study transcripts (or simply a description of the study)
Have trained researchers draft the content for the experience map
Use a general AI tool to refine the experience map copy
Use a general AI tool or AI ideation tool to generate possible opportunities to add to the experience map
Have trained designers or researchers build the visuals
Conclusion
Experience maps are great tools to help product teams better understand the broader customer experience and address unmet needs and pain points. A variety of AI tools can be leveraged to create these deliverables, from general AI tools that can help with data synthesis and copy to design tools that can put the content into a visual format. In our head-to-head competition of these tools and human researchers and found that when it comes to creating useful experience maps:
AI is not well-suited for creating the content for experience maps, given its tendency to provide inaccurate information or insights that are too general
AI tools can still accelerate the process by helping to define the structure for experience maps, including the phases and relevant attributes
AI can be leveraged to identify opportunity areas given human-generated insights
For these reasons, we consider AI a “useful assistant” when it comes to experience mapping. We’ll stick to generating the insights ourselves, but will be leveraging AI whenever possible to speed up the process and help us brainstorm opportunity areas.
Interested to learn more about how AI tools stack up against human researchers? Subscribe to our newsletter to make sure you don’t miss an issue of our AI vs. Human series – or our ongoing AI 4 UX video interview series featuring founders of some of the most popular AI tools for UX research.
