AI-Moderated “Interviews” Are a Fast Track to Real Human Insights
Part 2 of the AI vs. Human: A User Research Showdown series
Summary
There are numerous new platforms claiming to accelerate or even replace your typical user research with the use of AI. We put these claims to the test in a head-to-head study comparing the effectiveness of AI tools and human UX researchers throughout the generative research process. Perhaps one of the most surprising findings from our study was the success we saw replacing traditional (human) moderated qualitative interview sessions with AI-run sessions that blur the line between moderated and unmoderated research. Consequently, we now consider AI a “trusted researcher” when it comes to data collection - and one that can save your human researchers substantial time. That said, we have no plans to replace our invaluable human participants with AI-generated ones anytime soon.
Study Recap
In the rapidly evolving world of user experience research, the emergence of novel artificial intelligence tools presents a new frontier. As we stand on the cusp of potentially transformative changes, the question arises: How might AI impact traditionally human-led endeavors like generative UX research?
At Brilliant Experience, we performed an empirical study to directly compare the effectiveness of AI models and human researchers in conducting qualitative interviews. Our study centered on parents of young children planning international travel, examined under four distinct conditions, each varying in AI and human involvement. Across these scenarios, the researchers (or AI tools) produced standard deliverables: a slide deck report of key insights, a set of personas, and corresponding experience maps for each persona. We evaluated each approach not only for output quality but also for efficiency and depth of understanding.
Read more about our method and goals in our Executive Summary.
This Edition: Data Collection
In this post, we’ll delve into our findings from the data collection phase.
Many AI tools have recently emerged on the market, some promising to replace skilled qualitative interviewers, or even research participants themselves. Are these claims too good to be true? Can AI moderators and participants truly deliver the same quality of insights as their human counterparts? In our study, we put these claims to the test by evaluating AI moderators (AI Moderated condition) and AI participants (AI Only condition). In this post, we’ll share our findings and about how each type of tool stands to impact the future of generative UX research.
Choosing an AI Tool for Data Collection
There are three main types of AI tools that could be used to support researchers in the data collection phase - AI Moderated Interview Tools, Simulated User Tools, and General AI tools. We’ll talk about the capabilities and considerations for each of these in turn.
Using AI Moderated Interview Tools for Data Collection
Tools like Wondering, UserCue, and Listen Labs function as interactive surveys where participants respond to open-ended (and closed-ended, if desired) questions via text or voice. The AI tool “moderates” the session by dynamically asking follow-up questions based on participant responses.
There are several factors to consider when choosing an AI moderated interview tool:
Control Over Questions: Different tools offer different degrees of control over the follow-up questions being asked. Functionality may include predefining a set of follow-up questions, choosing a target number of follow-up questions, guiding the tool to follow up on specific topics or responses only, and providing it additional contextual knowledge or jargon on a technical subject matter.
Ability to Handle Visual Stimuli: Depending on your research needs, you may want participants to evaluate visual stimuli, such as images, videos, or prototypes. While some AI-moderated tools are equipped to manage and incorporate visual elements into the interview process, others may not support these features.
Response Modalities Offered: The modalities available for participant responses— voice, text, or video—can vary between tools. Some tools are versatile, supporting multiple modalities, while others are limited to text only, potentially causing a burden for participants.
Analysis Capabilities: The ability of the tool to synthesize and analyze the data collected is an essential factor. Some tools in this category provide basic summaries of responses, sentiments, and themes, while others offer insightful in-depth analysis that could include pulling representative quotes, generating personas, and even developing full reports. But be warned: the level of analysis capability generally correlates with the cost of the tool.
Using Simulated User Tools for Data Collection
Simulated user tools like Synthetic Users, UXPilot, and Qoqo represent a cutting-edge - and controversial - approach to qualitative research by generating hypothetical “transcripts” meant to mimic human responses in place of real participants. You provide your interview guide and target audience criteria, and the tool does the rest.
While there are not yet too many of these platforms to choose from, there are still several important factors when deciding to use a tool of this kind:
Customization of Interview Guide: For some research initiatives it may be critical to provide the tool with a full interview guide, while for others simply providing a topic or objective may suffice. Some tools of this kind place limits on the number of questions or topics you can “interview” your “participants” on, so ensure to check any such constraints as part of your evaluation process.
Control over Participant Attributes: Some tools function by generating simulated participants from a description that you provide, while others can also draw on existing data (e.g., customer care data) to create more realistic AI users.
Variation in Simulated Responses: Arguably the most important aspect of a tool like this is the quality and realism of the simulated responses. One potential issue with simulated user tools is the risk of generating transcripts with highly similar responses—an occurrence far less common with real human participants. We suggest closely reviewing each AI-generated transcript to ensure you are getting sufficient variation.
Analysis Capabilities: Similar to AI moderated interview tools, some simulated user tools provide some level of analysis, while others may only generate transcripts.
Using General Tools for Data Collection
If you are interested in simulated user tools but can’t find a platform that suits your needs, you may want to try general chatbot-style tools like ChatGPT for generating simulated interview transcripts. It will take a little more effort to get set up, but you can ask a general AI tool to provide a set of however many transcripts you wish based on the study’s research objectives, an interview guide (optional), and a description of your target audience. You can then request an analysis of the data within the same chat.
AI vs. Human Comparison: Data Collection
Can any of these AI tools truly replace traditional qualitative research interviews? As researchers, we were both surprised and impressed by some of these platforms, recognizing their potential for significant time savings in the research process. However, we feel that human-centered design still requires humans to be involved in the data collection phase. In the following sections, we’ll delve into where AI can best support qualitative interviews, and where humans are still a necessity.
AI Win: Dynamic Unmoderated “Interviews”
We were impressed by the ability of some AI moderated interview tools to conduct high-quality exploratory interviews. We felt that they asked good follow-up questions and liked that they could naturally incorporate both closed-ended and open-ended questions into the conversation.
In our preferred tool, participants could respond at their leisure and choose between responding with text or voice, which we think ultimately led to a low-stress participant experience. And because there is no human moderator involved, the risk of participants’ responses being impacted by social pressure to please the interviewer or “give the right answer” is eliminated.
Overall, we felt that the quality of data the tool was able to elicit from participants was high, and comparable to a traditional 1-on-1 moderated study.
We also loved the tool’s ability to give us near-immediate insights with its comprehensive synthesis capabilities - but depending on the platform, we might choose to do our own analysis (stay tuned for Part 3: Qualitative Synthesis to find out why).
AI Win: Quick Turnaround Qual Research
One of the significant advantages of AI-moderated interview tools is, of course, the quick turnaround time. These tools can conduct an entire set of interviews within 1-2 days, drastically reducing the time needed for qualitative data collection. This speed allows researchers to gather insights rapidly, ultimately enabling faster decision-making.
Human Win: Meaningful Interview Participation
We largely found AI-generated responses to lack depth and specificity. Simulated AI interviews resulted in more generic answers, missing the nuanced insights that real human participants provide. Moreover, we felt constrained by the simulated user tools available and their limitations regarding interview length. Real human interactions allow for a more comprehensive exploration of topics, leading to richer and more meaningful data.
Human Win: Ensuring Participant Comprehension
Human moderators excel in ensuring participant comprehension throughout the interview process. AI moderated interview tools follow up on whatever the participant says, which might not always align with the intended question. If participants are confused, AI can rephrase questions, but it might not catch nuanced misunderstandings. Human moderators, on the other hand, can continuously clarify and push for precise answers, ensuring that the data collected is accurate and fully addresses the research objectives.
When to Involve AI in Data Collection
In generative research, our primary goal is exploratory: we seek to uncover new insights, understand emerging behaviors, and identify unmet needs. For this reason, real human participants are invaluable for this type of research.
However, we do plan to incorporate AI moderated interview tools into our process for qualitative research in some specific scenarios:
Known Population: We trust AI tools the most for audiences we are at least somewhat familiar with. When charting new territory, we’ll still rely on human moderators who can pivot the focus of sessions when needed.
Minimal Visual Stimuli: While many AI moderated interview tools can handle visual stimuli, a human moderator can better guide participants’ attention to specific details.
Traditional Interview Structure: While we think AI is effective at running qualitative interviews and asking great follow-up questions, they aren’t yet equipped to take over co-creation sessions or other interactive study designs.
Conclusion
Several AI-powered tools have emerged, claiming to replace both qualitative interviewers and research participants - controversial claims that, if true, could save researchers a significant amount of time.
We put these tools in a head-to-head competition with human researchers and found that in the data collection phase:
AI moderated interview tools are a great way to save time during the data collection phase. We found them to generally elicit high-quality data from participants.
Simulated user tools do not produce the same quality of data as human participants as of this writing, and we would not recommend using them as a replacement for traditional qualitative studies - especially for exploratory, generative research.
Your research objectives will dictate the extent to which human moderators are needed. A skilled moderator can use verbal and nonverbal cues to ensure that participants truly understand the questions being asked of them, something that AI cannot (yet) do.
For these reasons, we consider AI a “skilled researcher” when it comes to qualitative data collection. While we won’t be switching from real human participants to simulated ones any time soon, we see an opportunity for researchers to save substantial time by leveraging an AI moderated interview tool for some or all of the data collection process during generative research.
Interested to learn how AI tools stack up against human researchers in data synthesis and deliverable creation? Subscribe to our newsletter to make sure you don’t miss an issue of our AI vs. Human series – or our ongoing AI 4 UX video interview series featuring founders of some of the most popular AI tools for UX research.
