Embrace AI for Research Planning – But Let Humans Handle Recruiting
Part 1 of the AI vs. Human: A User Research Showdown series
Summary
There are numerous new AI tools claiming to accelerate or even replace your typical user research. We put these claims to the test in a head-to-head study comparing the effectiveness of AI models and human UX researchers throughout the generative research process. We found that the study planning phase is one of the best times to incorporate AI into your research, though some tasks like creating recruiting screeners are still best carried out by humans.
Study Recap
In the rapidly evolving world of user experience research, the emergence of novel artificial intelligence tools presents a new frontier. As we stand on the cusp of potentially transformative changes, the question arises: How does AI stand to impact traditionally human-led endeavors like generative UX research?
At Brilliant Experience, we performed an empirical study to directly compare the effectiveness of AI models and human researchers in conducting qualitative interviews. Our study centered on parents of young children planning international travel, examined under four distinct conditions, each varying in AI and human involvement. Across these scenarios, the researchers (or AI tools) produced standard deliverables: a slide deck report of key insights, a set of personas, and corresponding experience maps for each persona. We evaluated each approach not only for output quality but also for efficiency and depth of understanding.
Read more about our method and goals in our Executive Summary.
This Edition: Study Planning
In this post, we’ll delve into our findings from the study planning phase.
Researchers now have the opportunity to leverage AI tools for several study planning tasks including desk research, workshop planning, study brief and protocol drafting, interview guide creation, and recruiting screener development. Specifically, we utilized AI tools to draft the interview guide and recruiting screener in the AI Only and AI Assisted conditions of our study, and we’ll focus on these deliverables here.
Choosing an AI Tool for Research Planning
If you read our Executive Summary or have been following the proliferation of AI tools, you already know that there are several distinct categories of these tools:
We found through our testing of numerous tools that when it comes to selecting an AI tool to assist in drafting study planning documents, specialized UX research tools aren’t necessarily needed.
We prefer Chatbot-like AI platforms such as ChatGPT, Copilot, and Gemini for study planning due to their vast general knowledge and flexibility to account for whatever the study objectives might be. We were also impressed by their understanding of user research, without any additional training, which was sufficient to produce standard planning deliverables in the format that any researcher or designer would recognize.
Another key advantage of chatbot-style tools is their ability to incorporate feedback and revise specific portions of deliverables. We found that several rounds of editing were typically needed, and working in a piecemeal fashion—focusing on one area at a time—yielded better results than requesting large-scale changes all at once.
AI vs. Human Comparison: Research Planning
So, is qualitative research planning better handled by humans or AI tools? Our research reveals a more nuanced answer. While AI excels in certain aspects of study planning, it struggles with others. In the following sections, we’ll explore the key strengths and weaknesses of AI in creating qualitative interview guides and recruiting screeners.
AI Win: Time Savings Drafting Interview Guides
One of the standout benefits of incorporating AI into your study planning process is saving time writing your interview guides. AI tools excel at generating standard introductory language, outlining a reasonable session flow, and suggesting an endless array of interview questions to suit your study objectives. In fact, we were impressed by the ability of AI to come up with creative questions that might not have been initially considered.
For example, when drafting the interview guides for our family travel study, the AI tool we used was able to quickly produce a list of relevant questions covering a broad spectrum of topics, from logistical concerns to emotional experiences. This rapid generation of content allowed the researcher in our AI Assisted condition to spend more time refining and tailoring questions to fit the specific needs of the study, rather than starting from scratch.
AI Win: Checking Questions for Bias
In addition to helping us generate questions, we also used AI to review our drafted questions in our interview guides and recruiting screeners for any signs of bias. Overall, we felt this was a promising use of AI, as our tool effectively provided suggestions on how to rephrase certain questions to ensure they were more neutral and inclusive.
It is important to note that while AI can do a decent job of identifying and flagging potentially biased questions when asked, AI chatbots might give you biased questions to begin with, so it is always worth a careful review.
Human Win: Crafting Recruiting Screeners
Despite the successes with drafting interview guides, recruiting screeners proved to be a challenging area for AI tools. Our study found that AI-generated screeners often required significant editing to be usable.
Specifically, we found that AI tools could generate basic questions for screeners, but they struggled with two important aspects: obfuscating the intent of questions and ordering the questions according to best practices in recruiting. It is critical to obscure the “correct” answers to the recruiting screener to discourage respondents from answering untruthfully in an effort to be included in the study. At the same time, we want to respect respondents’ time by ensuring that they aren’t answering unnecessary questions. For this reason, disqualifying questions should typically be placed at the beginning of a screener, which was a principle the AI-generated screeners often failed to adhere to.
In sum, we suggest leaving the task of creating recruiting screeners to skilled research and recruiting professionals for now to maintain the data quality in your qualitative studies.
When to Involve AI in UX Research Planning
Although we found AI chatbot tools like ChatGPT useful for drafting interview guides, we ultimately wouldn’t use the output without some level of human editing. To ensure the final planning documents meet quality standards and adhere to best practices in UX research, a hybrid approach is still needed.
Creating Interview Guides with AI
Based on our experience, we recommend providing an AI tool with your study objectives and asking it to generate a draft interview guide complete with a comprehensive list of questions. Then, pare down and edit these questions, and subsequently run the refined guide through AI again for further improvements (including checking for bias, as mentioned above).
Drafting Recruiting Screeners with AI
For recruiting screeners, we recommend drafting it “by hand” to start. As you do so, an AI tool might be helpful in generating basic questions such as demographics, which can then be incorporated into your human-made document.
Conclusion
A plethora of new AI tools have emerged on the scene, promising those in research, design, and product to revolutionize qualitative research.
We put these tools in a head-to-head competition with human researchers and found that in the study planning phase:
AI can speed up the process of drafting interview guides and may result in more comprehensive interviews.
A human touch is still needed to polish interview guides and draft recruiting screeners.
While human expertise remains crucial for high-quality research, researchers can enhance efficiency and depth of understanding by strategically integrating AI into the planning phase.
For these reasons, we consider AI a “trusted partner” when it comes to study planning. While we aren't ready to fully delegate our experimental design and preparation work to AI, we believe researchers should consider taking advantage of the benefits these new tools offer.
Interested to learn how AI tools stack up against human researchers in qualitative data collection and synthesis? Subscribe to our newsletter to make sure you don’t miss an issue of our AI vs. Human series – or our ongoing AI 4 UX video interview series featuring founders of some of the most popular AI tools for UX research.
