Breaking into UXR can be hard– really hard. As the industry continues to grow both in size and demand for skills, researchers are trying to figure out how to stand out from the crowd. Some vocal leaders and bootcamps claim that successful entry and advancement in UXR comes easiest (or exclusively) with a mixed methods toolkit. But what does it actually mean to be mixed methods, and is it really all that imperative to succeed?
tl;dr: nope, not at all. Let’s dive in.
If thought articles are to be believed, mixed method researchers are officially the UX equivalent of hand sanitizer in March of 2020– uniquely valuable, and imperative for success. “Mixed Methods or Bust” is becoming a common creed, especially among those just entering the field of UXR, and qualitative research is as out as bell bottoms.
However, dismissal of single-discipline research does not make sense in a variegated industry where the role, scope, and value of UX Research is still developing, and the true power of research is still being discovered. (Quick reminder, UXR is new. Really new).
But mixed methods seem to stand out as people try to fit into the increasingly competitive field — it’s logically appealing to be a ‘jill of all trades’. If you have every tool, no one can say you’re not qualified…right?
This approach misses two key points:
- There’s huge and unique value in each of the research disciplines (qualitative, quantitative, and mixed methods)
- Overstating qualifications for a method (most commonly seen when discussing quant skills) creates huge risk, both in the hiring pipeline and once hired.
In this article, I’ll define mixed methods, discuss three common myths that are over-emphasizing the value of mixed methods, how we got to this point, and why UXR as a discipline (and your career) will best thrive when embracing all of forms of research.
What Is UX Research?
Before we can talk about methods, let’s align: what even is the job of a UX Researcher? A UXR must be able to strategically acquire information, synthesize that information into actionable knowledge, and then communicate and socialize that knowledge to the broader team to make decisions, inform roadmaps, and understand the impact of decisions.
In contrast to an academic researcher, where methodological expertise reigns supreme (and reviewer 2 will always make sure you’ve gotten even the nittiest detail correct), in the tech industry, impact reigns supreme. Impact is the reason what you did mattered (what changed, who listened, how we’ll approach the problem, etc.).
In other words, UX Research is not just methods (surveys, usability, and so forth), but methods and the ability to get people to listen to their outcome.
What Is Mixed Methods UX Research?
Quick step back — while it may seem straightforward, for the sake of clarity, I offer a few associated definitions. Skip ahead if you’re comfortable with these concepts already, or check out this YouTube video if you want to learn even more.
Mixed method research: The process of using both qualitative and quantitative research within the same study, to answer a question.
Mixed method researcher: A researcher able to use both qualitative and quantitative research methods in the same study, and can triangulate these diverse data to create a more helpful or descriptive story that is more than the sum of its parts.
Triangulation: The process of combining multiple data sources to get a better, more precise understanding of the real world. While all researchers triangulate, only the mixed method researcher is expected to be able to gather both qual and quant data themselves to inform triangulation. (Fun note: triangulation of multiple methods within a discipline is called multi-method research)
Quantitative methods: Broadly, quantitative methods involve collecting and analyzing numerical data, almost always through deductive reasoning. These involve both observational (e.g. logs analysis) and interventional (e.g. surveys, card sorting) methods, as well as statistical techniques (e.g. t-tests, chi-squared, ANOVA, HLM). In general, these stats tend to tell us if things are related, how they are released, and how much the data you have explains the thing you’re interested in. Additional techniques like MaxDiff and Conjoint analysis can also help you look at the relative importance of things. You can learn more about some of these techniques here.
Range in Quant Expertise in mixed methods: Mixed methods researchers may have different skills, and variety thereof, depending on their level, focus, and background — one may only use basic statistics, while another is comfortable with Bayesian inference. For those trying to figure out where they land as a researcher, here are some common minimum quantitative requirements to be considered a mixed method researcher:
- Properly determine sample size, including a power analysis
- Able to defend the desired N relative to the desired impact/questions answered in the work
- Basic Stats Concepts: p-value, effect size, variance explained, data normality and non-normality
- Basic Stats: Chi-square, t-test, ANOVA, GLM, post-hoc weighting
- Able to describe how and when you might use each of these stats, and the requirements to do so.
The State of Mixed Methods Research Today
The use of mixed methods research has dramatically increased across many research disciplines since 2004, and this trend is also quite common in UX. Indeed, in a 2021 analysis of 68 recent UXR job postings, mixed methods were the most commonly requested skill set. Optimal Workshop went so far as to say that mixed methods were the number 1 research trend, and OpinionX opined that mixed method research was the #1 disruption to the research field in 2021.
So perhaps it’s unsurprising that newcomers to the field have their sights set on mixed methods– even Springboard and UC Berkeley’s UX bootcamps mention quant skills as “essential” for success in addition to qualitative skills.
So, what’s going on here? After decades of success in single discipline research, has something changed to drive the increased importance of mixed methods? Is this the faint ringing of the death knell of qualitative- only or quantitative-only research?
While we can spend time hypothesizing on where this obsession with mixed methods came from (the startup desire to wear 100 hats, poor understanding of research at some companies, bootcamp’s desire to upsell their products, researcher’s desire to gate keep, etc.), our time is better spent getting to the point:
No– the death knell hasn’t rung, and single discipline researchers will continue to thrive. While mixed methods are powerful, they are not the Marvel Infinity Stones. Let’s break down some of the most common myths around mixed methods research, and where we can go from here.
Myths of Mixed Methods Research
Myth #1: Mixed Methods research is the same as qualitative work + writing surveys — you should just write it on your resume for bonus points regardless of skill level.
This is one of the biggest myths that I see hurt researchers during hiring interviews. Applicants apply for a mixed method role with only survey writing expertise, without understanding of how to properly distribute or analyze the survey. While survey writing is valuable, it is insufficient for a mixed method role. More on this from Curiosity Research & Design.
Let’s break down the actual requirements for mixed methods:
Surveys: Surveys themselves have three distinct skills: writing, distribution, and analysis. If a researcher is only comfortable with writing surveys, they would be considered a qualitative researcher with survey writing skills.
Writing a good survey is hard. There are entire brilliant books simply on the process of writing surveys (I recommend Questionnaire Design by Ian Brace, and Internet, Phone, Mail, and Mixed-Mode Surveys by Dillman et al.). It’s not a practice to be picked up by simply writing a few surveys or the “guess and check” method.
Distribution entails both actually sending the survey (deciding the best way to send to maximize response reliability and reduce bias), and assessing the necessary sample size to ensure the data collected are representative of the target population (conducting and understanding a power analysis).
Statistical analysis involves both descriptive and inferential statistics, and the use of associated programs. Common methods are t-tests, ANOVA, MANOVA, GLM, while common tools are Program-R, SPSS, Qualtrics, and JASP. The mixed method researcher should understand when and why to use the given statistical test, as well as the requirements to do so (e.g. data normality).
While mixed method researchers are rarely required to code, some companies do look for coding expertise– check individual job descriptions before applying. In those cases, comfort with SQL and Python may be required, though this is much more common with quantitative UXRs.
Other quant skills: Specific other quant skills needed for a role will vary greatly (more on this above), but all mixed methods researchers need comfort with at least a few advanced quant skills like tree testing or advanced statistics).
You might be thinking, “ok, I’ve got some of these though. What’s the risk?”. Unfortunately, there are a bunch. Interviews can quickly become embarrassing and difficult to recover from when answers are not known, which ultimately results in rejection. Even more frustrating, you may have been able to pass a single-discipline interview with flying colors, but ended up with a rejection, and a loss of credibility, because you were unable to prove competency for the job you applied for (and no, the hiring manager won’t simply place you in the other appropriate role instead). If you do make it through the hiring process without the appropriate skills, you risk quickly losing both your credibility, and the credibility of UXR as a whole within your team. I strongly advise not risking either of these outcomes.
Myth #2: Mixed method UXRs are more impactful than single-discipline researchers.
This is, perhaps, the most detrimental myth not only to mixed method research, but research as a whole. This often boils down to two key arguments:
- Mixed method research is able to deliver insights that are deeper and more powerful than single-discipline research. In a 2021 article, Optimal Workshop goes so far as to say these researchers are able to “gather data, dive deeper and generate insights that provide more information on our users than ever before”.
- Mixed method research is more powerful because it speaks the same language as the engineers and product people we work with, enabling better discussions and decision making.
Let’s look at each of these a bit more closely — mixed method researchers are certainly able to generate a wide variety of insights, but variety is not always equivalent, or even related to, depth. Surface insights from a dozen methods are unlikely to compete with a deep dive with one method.
Further, this discredits the value of leveraging the expertise of other researchers or partners (e.g. Consumer Market Insights, Data Science) to create more varied insights, regardless of the researcher’s background. Each discipline is uniquely capable of generating nuanced, impactful, powerful work (more on this below). How those insights are generated (qual, quant, mixed method) should be determined by team and project means– not the arbitrary use of mixed methods.
This is not to say that mixed method research is never valuable, never strong, or never high quality (the author is one herself), but rather, that it is not inherently better, or inherently a skill worth working towards. Researchers must value great research, not specific methods.
Researchers mistake the idea of being a mixed method researcher with knowing a broad swatch of methods, often meaning they don’t deeply learn any single method.
Paul Pattishall, Founder and CEO of Datsfire, former head of UXR at IRL, Lyft.
In reality, there is immense value in going deep within a single method to uncover nuances and depth. Rather than focusing on doing all the methods, it’s more important that researchers understand the tradeoffs across methods and when to bring in other researchers or partners who are better equipped to answer the question.
The argument around “using the same language” raises particular concern, as it seems to argue that the strongest data are data that are the same, or communicated similarly to, data the team already has from other sources (This sameness is often referred to as “data currency”). While alignment across data sources and approaches (e.g. understanding how experiment results support or don’t support user research) is important, sameness is not. Diversity in our work, just as it is in our teams and workplaces, is our greatest strength. Just as researchers seek to understand business goals and engineering constraints, a strong team has cross-functional partners that seek to understand users and research.
Sameness also has the potential to cause confirmation bias, leading researchers to miss out on huge issues that simply have not occurred to the team (this especially comes into play with DEI, and at risk populations). We should seek to build products with a wide variety of inputs, from a wide variety of sources, based on a wide variety of professionals, to ensure we understand and design for differences across our users.
Certainly, there’s an ease in deferring to quantitative data, and the appeal is evident. It’s hard for anyone to push back with the researcher say ‘70% of people want a thing.’ But often, that’s the thing the PM or Lead already decided they wanted to do — they would have just as easily ignored the data if it didn’t agree.
Heath Kessler, Senior Design Manager, Meta. Former Google, Samsung, Apple.
The most impactful UXR isn’t ever from a specific method, or set of methods. Rather, it’s the research that makes us question, reevaluate, or dive deeper into a path to ensure our next steps align with a broad, strategic vision.
Myth #3: There is no longer space for single-discipline researchers
Strong companies worldwide (Google, Lyft, Meta, Microsoft, Airbnb to name a few) embrace single-discipline and mixed method research as highly valuable and viable long-term career paths. As these researchers advance in seniority, they hone their craft, up-level those around them, drive broad influence around complex, ambiguous topics, and know when to partner with others for research best answered with a method outside of their skill set. (Read: not by becoming mixed methods by default).
As a researcher you can specialize by being excellent at both qual and quant, or you can specialize in other ways. There’s. no ‘right’ way to specialize, it’s about what skill sets you are interested in.
Alec Levin, Founder and CEO of Learners and UXRCollective.
It’s worth noting that almost all the people I know who lead research functions were promoted to those roles because they are great storytellers, team builders, and strategists — not because they are skilled at statistical significance tests.
Why be (or hire) a single-discipline researcher when you could be (or hire) someone who does multiple?
- There are many skills beyond methods that a company values, and it’s rare to find a single person who can do everything. Depending on the needs of the team, a hiring manager might prioritize expertise in storytelling, service design, workshop design and facilitation, vision and strategy, leadership and cross-functional influence, and/or business acumen over mixed method skills. The impact desired from a researcher should be fully thought out — Mixed methods is not shorthand for “best fit for everything”. Lean into unique value.
- Building teams with diverse backgrounds and complementary expertise across methods and other skills enables team members to learn from one another, stretch how they approach problems, and leverage each other’s strengths when collaborating. This diversity brings huge strength (and even job satisfaction).
- Some teams with a smaller user base or hard to reach populations are simply better suited for a qualitative researcher. The same may apply for topics (e.g. Trust and Safety, Kids) where qualitative research will generate better insights.
- Quantitative researchers often have more advanced quantitative skills than mixed method researchers. They are able to work with much larger datasets and leverage programming and advanced statistics to better assess user behavior, and the gap between what users say they will do and what they do in practice.
In Summary
Research is not a one-size-fits-all discipline– indeed, the diversity in our backgrounds, education, experience, and skills are our greatest strengths. These differences enable thought partnership, collaboration, and mentorship that push us to try out new approaches to solve problems that deliver value to users and the business.
For job seekers: bring your whole, authentic self to your job search. If you are qualitative, embrace it! Your understanding of user motivations, needs, and the true “why” of it all pushes us to think more broadly, inclusively, and creatively. Quantitative UXRs, your ability to scale and move forward with speed is often just the thing we need to round out our understanding of a problem and determine next steps. Mixed methods– for those who do truly love both worlds, find your specific niche and lean in– your understanding of how you want to drive impact relative to changing job needs can unlock great things. For everyone– be the researcher you are; open, honest discussions of your strengths and weaknesses give hiring managers a true sense of what you can bring to their team and company (and how it might fill gaps you don’t know about!), and will give you the best chance of being satisfied in your role once hired.
For Managers: It can be easy to say, “I want my next role to be mixed method because I don’t have that right now, or it just seems easier, or I saw a great thought article”, but we encourage you to think through the desired impact more fully– What skills are your team missing where you are actually feeling the pain of that gap? What skills would your current team like to learn more about and flex into, and could use a mentor? What role can help your team drive the type of impact (product, business, etc) that you need right now? And, are you hiring because it’s the right person, or just because it seemed simplest? As we do so effectively once folks are hired — chase the impact, not the title.
Acknowledgements
Thank you to my former research manger, Londa Overbeck, for her invaluable insights on this paper, to my friend Amanda Gelb for her support, edits, and general brilliance, Alec Levin for his commentary, Paul Pattishall for his endless support, and my many mentees on ADPList who trusted me to help on their journeys.
About the Author
You may have read this with one thought nagging in your head — isn’t this author is a mixed methods researcher? Isn’t it weird for her to say ‘it’s not the way’ when it worked for her? And, yes, I am, and sure, I suppose it did — of all things, I used to be a wildlife biologist, doing things like population modeling and running surveys to look at wildlife value orientations. I came into qual later, and product after that. I’ve learned through trial and error that the best UXR you can be is the one you authentically are — each person is going to bring their own talents and skills. The point here is simple: Be you, drive impact, avoid worrying about what you haven’t done; shine with all you really have.
