BLOG: The value of qualitative data and why I love interviews
Bridget Barry
As an evaluator, I’m no stranger to surveys. Actually, more than once you might even have caught me saying that I am quite fond of surveys. I’m not ashamed! I love surveys! I like how clean they can be. I like putting myself in the shoes of the survey participant and visualizing different scenarios which would require different response options. I love testing my surveys and finding one teeny glitch or broken survey logic before they go out the door. There’s something satisfying about a survey. It’s a puzzle.
Through the Research, Evaluation, and Learning (REL) Unit at IIE, we evaluate some of the most largescale international scholarship programs in the world. Program participants and alumni reside in every country worldwide, from cities to villages. They’re 18 years old and 80 years old. Sometimes a survey is just the best way to reach everyone. And who doesn’t love a massive dataset?
But in 2020, we wanted to speak to alumni of the Ford Foundation International Fellowships Program (IFP) about something a bit less conventional. We weren’t asking about professional trajectories, knowledge acquired, skills gained. We weren’t even explicitly trying to quantify impact. We wanted to understand decision-making behind arguably one of the most important and personal decisions a person can make – where to make one’s home.
Through our Global Alumni Survey conducted in 2018, we came to realize that 13% of IFP alumni were living outside of their home country. And we discovered that via the following survey questions:
Which brings me to that three-word phrase at the end: “Please explain why.” Yikes! Detail your life’s decision-making to us please in this optional open-ended short answer question format. It just doesn’t cut it, you know? No shade to surveys, but sometimes they can’t get at the human behind the survey-taker. And this is where qualitative data enters the chat. Focus groups! Individual interviews! Dialogue! Probing questions! Ooh la la!
My colleague Nora Nemeth and I had the chance to speak via virtual interview to 59 IFP alumni about their experience living outside of their country of origin. We designed an interview protocol that made no assumptions, included no judgment, and sought to understand why. We hoped to inform not only future scholarship program design but also scholarship program evaluation with our findings – program success does not have to hinge on alumni return rates to their countries of origin.
I won’t go into the explicit findings here – you can hear more about those over at our final product, four-part podcast Between Two Worlds – but the range of findings we made is a reminder of the power of qualitative data. As researchers, and just humans, we enter every conversation with our own assumptions, experience, and bias. This can inform how we design evaluations, perform analysis, and perceive data. I’m not saying that qualitative data is the solution for eliminating bias in researchers, but I think it’s a good start to make sure that the participants are front and center.