I distinctly remember a response former Secretary of Defense Donald Rumsfeld gave at a briefing back in 2002 about the evidence, or lack of it, the United States had linking Iraq to weapons of mass destruction:
Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know.
Donald Rumsfeld, February 12, 2002.
Politics aside, this is a solid distillation of complex epistemology, and the basis for in-depth interviews in market research. We can’t research, let alone run any analysis, on what we don’t know we don’t know; in-depth interviews aim to uncover those unknown unknowns.
My colleagues have already written on drafting qualitative discussion guides and establishing interviewer trust with the consumer. I’m going to discuss the management of this data collection effort: scheduling in-depth interviews. While not the meat of the matter, here, scheduling is the difference between data and no data, yellow cake and no yellow cake. Scheduling is first foil from unknown unknowns remaining unknown.
We recently completed the in-depth interview phase of an assessment of employee retention among the ranks of an organization operating in the defense industry. We needed to conduct interviews in five different segments to arrive at statistically sound generalizations while keeping diversity of backgrounds and opinions within each. A screener survey had given us a segmented candidate pool. So, the first challenge with in-depth interviews is hitting those internal targets, which amounts to scheduling.
Scheduling in-depth interviews is the effective leveraging of limited resources: (1) the human capital required to conduct interviews and (2) the time during which they can occur to be able to deliver the final report to the client on schedule. This isn’t hard to manage, or even worth writing about, when the number of in-depth interviews needed is small compared to the number of staff and number of appointment slots available. That was not the case with this particular project, where with the same limited resources we needed to deliver much, much more.
Robust scheduling standard operating procedures are nice, and a clear allocation of resources towards the project alone is even better. But, that preparation does not avert its biggest hurdle: time. Precious time is spent in vain engaging interview candidates who never respond, or worse, confirm an appointment and don’t show up. Here, we had the interviewers and time to do double the work, but that meant nothing when confronted with the low response rate of a silent candidate pool. Of course, two other factors unique to this project compounded the low response rate: the client chose not to offer any incentive for employee participation in the study, and using only work email addresses when many employees worked part-time.
My advice is to understand that time is the greatest limitation to scheduling, and despite your best efforts, there are other time consuming externalities that are out of your control. Do what you can to moderate them: limit the time spent trying to engage an interview candidate to two business days, during which you should follow up initial requests with an email and a phone call. Drop anyone who doesn’t respond in that time, because if they wanted to participate, they would have usually indicated so by then.