How many interviews are enough?
| 18 April 2023 | News
A typical QuIP evaluation study will be based on either 24 or 48 interviews (and sometimes 4 or 8 focus groups). This invites a lot of questions! Is that enough interviews? Why start with 24? How can that be representative? We have lots to say on this – and if this interests you, check out our resources on case selection, including a journal article by James Copestake discussing our approach to sampling in more detail.
All qualitative research is based on small sample sizes, mostly not aiming to be representative. However, it’s important not to discount the value of the findings from a small sample if selected according to some robust principles. The concepts of Bayesian belief updating and thematic saturation are important foundations to this approach – and may change how you think about sample selection. You don’t need to write out complex equations to understand the basic premise of Bayesian updating – we all do it in everyday life. We are looking for evidence to add to what we already know, acknowledging that we are usually starting from a point of some rather than zero knowledge.
The purpose of a QuIP isn’t to collect average statistical data on impact, it is to understand how and why a project is or isn’t achieving impact – the causal mechanisms which are associated with positive outcomes. Most projects we are evaluating will have some monitoring data or project reports which give an idea of what is and isn’t working – or places/groups where things seem to be working better/worse than others. This is a good starting point for any sample selection – rather than trying to speak to an average representative sample, why not speak to the people who we know will have something interesting to tell us about how and why an intervention is or isn’t working well? This should add information to what we already know and help to build our understanding of how and why change is or isn’t happening, and how closely this relates to the project’s theory of change.
Why 24? Two main reasons… First, much research on thematic saturation within narrative data shows that qualitative data analysis of more than approximately 20 interviews offers little in the way of additional new codes or information. The rate of diminishing marginal returns slopes up steeply after this point if your sample is a relatively homogenous group that share similar experiences. 24 interviews allow us to provide a cost-effective and timely evaluation that explores the experiences of a certain group (24 interviews are usually achievable within a week by a team of two researchers). If you have very different types of groups/places/interventions – then scale up in sets of 24 (hence studies which interview 48, 76 etc.)
Second, 24 is such a beautiful number – with oh so many factors! You will want to split your sample purposively, for example age, gender, location. 24 enables you to, for example, interview 12 people in each of two locations or 8 in three locations (to control for any unexpected factors which may be pertinent to just one place), and split that whole sample between men and women and older and younger respondents. Using sub-clusters of 6, 8, 12 gives you a lot of flexibility – all hail the great number 24!
Comments are closed here.