The Qualitative Impact Protocol, known as QuIP, provides a straightforward and cost-effective mechanism to ask people about significant drivers of change in their lives, and to analyse and present the data collected. It was designed to help organisations to assess, learn from and demonstrate the social impact of their work. It places intended beneficiaries’ voices at the centre of reporting and demonstrates a genuine commitment to learning about what your most important stakeholders really think makes a difference to their lives and livelihoods.
Quantitative change data, while important, is rarely a sufficient source of evidence of social impact. However, it can be difficult to access and apply good qualitative research methods within limited budgets. It can also be difficult to convince funders that qualitative research is rigorous and reliable enough. The QuIP has been developed to try to address all these issues, creating an innovative and trusted approach that has gained recognition by leading donors and non-governmental organisations.
Watch this short video for an overview of the method and read more about the main features of QuIP below. For more detail please refer to the QuIP briefing paper here, or see summaries by Intrac and Better Evaluation. We have lots of free resources available on our website!
Key features of QuIP
QuIP is a non-experimental, qualitative approach to impact assessment. Put simply, we believe that asking people what they think has changed and why is a valid and important part of social research. QuIP gives you some simple tools to ask questions in a way which tries to reduce ‘confirmation bias’, the risk that people will tell you what they think you want to hear. This includes interviews with little or no reference to the intervention being assessed, letting participants tell you what they think is important rather than asking them direct questions about specific inputs. The approach works ‘backwards’ causally from outcomes to understand their perceptions of the main drivers of any changes, rather than having a conversation that proceeds ‘forwards’ by starting with specified interventions. This is particularly useful in complex contexts where a variety of factors that are hard to disentangle can influence the outcomes of an intervention.
The approach also involves guidance on how to analyse the narrative data using a form of qualitative data analysis called causal mapping. This enables the visualisation and aggregation of causal pathways across a whole dataset and helps to add rigour to a process which can otherwise be seen as ‘cherry picking’ the best bits from interviews to add to a report.
Scoping and case selection
The scope of a QuIP study is co-designed with the commissioner to build on their prior knowledge and complement other monitoring and evaluation activities. The first task is to establish what you already know, what the gaps are and therefore what the key research questions are. This will help to refine the approach to interview design and sampling.
Qualitative research relies on interviews with a relatively small number of people – this is not statistically representative sampling! QuIP studies are normally planned with 24 individual interviews; focus group discussions can be added to complement these. 24 is not a magic number, but it does divide beautifully into smaller clusters of respondent types – for example, 8 men and 8 women in two different locations. It offers a manageable number of interviews for a pair of researchers to conduct within a week, and in our experience it provides sufficient saturation/ repetition of stories of change within a selected stakeholder group to warrant avoiding additional costs for marginal gain. Selecting who/ which groups to interview will be a product of your research questions; who can help you gain a deeper understanding of the range of causal mechanisms influencing outcomes to explain quantitative findings?
A control or comparison group is not necessary since QuIP relies on self-reported attribution contained in the stories of change generated in the interviews rather than statistically inferred attribution. Selection is better focused on people who you know have relevant stories to share. Read more on the theory behind QuIP sampling here.
The QuIP approach to data collection recommends interviews and focus groups are conducted by a team of local researchers who are fluent in local languages and dialects and who can put respondents at ease in discussions because they are familiar with the culture and context.
QuIP researchers are usually ‘blindfolded’ to some extent – in that they have little or no knowledge of the programme being tested and work completely independently of the project team and commissioning organisation. This reduces the risk of pro-project and confirmation bias. Complete ‘blindfolding’ is not always possible, for example in low trust contexts or when interviewing government or organisational representatives, but it is still possible to conduct a QuIP without blindfolding. The open-ended, outcomes-based approach to interviews is often more important to mitigating confirmation bias than blindfolding is, and this is still possible in any context.
QuIP has developed a particular approach to analysing narrative data to identify causal mechanisms or claims, connecting cause and effect with links. Coded links are made up of an influence factor and a consequence factor, rather than simply coding sections with isolated factors. This is a form of causal mapping which can be done manually using software such as Excel. We have invested in the creation of the Causal Map App, bespoke qualitative data analysis software which speeds up the coding process and enables users to visualise the data in the form of causal maps, aggregate and filter hundreds or thousands of causal links across a dataset.
Once a dataset is coded the aggregated map can then be filtered by thematic areas of interest or by demographic data, and sub-maps can be analysed to understand the relative strengths of the main stories of change generated from interview transcripts, and to answer the main research questions. There are ways of tagging and flagging the links to make it easier to filter for stories which are relevant to attribution questions at the analysis stage.
Respondents’ voices remain at the centre of this process, the statements behind each link are easily accessible and often used in reporting – opening the black box of data analysis which often sits in between transcripts and reporting.
QuIP and other approaches
The QuIP draws on many existing qualitative approaches to impact assessment. It is the product of a learning-by-doing, iterative approach to methodological development that borrows from several other approaches including Realist Enquiry, Contribution Analysis, Process tracing, Outcome Harvesting and Most Significant Change. These are all mutually affirming approaches that belong to a broad family of more qualitative and interpretive approaches to assessing change. If you are interested in how QuIP relates to this family of methods (and more!) please see some comparisons in the links below: