60 Decibels publish Lean QuIP learning brief
| 28 November 2024 | News
Written by Hanadi Al-Saidi of 60 Decibels and Hannah Mishan of Bath SDR
Evaluations face many constraints – both external and internal, making it challenging to apply a one-size-fits-all methodology (see more about methodological bricolage here). What proves effective in one context may be ineffective or not relevant in another. Bath SDR had the privilege of supporting 60dB as they adapted the QuIP methodology to suit the practicalities of their evaluation, adding it to their lean evaluation suite of tools. They have created a Learning Brief on the Lean QuIP approach which you can read here.
60dBs is a tech-powered impact measurement company who speak directly to customers, employees, or project beneficiaries, usually over mobile phones. They collect rapid data and provide impact reports to corporations, investors and NGOs, to help inform their programmes. Since 2022, they have been working in partnership with the Busara Center for Behavioral Economics and the Bill & Melinda Gates Foundation to improve access to digital solutions aimed at helping smallholder farmers.
60dB conducted two lean QuIPs on this programme, exploring the impact of:
Standard vs Lean QuIP
Interviews
The interview structure takes a familiar QuIP approach; using outcome domains as the basis for questions and not mentioning the intervention at all in the questions. Farmers were asked to talk about changes within a specific period related to selected outcome domains. For example, “Since Easter, has anything changed in the way you farm?” They were then asked further questions to ensure the understanding of (1) the main driver of that change, and (2) to whom/what they attribute that change.
In many QuIP studies, researchers and respondents are not briefed on the programme details. This reduces confirmation bias and encourages respondents to speak about what is most significant to their lives without being guided to focus on the intervention. In these studies farmers were informed why 60dB was conducting the research to enable them to ask process evaluation questions on customer experience after asking the open-ended QuIP questions. They chose to collect data remotely, over the telephone, to allow for rapid data collection with less resources, and reaching respondents over a larger geographical area.
Sampling
Sampling for QuIP typically starts with a unit of 24 purposively selected respondents who share similar characteristics. This number is based on the principal of thematic saturation, the idea that you will reach a number of interviews when are unlikely to hear any new themes. 60db lacked detailed data from the companies to help them sample purposively so felt they needed a larger number to reach this saturation point. Since the interviews were shorter, this was possible with the resources available, and they found that a sample size of approximately 50 was sufficient. They interviewed 51 respondents for the Ignitia study; when this was exceeded (they interviewed 100 people for DigiCow), the amount of time required for coding and analysis became burdensome and no new trends emerged from the additional interviews.
Analysis
The goal of a QuIP is to understand the causal connections that respondents report between drivers and impact. 60dB chose to use analysis tools that were familiar to their organisation, using Excel to code connections in the data which enabled them to create charts of links made between factors, rather than causal maps. For example, they found that about half the farmers they spoke to told them that Ignitia’s SMS advisory services led to a rise in their crop yield or production.
![Text on blue back ground. Title reads Better crop quality and yield. Quotation from male, 46 reads " Yes, [quality improved] as a result of advice I got from the company. They advised that I should remove weeds before applying chemicals and it was very helpful because there is this black insect that usually infects my cowpea. Since I used this method I have better cowpea without insects, compared to last harvest"](http://bathsdr.org/wp-content/uploads/2024/11/quote-700x402.png)
DigiCow and Ignitia found the insights 60dB provided very useful. The DigiCow report is available here. The QuIP lean findings have allowed them to feel more confident in their theory of change and provide funders with evidence of impact. The process improved 60dB’s knowledge and expertise in the lean QuIP approach, a methodology they have added to the lean evaluations they offer and promote.
Combining the key features of QuIP with 60dB’s existing capacity to conduct telephone interviews allowed for rapid, remote data collection whilst retaining the benefits of a theory based approach to impact. We have seen this approach to contextual adaptations of QuIP in other external evaluations conducted for the World Food Programme, IOM and UNICEF (and look forward to sharing those in due course). And don’t forget, if you are interested in using remote interviews with a larger sample size, check out QualiaInterviews, a new tool designed by our sister company Causal Map. The interview bot has been designed to probe for causal stories, allowing you to gather QuIP-type information at scale.
Comments are closed here.