Triangulation conversations and endgames
| 2 September 2015 | ART Project Updates
(Originally posted at Devlog@Bath)
With field work complete, the ART Project is now in its final stages; workshops in Ethiopia and Malawi during July brought many of us together to reflect on what we have learnt, and on what to do next. Many thanks to all who participated, especially to staff of Farm Africa and Self-Help Africa in both countries for hosting us. If I had to pick out one recurring theme from these workshops it would be triangulation – i.e. learning about the same subject by confronting data about it from acutely different angles.
The figure below identifies at least six sets of triangulation conversations that took place at the workshops.
Second, was feedback from NGO staff on the formal QUIP analysis (Cell 3). Naturally, much of this focused on respondents’ explicit or ‘smoking gun’ attribution of impact to project activities – positive and negative (see page 7 of the QUIP Briefing Paper for more on ‘smoking guns’ and Rick Davies’ annotations on ‘necessary’ and ‘sufficient’ causal conditions for passing hoop and smoking gun tests). These discussions served as a data quality check and usefully highlighted some respondents’ misperceptions about project activities. They also stimulated further discussion and understanding of what worked, what didn’t, and why – e.g. with reference to farmers’ access to services provided by primary cooperatives.First, were conversations between our previously ‘blinded’ field researchers and the NGO staff responsible for the project they studied (Cell 2). The QUIP Methodology emphasises the potential to mitigate confirmation bias through blinding researchers in the field, but a downside of this is that they are less able to probe deeply into project related issues. This accentuates the importance of debriefing conversations with NGO staff once the research is complete. For example, Professor Zenawi from Mekelle University was interested to explore with Farm Africa staff how to deal with some beneficiaries’ distress at being required to pass on to others the offspring of the goats gifted to them. The workshops highlighted the huge scope to strengthen within-country collaboration and reflective conversations between NGO staff and university researchers.Here are just three examples of some of these.
These conversations also turned to QUIP methodology, particularly sample selection. We explored the importance of linking evaluation to routine monitoring of key performance indicators, enabling the QUIP samples to quota sample recipients of different project packages, and to deliberately select outliers from whom more could be learnt than from a more uniform sample of ‘typical’ project beneficiaries. (This echoes Atul Gawande’s emphasis on learning from “positive deviants” – or what Chris Roche calls “exceptional responders”). The priority attached to formative and summative goals is also critical to decisions about the timing of studies: late enough to test the full theory of change, but early enough to inform mid-course adjustments.
A third triangulation conversation (from Cell 5) arose from comparing estimates of the size and composition of changes in disposable income per adult equivalent (DI/AE) of selected households (using monitoring data from the Individual Household Method surveys as undertaken by Evidence for Development) with narrative accounts of drivers of change obtained through the QUIP studies. Irrigated vegetable cultivation can be a strong cash earner in both countries, for example, but how profitable it is depends heavily on the terms under which land can be rented or sharecropped in. IHM data identified multiple and diverse reasons for changes in DI/AE, whereas the QUIP narratives often highlighted a few dominant stories, often linked to family illness, weather shocks and price shifts. Perhaps the biggest overarching story emerging from concerns the tricky judgements about risks and returns to market specialisation and diversification farmers are having to make as they shift from staple food production for own-consumption to earning cash. This serves as a warning against assuming that simple ‘one shot’ asset transfers can contribute to rapid, transformative livelihood improvements on their own.
These kinds of conversation are of immense value – adding much to an evaluation that mechanical treatment-control comparisons miss. They also highlight the dangers of focusing dissemination too narrowly on set-piece final reports. If face-to-face workshops are not possible, online knowledge sharing technology can also sustain deeper conversations (in line with John Gargani’s EvalBlog prediction No.5 for the future of evaluation). Achieving closure in triangulation debates is also important, but a measure of the depth of mutual understanding achieved is also the willingness of commissioners and evaluators to work together again.
Meanwhile, what’s next for the ART project? To misquote Churchill, this is not so much the end as the end of the beginning. We still have much to finalise from the pilot studies. But in response to positive feedback from many evaluation experts, we are also exploring options for disseminating the QUIP methodology more widely and adapting it for impact evaluation of interventions outside agricultural development. This is likely to include, for example, assessing the impact of changing access to financial services.
Comments are closed here.