“Tell me in a paragraph, but make sure it’s robust”
| 26 November 2024 | News,Latest QuIP News
Summarising complex pathways of change for a Cash Plus project.
This year saw us undertaking analysis of over 100 QuIP interview transcripts for one of our largest evaluations to date, for Save the Children, with the support of our longstanding research partner in Malawi – Palm Consulting. Such a large and complex evaluation has forced us to think very hard again about how best to summarise nuanced findings in ways that convey key messages for policy, but in ways that don’t oversimplify. Is this possible? We’re still experimenting, and the conversation continues within the Causal Pathways network and in other fora, and we hope to hear feedback from anyone who would like to contribute.
We have worked with Save the Children over many years in different countries on their nutrition and food security integrated interventions to improve child and maternal health, family wellbeing, and gender equality (this case study provides an overview of this work and links to all the published reports). The most recent evaluation of the MAZIKO programme in Malawi was completed earlier this year, culminating in a learning event in Lilongwe on 21st November 2024. MAZIKO is a cash plus pilot project, linking regular cash payments with community work, jointly delivered with GiveDirectly and the Government of Malawi. The community work focuses on providing advice on nutrition, early learning and maternal mental health via care groups, promoting shared caregiving responsibilities through designated male champions, and promotion of sustainable agriculture techniques and effective use of village savings groups to help manage finances.
A mixed methods midterm evaluation
The MAZIKO evaluation also marks the latest in a series of studies where we have worked alongside a randomised control trial, in this case managed by the International Food Policy Research Institute, IFPRI. In May-July 2024 the Palm Consulting team conducted 103 deep-dive qualitative interviews with mothers of young children – selected from the midterm survey sample, with 96 interviews considered for the final report (without pilots). The sample and hypotheses were informed by the early results from the midterm RCT, enabling us to focus on the areas where more detailed information was required about the mechanisms by which intended outcomes were, or were not, being achieved. QuIP is one component of a mixed method monitoring and evaluation strategy, its main contribution being to cast additional light on key causal pathways arising from different intervention packages.
The sample spanned the two project districts (Ntcheu and Balaka) and was split equally between three intervention arms: community work only, community work plus a small monthly cash transfer, and community work plus a larger monthly cash transfer. We also conducted six separate group discussions with men participating in the male champions training programme.
The evaluation found that health and nutritional advice was mentioned as a positive driver by all arms, but as a mechanism of change was most effective in conjunction with cash transfers which enabled mothers to act on the advice received (e.g. being able to buy food even when yields are poor). Economic and climatic crises had a strong counter effect, particularly increasing food insecurity amongst those household receiving either low or no cash transfer – with most positive stories of behaviour change arising from community-based groups when enabled by simultaneous receipt of the larger cash transfer.
Making sense of complex pathways
A key skill we have grappled with over the last decade is striking a balance between capturing detailed stories and providing an accessible summary of the main findings. Our approach to causal mapping is part of this process, helping to code and then aggregate individual causal pathways and to see patterns across a large set of case studies. In this study we experimented with going a step further and synthesising the map findings into a summary causal matrix. Although this removes much of the detail along the causal pathway – and is therefore not a substitute for causal mapping – it does make it easier to convey headline links between the main drivers and outcomes.
You can find the learning brief and full report on the Save the Children website: the learning brief summarising findings from both the QuIP and the midline RCT survey, while the full QuIP report includes detailed causal maps linking all intervention arms and outcome domains. However, in this blog we would like to focus on how we distilled the many detailed maps into summary overall findings. The key areas of interest for this evaluation were:
- Diversity and quantity of food consumed by all family members, including babies
- Health outcomes and hygiene practices
- Maternal and family wellbeing
- Family relationships and childhood development
- Gendered household roles
The overview map taken from the learning brief displays the main drivers and outcomes across the whole group, and elaborates on the change mechanisms in between. The map has been recreated manually but based on all the original data, where a thicker line indicates that more people reported this link. Overall, you can see that women reported that the cash transfers and the Maziko Care Groups were contributing to increased and more diverse food consumption, improved health, wellbeing, and resilience (ability to cope). Advice and guidance from Health Surveillance Assistants (HSAs) and health centres were also mentioned by most women interviewed. Advice and guidance led women to report improved understanding, improved skills, and intended behavioural change. However, substantive change in food consumption and health was contingent upon having the cash resources to act upon the advice and guidance received. For example, although high numbers of people report having gained better nutrition skills, we see a significant decline in numbers of people linking this to actual improved food consumption or health.
While the map shows us the main mechanisms reported by all mothers interviewed, the table below focuses on the differences in the numbers of positive outcomes reported between the members of the different intervention groups, and what the main drivers of change were for each type of outcome in each group. All attribution is based on what the participants reported as the source driver of change. Positive reported outcomes are in the column on the left, with the main attributed drivers written in the corresponding boxes in each row (primary and secondary drivers where relevant). All colours relate to the intensity of the outcome; deeper shaded boxes indicate that more women mentioned the outcome in that treatment group, and a red border indicates that over 50% of women mentioned the outcome. The highest level of reported outcomes was 91% of that group (top right), the lowest was 6% (bottom left).
This summary table shows us that income from the cash transfer is the strongest driver of change in the ‘Low’ and ‘High’ cash groups, alongside knowledge gained from the Care Groups and HSAs. However, although the number of positive outcomes is slightly lower in the group not receiving any cash, we see few differences in the level of reported positive outcomes between the No and Low cash groups, implying that outcomes could be similar without the smaller cash transfer. The High cash transfer group has markedly higher reporting rates of positive outcomes. We have deliberately avoided using numbers since this is not a representative sample, but the intensity of reporting is indicative of different experiences between different groups.
It may be simplified, but it’s not simple… and we invite you to reflect on the different messages you glean from each visual, and how adequately these summarise the 100ish pages of detailed (manual) qualitative analysis which lies behind them. Can we further simplify without losing the key nuanced findings? Or should we not even be attempting to consolidate qualitative findings like this? Continue the conversation on LinkedIn.
Comments are closed here.