News & Views

Using QuIP to Evaluate a Holistic Community Development Program (World Concern)

By Fiona Remnant | 30 April 2025 | News

A guest blog written by Kristen Check, Vice President, EIG Insights and Katie Toop, Senior Director of Transformational Development, World Concern

In 2021, World Concern commissioned an external evaluation of the OVT program to assess its community impact. This blog details the rationale and process, but you can see a summarized version of the report on the World Concern website). World Concern’s objectives for the evaluation were to:

1. Understand OVT’s overall efficacy
2. More effectively tell the story of the impact of our partnership with communities
3. Adapt OVT program design to reflect lessons learned

Several aspects of the Qualitative Impact Protocol (QuIP) approach made it conducive to meeting these objectives in a way that is consistent with how World Concern operates. Of particular importance was the centering of the voices of those directly involved in and impacted by the project – the project participants themselves. In QuIP drivers of change are identified by the communities themselves rather than being predetermined and tested by the researchers. For an organization committed to learning from and alongside those we serve, this was an overwhelmingly positive factor.

QuIP’s flexibility was also ideal for a multi-sector, highly contextual program. It allowed World Concern to inquire broadly about program participants’ perceptions of impact across several domains and to discover both intended and unintended drivers and impacts.

In addition, the ability to identify and map participants’ views was of great benefit. In applying lessons learned for the strengthening of the model, it is invaluable to understand what drives impact and how. Part of World Concern’s aim was to tell the story of their program more effectively, and this approach guaranteed the capture of many stories.  In communities that are oral by nature, the telling of stories is a regular practice that helps build people up; this was preferable to relying purely on closed-ended survey questions that can be experienced as mainly extracting data for external means.

Approach

World Concern selected two OVT program sites—one in Uganda and one in South Sudan. Due to limited monitoring data, participant selection relied on World Concern staff using predefined criteria:

• Four focus groups per country, segmented by age and gender, using participants not involved in household interviews
• 24 household interviews in each country, covering a range of program involvement
• Inclusion of participants with diverse experiences, outcomes, and demographics (age, gender)

As per the protocol, each interview guide included open-ended questions by program domain, followed by closed questions assessing perceived change over a four-year recall period (Better, No Change, Worse, Not Sure), and attribution questions on causes of change. Although the OVT program began in Uganda five and a half years ago, a four-year recall period was applied consistently across both countries to ensure comparability and reduce recall bias. Due to implementation logistics, full blindfolding was not feasible. Respondents were informed that the study was commissioned by World Concern, though the interviews were conducted by an independent team. This may introduce some response bias.

Domains were identified using each village’s unique strategic plan, an essential part of the OVT approach. The guides differed slightly between countries to reflect local programming, between them covering:

• Household Health
• Education
• Spending and Saving
• Agriculture and Food Consumption
• Relationships
• Leadership
• Wellbeing

All interviews included an informed consent section, demographics, external relationships (identifying other organizations active in the area), and interview observations. Focus group guides were adapted from household instruments.

Interviews were audio-recorded, transcribed verbatim by a professional firm, and annotated with interview notes. Informed consent included disclosure of the study purpose, voluntary participation, and recording consent. Focus groups were conducted in public settings.

Each day, recordings were transferred to secure storage, and metadata summaries were created, documenting context and any issues. Transcripts were reviewed for accuracy and consistency with audio recordings. Interview notes were scanned and uploaded for analysis. Quality control involved review by a non-field team member to ensure QuIP protocol adherence and instrument refinement.

Data analysis

To minimize bias, the analyst conducted coding independently from the field team. Transcripts were exported from MS Word to Excel, then imported into the analysis software Causal Map. The analyst reviewed each transcript and guide before coding all statements as causal claims, labeling factors (drivers/outcomes) inductively. Causal maps were generated per source and aggregated to identify themes. Analysis began with South Sudan and included:

• Top 20 most cited causal links
• Most frequent drivers and outcomes
• Domain-specific causal pathways

These findings were compared against World Concern’s logic models and the researchers desk review to assess program efficacy and identify gaps or unintended outcomes. Participant lists provided by World Concern were cross-referenced to confirm attribution to OVT programs. When savings groups were mentioned without specifying OVT, participation was inferred based on these lists.

Maps include color coding for what participants viewed as drivers (purple) and outcomes (green), depending on map orientation. Numbers in boxes reflect the number of sources citing a factor; numbers on arrows indicate the number linking a driver to an outcome. Zeroes indicate absent causal links in that map. Closed question responses were summarized as percentages. Each focus group was treated as a single source (total N=28). See below for an example output from Causal Map. [Note from Bath SDR: Although this is clearly explained, we advise caution on using percentages with such small sample sizes, consider using simple response numbers in charts instead. We also don’t advise mixing individual interviews and focus groups in aggregate maps to avoid confusion about the relative value of each source].

Country-Specific Results

Uganda

Interviews in Uganda revealed widespread positive change across all major domains—health, agriculture, finances, relationships, leadership, and wellbeing:

• Leadership: 75% of respondents reported improved leadership engagement, citing more transparent and proactive leadership. However, concerns remained about trust in group leaders and negative behaviors like drunkenness.
• Wellbeing: 82% of respondents reported improved wellbeing over the past four years, and 93% expressed optimism for the next four years. Spiritual practices, especially prayer, were cited by 50% as key drivers of wellbeing.
• Health and Hygiene: 80% of respondents reported improved household health, linked to better hygiene practices, construction of pit latrines, and more diverse diets from improved farming.
• Spending and Saving: 71% of respondents reported increased purchasing power, attributed mainly to participation in OVT savings groups.
• Agriculture and Food Consumption: 75% of respondents reported that they experienced increased food harvests, primarily due to improved farming knowledge, such as line planting.
• Relationships: 75% of respondents noted stronger community cooperation, and 80% felt gender relations had improved, with credit given to the OVT Gender Champion and savings group initiatives.

South Sudan

Respondents in South Sudan also described notable improvements across most domains, with some areas demonstrating particularly strong outcomes:

• Wellbeing: All respondents (100%) reported improved wellbeing, and 86% anticipated continued improvement in the future.
• Health and Hygiene: 96% of respondents reported improved health, attributed to better access to healthcare, WASH education, and borehole construction facilitated by OVT.
• Education: 89% of respondents noted improved household education, due in large part to increased school access through partnerships with World Concern.
• Spending and Saving: 89% of respondents said they could afford more goods and services than four years ago, though only 54% felt increased control over money. Membership in World Concern savings groups was a key driver of financial change.
• Agriculture and Food Consumption: 82% reported more food harvested for home use, 86% for income, and 93% said their household consumed more food. Nearly all respondents (96%) noted increased dietary diversity. The introduction of ox ploughs—linked to World Concern—was credited with expanding cultivated land. However, some reported setbacks due to flooding.
• Relationships: 96% said community cooperation improved, with communal agriculture and savings groups playing a central role in fostering social cohesion.
• Leadership: 82% reported more effective community engagement by leaders, primarily through improved communication, frequent meetings, and greater responsiveness to community input.

Use of findings

World Concern has benefited greatly from the QuIP study and has put its findings to use in various ways. Findings have been shared internally with teams globally and have shaped the way OVT and other programs are designed and implemented, seeking to address gaps that were identified while also building on what was shown to be working well. Some of the strengths highlighted in the report have helped World Concern hone its understanding of their competitive advantage in the industry and improved World Concern’s ability to share an evidence-based story of impact. A summarized version of the report has been made available on the World Concern website, allowing supporters, potential supporters, partners, and other stakeholders to learn more about the OVT model and its impact in these communities.

EIG Insights is a research consulting firm providing actionable insights to mission driven organizations globally. EIG Insights specializes in the development of Theories of Change, Measurement Frameworks and Program Impact Evaluations for non-profit organizations. Edessa Research Consultants were contracted by World Concern to carry out the original QuIP study. In 2024, Edessa Research Consultants became part of EIG Insights. Contact: kristen[at]eiginsights.com

World Concern is a Christian non-profit serving underdeveloped communities in Africa, Asia, and Haiti. Its One Village Transformed (OVT) program uses a holistic, six-year approach to identify and build on community needs and strengths—physical, spiritual, and social. Contact: katiet[at]worldconcern.org

• Share: