Developments in QuIP coding
25 June 2019 | News|
We never stand still at BSDR, and the last few months have been all about reviewing the approach we take to coding our data! This short blog will summarise some of these conversations and link to a new resource for QuIP analysts to understand the implications of any changes for coding QuIP data.
The QuIP approach to coding is fundamentally about looking for stories of change and documenting causal claims in interview transcripts. We are specifically looking to document the relationships between reported cause and effect; what do respondents say caused reported changes? Summarising this data enables us to see a big picture causal network of the many variables which occur in often non-linear relationships between inputs, outputs and outcomes.
As QuIP coding developed, we focused on coding drivers and outcomes in the text, as well as allocating attribution codes. We have now moved away from defining the tags created through inductive coding as either drivers or outcomes, instead using the analysis software to determine what is a driver or an outcome based on its position in the causal chain. This gives the analyst more freedom to use the same descriptors/tags as either drivers or outcomes – depending on how different respondents document a change story. A starting point for some respondents may be a mid-point for others, so forcing one event or input to be a ‘driver’ sometimes creates an artificial distinction.
- A tag followed by any other tags in the same row of text is now automatically be considered a source, or Driver of change. It is considered to have a causal relationship with the tags following it.
- Any tag at the end of a row of tags, with nothing following it, is automatically be considered a target, or Outcome.
More information on this change, including a change to the type of tags we use, and the codes used for attribution coding – is contained in a briefing note for QuIP analysts. (Please note that this is aimed at analysts who are already familiar with QuIP coding, this is not a step-by-step guide to coding.)
In addition, we have been thinking very hard about how to code the type of relationships between cause and effect, and how to gather more information about necessity and sufficiency; what packages of ‘drivers’ and other contextual factors are either necessary or sufficient for a particular outcome to occur. For more on this, please read Rick Davies’ recent M&E News review of our book, Attributing Development Impact, where he goes into detail about some of the discussions we have had thus far on the potential to delve further into QuIP data – including using a Confusion Matrix. Rick makes the important point that it is currently not possible to use the dashboard to automatically identify “instances of ‘configurational causality’… packages of causes that must be jointly present for an outcome to occur.” This is an exercise currently communicated in narrative form by skilled analysts, using network visualisations to help evidence claims. Automating this task is a much harder nut to crack; this is not a purely theoretical or mathematical discussion about which numbers to use – but has to be considered in the context of how messy qualitative data can often be, and the extent to which we can apply mathematical formulae to semi-structured responses.
For more on this, read a recent American Evaluation Association blog – Measuring Necessity and Sufficiency by Ryan Watkins (which cites Rick Davies’ work on EvalC3).
QuIP does not use a control group to isolate differences between those who received an intervention and those who didn’t, but relies on self-reported attribution to establish causal pathways. At a theoretical level, the idea of using that data to establish whether an intervention is ‘necessary’ or ‘sufficient’ for change to occur is very appealing, but we have to be very careful how much such data is quantified and potentially taken out of context (see our paper on how to use QuIP numbers and visualisations for more). Caveats notwithstanding, we continue to explore the potential to code more efficiently, including new software (working with Steve Powell of Theory Maker). One feature would be the ability to code stories in pairs rather than chains (‘x leads to y’ and ‘y leads to z’). These pairs can then be put together to create larger network maps, whilst allowing for one pair in a chain to be negative, and another positive – reflecting the real complexity of non-linear life!
We’ll keep you updated on future developments – and look forward to more detailed discussions on coding! Join the discussion on twitter – @bathsdr