Contribution Tracing: A Brand New Evaluation Approach to PROVE Your Programme's Impact
Perhaps one of the most popular buzzwords in International Development right now is “Impact”. In the past organisations only had to brag about the number of schools they built and the number of children that were enrolled. No mention of whether these children learnt anything or if they were actually empowered through education to rise out of poverty.
Nowadays taxpayers, whose hard-earned cash fund development programmes, want to know whether an intervention actually made a lasting difference in people’s lives. This is what is meant by ‘impact’.
Impact is what happens long after the schools are built, after the children graduate and even what happens after these children become highly educated adults able to provide for their own children, and in so doing, effectively breaking the cycle of inter-generational poverty.
However, measuring impact can be challenging. After all, you are talking about results that are only seen in the future, sometimes years after a programme or intervention ends. Even if you see results, the challenge is to prove that it was actually your intervention that contributed to the change that is now observed. No wonder some organisations just show the number of schools that were built! That is easier to prove as there is no dispute as to who built them.
Furthermore, many interventions (such as policy influencing) don’t produce outcomes that materialize right away. Imagine an organization that spent months lobbying for the rights of children in a political environment that places little value on the lives of children. Their national government that was so resistant to a child protection policy, finally adopts it.
Can that organisation claim that they contributed to this change of heart by the government? Of course they can. Anybody can claim whatever they like. The more interesting question is whether the organisation can test the claim to prove (or disprove) that they contributed to the adoption of the child protection policy. The answer to such a question is ‘Yes’. A “contribution claim” can be tested with the use of “Contribution Tracing”.
What is Contribution Tracing?
“Contribution Tracing (CT)” is a rigorous quali-quantitative approach that is used in impact evaluations to test the validity of claims. In other words, this approach brings a bit of ‘science’ to determine how confident you can be about a contribution claim.
CT allows you to gather evidence that supports (or is against) your contribution claim. Once all the evidence is gathered it can then be analysed using mathematical formulas (Bayesian updating to be exact). This puts a numerical value on your level of confidence in a particular claim.
For example, after applying the CT approach, the fictitious organisation from earlier may be able to say with 95% confidence that they are reasonably certain that their lobbying efforts were responsible for the government adopting that specific policy. With CT, there is actually a calculation for how the ‘95%’ was derived.
Effectively CT allows you to do two things: test your contribution claim (is it valid or not?); and, if it is found to be valid, determine, quantifiably the level of confidence in your contribution claim. The CT developers call it, “putting a number on it.”
Contribution Tracing allows you to prove that the outcome was not just chance.
History of Contribution Tracing
The approach is developed by Barbara Befani and Gavin Stedman-Bryce. Though the approach is brand new and was recently published online by the international journalEvaluation, CT is based on a very well-known and established approach called Process Tracing (which has been in use since the 1940/50s). CT is innovative and marries the principles of Process tracing as well as Bayesian Updating. The approach was presented at the 2016 UK Evaluation Society Conference and was received with much enthusiasm.
How is Contribution Tracing applied in practice?
CT is a participatory evaluation approach that involves consultation with all relevant stakeholders through a series of steps. Some of the steps happen in parallel and don’t necessarily flow in a linear order. For example, steps 1 and 2 would be conducted at the same time. Likewise steps 2 and 4 happen in parallel.
Step 1 – Making the Claim
The first step is all about making a claim, or claims. Using the example from earlier, the claim is that the lobbying efforts led to the government adopting the child friendly policy.
Step 2 – Develop a Theory of Change
Once the contribution claim, or claims have been stated, key stakeholders in the programme must be able to explain what makes them believe that what they did (their activities, etc.), contributed to the changes that they claim. This is most easily achieved by revisiting an existing theory of change or developing one retrospectively if one doesn’t exist.
For example, they could state that before their lobbying and advocacy efforts began the government refused to even acknowledge the rights of children in the country. It was only after the campaigns started that the government recognised children as having rights. Additionally, there is specific clause (on child marriages) within the newly adopted child protection policy that only their organisation was pushing for. These are things that make the organisation believe that their lobbying and advocacy efforts may have worked.
CT is based on a mechanistic model of causality, so a clear theory of change is a great way of stakeholders from a programme or project describing how they believe their actions led to the changes they are making a claim over.
Step 3 - Hold a ‘Contribution Trial’
The staff of the organisation, other experts, stakeholders and the evaluator(s) gather for a symbolic ‘contribution trial’to discuss what evidence would prove, and disprove, the existence of the mechanism they jointly developed in step 2; and hence prove or disprove their claim. This is theevidence they expect to see.
For example, an evaluator would expect that most stakeholders involved in the campaign, believe in the contribution claim they are making. Given that the probability of finding such evidence should be high, not finding it, weakens the evaluators confidence in the claim. This process would be repeated for each link, or component in the theory of change. That is, identifying evidence that proves empirically whether the claim exists in reality, or not.
If no evidence is seen for the claim, it can be said with reasonable certainty that the lobbying and advocacy actions are not working in the way as thought. This means that something else must have caused the adopting of the policy by the government.
However, if they do see the evidence they expected to see for the claim, then they know that the claims they are making are true.
By this stage the staff would know that if the claim is false; or if the claim is true. However, there could still be another reason for the government to adopt the policy that the organisation did not think about. Finding ‘expect to see’ evidence means that the claim survives, that it is a plausible cause of the observed change, but it may not be the only cause.
Step 4 – Identify alternative causes (which would have been identified in Step 2 when constructing the Theory of Change)
All the stakeholders would work together to think about all the possible things that could also have caused the government to change course.
It could be that there was an application to host the 2024 Olympic Games and in order to increase the chances of winning the bid to be the host country, the government wanted to show the world that they take the rights of the child seriously.
Once ‘Good Impression on the International Olympic Committee (IOC)’ is identified as an alternative reason for a change in government’s stance, the process highlighted at Step 3 is repeated, using this as another claim.
In other words, there would be a discussion on the evidence that one would expect to see to support the claim that the ‘Good Impression on IOC’ was responsible for the new policy now being in place.
From Steps 1-4, the organisation will know if their lobbying and advocacy actions influenced the government. They would also have a better understanding of what else is happening to cause the government’s change of heart.
Step 5 – Apply Bayesian Confidence Updating (putting a number on it)
The final step (which is optional) is to put a number on the confidence of the claim. The goal of the CT approach is ‘not to assess impact in terms of net change, but more specifically to assess changes in our ‘confidence’ that an intervention had impact' (Befani and Stedman-Bryce, 2016). This is where the ‘science’ comes in.
In more technical language, Bayesian confidence updating is used to quantify our post-observation confidence after finding specific pieces of evidence in relation to a contribution claim.
What are the Strengths of Contribution Tracing?
One of the advantages of this approach is that it tells you, as the evaluator, what to look for and if you find it, it tells you what it means in relation to your claim. In other words, what data to collect and the potential sources of this data.
CT only looks for the most powerful evidence. No time is wasted asking people questions, when it is already known the answer won’t be of use. The ‘right questions’ are asked of the ‘right people’ and the ‘right evidence’ is sought in the right places. The developers of CT call this “evidence with the highest probative value”. Probative value is the power of specific items of evidence to increase or decrease our confidence in a specific claim (Befani and Stedman-Bryce, 2016, p.2).
CT as a result ‘forces’ organisations to refine Theories of Change to make the statements more precise and testable. In other words, CT is not suited for Theories of Change that are vaguely expressed, unfalsifiable and have untestable claims.
Due to how specific a contribution claim needs to be, application of CT actually increases the conceptual precision, clarity and scientific quality of Theories of Change.
Additionally, a major strength of CT is that it brings a certain robustness to qualitative methods and research. Within the scientific community, the use of qualitative methods for impact assessment has traditionally been considered subpar to quantitative methods because of their low internal validity and replicability. CT allows for an organisation to use mathematical probability to back up their qualitative contribution claims.
CT also minimizes the confirmation bias of only looking for evidence that supports one’s claims. This is done by the inclusion of ‘critical friends’, during the ‘contribution trial’ who may represent other plausible sources/explanations for the observed change.
This leads us to the key strength of the CT approach – its participatory and collaborative nature. Unlike some other evaluation approaches that have an external evaluator coming in, collecting the data, analyzing his or her finding and then presenting it to the organisation, CT involves the stakeholders at every step of the evaluation.
The stakeholders are involved in making the contribution claim, determining what evidence to look for, testing the evidence and calculating the confidence levels of a contribution claim (all with the support of someone who understands the approach).
What are the potential challenges of Contribution Tracing?
Though CT is remarkable, there are potential pitfalls that have to be managed to derive the most benefit from the approach.
A major issue to consider is that the technicality of the mathematical calculations may pose a challenge for programme staff who are not mathematically inclined. As such, an external evaluator who is versed in mathematical analyses may have to be brought in to aid an organisation with Step 5 of CT. However this step is optional and only for those organisations who want, or need to put a number on their confidence.
Furthermore, not every one may be enthusiastic about quantifying confidence and probabilities in qualitative research. Rather than doing this, they may prefer to just use an evaluation method that is purely quantitative and not a hybrid.
Another challenge is that CT may not be so useful at answering some questions. For example how a programme compares with other programmes. In this approach you need to be clear about the questions that should be answered before the evaluation starts. To overcome this challenge you may have to combine CT with other evaluation methods, like Qualitative Comparative Analysis (QCA).
In order for CT to be effective, the timing needs to be just right. As the name of the approach suggests, the word ‘tracing’ means that you are looking for the traces left behind by a programme. The programme needs to have been working for long enough for those traces to be ‘visible’. To help with this, you should always make sure that the change you are interested in has actually happened. If it hasn’t happened yet, you may be able to look for another change that happened earlier, such as one lower down your theory of change.
Additionally, you must spend the same time and resources on exploring other potential causes as you do on investigating your own programmes. If you don’t do this you can end up with a limited view of how the change happened.
In sum, Contribution Tracing is a brand new and an innovative approach that may revolutionize the way impact assessments are done in the future. Harnessing the power of Bayesian probability, Contribution Tracing gives organisations the ability to scientifically back up their claims that their interventions had an impact; and steers the evaluator to find conclusive proof.
With Contribution Tracing others can take your word for it!
If you are interested in learning more about the approach you can participate in the Contribution Tracing workshop to be held on 11-13 July 2016 and hosted in London. Facilitators are Gavin Stedman-Bryce and Barbara Befani, developers of Contribution Tracing.
Visit https://www.pamoja.uk.com/training/course/Contribution-Tracing for more information.
What are your general thoughts about Contribution Tracing? Feel free to leave a response in the ‘Comments’ section below.
Thanks Gavin for giving me an exclusive interview so that I was first in presenting this fresh and brand new approach to my readership.
Reference
