Impact Evaluation: A force for doing good, better

Organisations in the third sector unite under the canopy of doing ‘good’. As the current landscape shifts towards evidence-based accountability for funders and beneficiaries alike, people are asking:  

“How can we do good, better?”

The question poses challenges and opportunities. There are practical, concrete steps organisations can take towards doing good, better – one of which is impact evaluation.

What’s impact evaluation?

Impact evaluation uses a methodological approach to collect data, assess actual outcomes of programme activities, and attempt to explain why (or why not) impact is generated. 

Why impact evaluation?

Impact evaluation is a win-win for several reasons, including:

• Validating current impact and organisational success
• Increasing accountability and credibility to funders and beneficiaries
• Providing insights for more effective programme design and delivery

Interesting, but how? 

Some organisations choose to evaluate impact relying solely on internal resources. This can be time intensive, require specialised skills, and risks not being perceived as credible by key stakeholders, such as funders. Hiring an independent evaluator is one way organisations can ensure ease, quality, and credibility in the process of evaluating impact. 

Collaborating for impact evaluation

Earlier this year, the Masonic Charitable Foundation (MCF) commissioned Renaisi to conduct an impact evaluation for their Charity Grants programme. We combed through the data of more than 270 grant recipients, evaluating outcomes in relation to MCF’s theory of change, and produced a 33-page report as well as a presentation summarising key insights for internal stakeholders.

MCF and Renaisi have collaborated on impact evaluations in the past. This time would be different both in terms of the method used and – more interestingly – the overall strategic purpose.

Moneer Moukaddem, Associate Evaluation and Learning Consultant (Renaisi) and Project Manager for this project, sat down with Jo Haffenden, Evaluation Manager (MCF), for a conversation on impact evaluation following their recent collaboration.

1. Moneer: From your experience, why may an organisation want to evaluate their impact? 

Jo: You will often find impact practice seeks to address three principles: accountability, learning, and decision-making. I’ve noticed organisations may give different importance to one principle over the other.

Accountability is important to fulfil responsibilities to funders/donors and others, such as colleagues and grantees. Impact evaluation helps not only demonstrate the services that organisations intend to deliver, but uses evidence to show that services are meaningful, important, and successful in bringing out the intended outcomes for the people they intend to serve. This sends a strong message to funders/donors that our activities are worth investing in, and empowers us to continue to attract funding with confidence. Among other stakeholders, accountability helps to convey a message around transparency and how they contribute to the organisation to achieve its purpose.  

Learning – I’ve also observed that organisations might want to use impact evaluations to learn and reshape their practice. Charities want to learn about themselves, the services they provide, and to whom. Ultimately, they ask, “How can we do good, better?” More and more charities and funders are opening up to sharing lessons learnt. Hubs and spaces have been created for organisations to share their journeys and explore how to approach systemic change for the groups/beneficiaries being supported. 

Decision-making – The aspiration is that insights resulting from impact evaluation help foster a culture of continuous improvement and innovation, and that an organisation’s strategic direction will reflect these learnings based on evidence.

2. Moneer: Throughout our collaboration, you were passionate about both evaluation ‘of impact’ and evaluation ‘for impact’. Why? What’s the difference?

Jo: Evaluation ‘of impact’ is about assessing the outcomes of an initiative or support intervention. Evaluation ‘for impact’ places the emphasis on using evaluation for decision-making and improving programme design/delivery. 

Throughout the multi-year analysis conducted by Renaisi, we focused on understanding the short-term (1-3 years) impact of our Later Life and Early Years funding under our Charity Grants programme. As a starting point, we referred to our Theory of Change (developed in 2018), which among other areas, highlights the short-term impact on beneficiaries and charities we could expect from our support. 

As the insights from the impact evaluation started to emerge, we accessed new understandings of our programme, which generated good questions: about the framework itself, underlying assumptions, data collecting, and operational delivery of the programme.

Tapping into the new insights raised more questions about lessons learnt and what questions we might want to consider for decision-making and reshaping the new strategy of the Charity Grants programme and considerations for the Impact & Evaluation team.       

3. Moneer: How might Renaisi’s impact evaluation of MCF’s grant activities have broader strategic relevance for your organisation?

Jo: It’s very easy to make assumptions on the positive impact of an intervention. Originally, we had a huge amount of qualitative data that had been gathered as part of the operational delivery of our programme, which had mainly been used only for operational delivery and to assess payment of the next instalment of an award. Since MCF first started awarding grants in 2016, this data had never been analysed at the aggregated level. This meant we didn’t really know – nor had robust evidence of – the impact of this part of the programme. This has now changed, we hope.

As mentioned previously, we used our Theory of Change as a framework to conduct the impact analysis. The analysis used additional lenses and helped us gain further understanding of our programme. For example, Renaisi provided insights by comparing our large and small grant outcomes, in addition to identifying emergent outcomes not originally captured in our Theory of Change.

The MCF is generating proportionally robust evidence to help inform-decision making. For example, we are developing a new Theory of Change for our charity grants programme, for which we’ll be making use of the insights from the evaluation. 

In an effort to become an evidence-led funder, we also asked Renaisi to provide baseline data so that we can track change, if needed. Although we don’t know exactly how we will use it, I know that this evaluation will provide the building blocks for what comes next. 

4. Moneer: What advice would you give to organisations who are curious about impact evaluation but unsure of its value?

Jo: For organisations who experience lack of awareness or knowledge about the value and benefits of evaluation, I have three pieces of advice.

Put learning at the heart of your impact evaluations. This will promote organisational behaviours to move away from language alluding to ‘failure’. I remember an interview with a tech leader in McKinsey, who always started the monthly staff briefing with ‘Mis-TAKES’ – or something they had learnt – thereby setting the tone and encouraging the workforce to try and learn from experience for the sake of improvement. So I suggest doing the same when conducting evaluation! Approach it from the ‘discovery lens’.  

Adopt a clear plan on why you are trying to learn from an evaluation. I always advise to start with the strategy: make sense of your programme, how you think it pieces together, some areas you might be able to influence but perhaps others less so. By taking a strategic approach, you will be able to see the full picture of what you’re ultimately trying to achieve. Subsequently, your evaluation will be executed with a defined purpose. Whatever you learn outside this will be an added value. 

Impact evaluation will unleash new knowledge – so remain open to that learning! A growth-mindset is key, and don’t keep the new learning to yourself. Bring people along with you; share examples of your vision. If you have some data; what this might mean for the programme? When facing resistance, it is ok to create some disruption for innovation too. As I always say, ‘you don’t make an omelette without breaking eggs’.