The why, how and what of impact measurement

Awdur: Dr Lucy; Amser Darllen: 10 munud
Mae hwn yn adnodd agored. Mae croeso i chi ei gopïo a'i addasu. Darllenwch y telerau.

This blog is the second of a two part series following on from a set of workshops carried out for Catalyst and The National Lottery Community Fund COVID-19 Digital Response grantees on the subject of impact measurement. Bobi Robson’s first workshop and blog opened up our thinking about ‘why’ we should measure impact. This blog talks in detail about the ‘how’ and ‘what’: it summarises Lucy’s workshop and is packed with take away tools to get going with impact measurement.

The why-how-what (Golden Circle)

Most people and organisations I have supported with understanding their impact start with ‘what’. The most commonly asked question is usually something along the lines of ‘what tool do I use to measure this for my funder?’ Every time I respond with ‘well, it depends what you want to know and why you want to know it?’ However, if tools are what you need right now (I’ve been there, and I get it!), then here’s a link that can set you off with a quick hit of tools. But before you go, why not read on and get more context – I promise to give you the link again further on in this post for you to click through to! 

What we’re clearly doing here is adopting Simon Sinek’s now popularised approach to starting with ‘why’. His focus on leadership is equally applicable here in our approach to impact, as this is about creating an impact and learning culture that is proactive and strategic, rather than reactive to, for example, funding or commissioning reports. Otherwise, there’s a bit too much tail wagging the dog going on for me!

Funders are an important stakeholder and influencer, but there are other stakeholders and influencers that can come together to lead a culture of being impact focused – and I don’t mean that in as corporate a way as it sounds! I mean, our services being well thought through, justified and evidenced from theory and practice, and clearly articulating why we do what we do, how we approach it and what we actually do.

This is a purposeful and meaningful approach to impact that isn’t an afterthought or quick feedback form and scramble to gather data, to appease a funder (I think a lot of us have slipped in to this, as we are so busy on the ground with the complexities of our services and projects and our ongoing struggle with resource). This is exactly why it is a leadership priority and exactly why it’s about creating an impact culture – threaded throughout governance, leadership, and practice. 

Bobi’s workshop and blog asked us to back up a little and first consider what are our questions (i.e. what do we want to know and why do we want to know it?). I’m convinced many of us are collecting data just because we always have, or because we are told to, without being purposeful and considered about it – an approach which would lead to more meaningful data that we can do something with. 

From ‘WHY focus on impact’ to ‘HOW to approach it’

Here’s where we can further challenge our reasons and rhetoric! What if we replaced ‘measurement’ with a term like, ‘investigate’ or ‘explore’? For me, these feel a little more inclusive, welcoming and maybe even engaging, rather than something done by the research elite, or indeed the tones of managerialism that some have come to fear. My quest here is to demystify impact under these parameters and make it a part of our culture. Stripping out the academic and sometimes inaccessible language, and replacing it with something more engaging and meaningful. 

I call this a process of Awareness-Choice-Action (ACA) that supports people to become more aware of themselves and their context, realise more choice, and take action. This, more empowering process, enables people and organisations to have agency and thus to take more meaningful decisions and actions in exploring their impact. 

This shift in approach opens the door to understanding, knowledge, and development (i.e. improving as well as proving). This is particularly important in the innovation and development world that readers may have been engaged with through their relationship with Catalyst. It encourages us to consider impact in iterative cycles of researching the context and needs, innovation and development to meet these needs, and evaluating the contribution of our development to these needs. This aligns with the design thinking methodology that Catalyst advocates, and which has been detailed by  some great thinkers in this space, such as in Ann Mei Chang’s Lean Impact. 

I come from a youth development background and the work of The Centre for Youth Impact (CYI) has changed the landscape of the way we approach impact. Their ‘Asking Good Questions’ framework provides an approach that is much more helpful than asking ‘does it work?’. 

They state, “…it is more powerful than saying ‘we know it works; we just need to demonstrate it’”. They further qualify, “‘Does it work?’ is actually a collection of other questions, bundled up together, all of which matter for different reasons”. The first two questions are about understanding context, and designing a high-quality ‘offer’ that responds to this context; questions three and four are about monitoring the delivery of activities; and questions five and six are about listening and responding to feedback. 

Theory of change at the heart of impact 

At the heart of any impact discussion is your theory of change (ToC). From a broader social sector perspective NPC (New Philanthropy Capital) are the go to think tank and resource for anything social impact related. They define theory of change as a process that encourages us to reflect on our aims and plans, to discuss them with others and to make them explicit, and see this process as the foundation of strategy, evaluation and communication.

We can break ToC down into the following areas based on the works of both CYI and NPC:

• CONTEXT – the needs, issues, assets, stakeholders, policy, funding, etc. 
• IMPACT – the contribution our organisation/service/project makes to this context
• OUTCOMES – a breakdown of indicators of impact that participants achieve 
• MECHANISMS OF CHANGE – the conditions and experience we create to enable participants to achieve outcomes
• QUALITY – how we ensure quality and consistency across our offer so as participants can benefit from the mechanisms of change.

The broadest questions we might have in relation to this are:

• Are we working with the target participants identified in the context? 
• Do we achieve the intended impact? 
• Do participants achieve the intended outcomes? 
• Can we attribute these outcomes to our programme? Which mechanisms? 
• Did we achieve this consistently well?

So, we can consider ToC as a framework for how we approach investigating impact. It helps frame our questions so we can plan how we investigate these in practice. An impact plan is the bridge between ‘how’ and ‘what’ we actually do – i.e. data collection. 

From HOW to WHAT: from theory of change to data collection

Again, we need to challenge the rhetoric around the term ‘data collection’ so as it is engaging and meaningful. This starts with an understanding that there are lots of types of data and a holistic approach that values multiple types and sources of data is helpful. In its simplest form, I would always say ‘no stats without stories and no stories without stats’. Academics call this ‘triangulation’ and some call it ‘crystallisation’ (depending on your approach), but ultimately what we’re talking about here is not thinking we can use one source of data to understand (prove/improve) our impact. 

NPC again provide a great resource for us to draw from as they describe 5 types of data in relation to ToC: 

1. User data:

• Asks is your [service] effective at reaching the intended target group?
• Establishes the characteristics of your service users.

2. Engagement data:

• Asks how effective is your service at continuing to engage your target service users?
• Establishes the extent to which people use your service and how they use it.

3. Feedback data:

• Asks what people think about the mechanism of change within your service?
• Establishes what mechanisms help people achieve outcomes and how.

4. Outcomes data:

• Asks how have people been influenced or helped by your service in the short-term?
• Establishes the immediate resources, benefits or assets that your users gain from the service.

5. Impact data:

• Asks have the outcomes achieved (above) helped people to change their lives for the better?
• Establishes the long-term difference achieved for individuals, families, and / or communities.

Creating an impact plan

These 5 types of data can form the sections of an impact plan. Here’s a favourite example I like to use (assumes permission, privacy and data protection are in place):

No really, WHAT do I actually do?

As I promised at the beginning, here is a link to the companion website of a book I co-authored on creative research methods for practice. There are links to useful data collection templates and tools (plus for the links in the table above, see journey maps here and the case study template here) – each downloadable with an overview, description of what you’ll need, a step by step process, how to analyse your data, and strengths and weaknesses of the tool. 

What about return on investment? 

Lastly, a question I’m often asked is what about cost benefit analysis and return on investment? Warning: This is a complicated and contested field! Social Return On Investment (SROI) has made inroads in developing this area and NEF (New Economics Foundation) are one of many experts. They state that SROI is an outcomes-based measurement tool that helps organisations to understand and quantify the social, environmental and economic value they are creating. This is a participatory process which they describe as capturing in monetised form the value of a wide range of outcomes, whether these already have a financial value or not. An SROI analysis produces a narrative of how an organisation creates and destroys value in the course of making change in the world, and a ratio that states how much social value (in £) is created for every £1 of investment. 

I warned you it is complicated! However, one thing that might help this is what the starting place is… yep, you’ve guessed it – theory of change! 

So what next?

Here’s some next steps you could take:

• Draft your ToC and test it with representative stakeholders (co-creation)
• Evidence your ToC – ensure why you do what you do is justified and evidenced
• Create questions about your ToC
• Create an impact plan – consider sprints to focus resource
• Decide data collection – tools and analysis
• Ensure this links to any current monitoring systems and processes

Lucy has a PhD in wellbeing development and is a freelance consultant and coach supporting people and organisations with social purpose towards greater impact. You can connect with her on LinkedIn. 

‍

Photo by Andrea Piacquadio from Pexels

Wedi'i gomisiynu gan Catalyst