This blog has been co-authored by Graeme Sinnott (Active Partnerships), Dr Katie Shearn (Sheffield Hallam University), and Darcy Hare and Chad Oatley (Sport England). The intention is to help thinking and conversations around measurement to keep progressing, locally and nationally. To stimulate discussion between people within and across places. To encourage innovation and connections.
Where is the measurement conversation currently?
In March over 150 colleagues gathered to explore ‘What is relevant to measure and how do we do it together?’ when we are interested in mobilising whole-systems approaches around physical inactivity.
The session was a space to explore what we are learning about measurement methods and principles that are sensitive to the complexity of addressing physical inactivity and the inequalities within this.
From discussing the session with colleagues, some were fascinated by the ideas raised. Some loved it. Some were put off by the perceived difficulty of it. Some struggled with the language. Some are now adopting the ideas. Some are now asking for support to identify what next.
The range of feedback is reflective of the diversity of perspectives and understanding of complexity friendly measurement. Reflective of the diversity of how each locality is approaching it. And this diversity is an asset. An opportunity to learn with and from each other around something that we are all grappling with.
The uniqueness of each place and varying readiness for adopting a whole systems approach to physical inactivity means that there isn’t a single measurement framework or blueprint that we can, or should, all copy. And if people are not taking a whole systems approach, then the immediate necessity to adopt new measurement methods is perhaps not as acute.
What appears to be emerging however are some common principles that could help us all to measure and learn together about our work in a complexity friendly way. Principles that may provide us with enough clarity for now to help us all transition into a world where our measurement helps us to meaningfully understand how we create the conditions for an active nation and the impact of our efforts. To improve, rather than prove.
By way of introduction to, and context for, the principles, we have sought to provide further explanation to the common questions of What is relevant to measure, why, and how could we do it differently when adopting a whole systems approach to physical inactivity?
What is relevant to measure and why?
If systems change is about collectively shifting the conditions holding the problem in place (the ‘problem’ being physical inactivity and the inequalities within this in our case), then the primary purpose of measurement in this context should be to support iterative learning, improvement and adaptation. Accountability should therefore be focused on this, rather than on the delivery of outputs and outcomes.
Targets linked to outputs and outcomes are often arbitrary and can drive gaming and tunnel vision. And measuring outputs and outcomes in isolation doesn’t reveal anything about what value an organisation or specific activities might be adding.
Of course, outputs and outcomes remain important. But systems produce outcomes, not individual organisations. And factors beyond our direct control will be influencing the outcomes we seek. It is our job to keep identifying and addressing those factors. Creating relationships that enable others to address those factors.
What is important to a Director of Public Health (e.g., reduce hospital admissions), might be different to what is important to a person involved in the work day to day (e.g., innovation and credit), and might be different to someone benefiting from the work (e.g., improved wellbeing).
As individuals with different underlying ideologies, we value different things. We often have different motivations for being involved in the work, hence differing views on what outcomes are important to us as demonstrated above. Different perspectives can provide us with valuable insight into the underlying reality of things.
We need to engage with and seek to understand different interests and perspectives. Not necessarily seek to aggregate and synthesise into a single view if the primary purpose of our measurement is to support iterative learning, improvement and adaptation. One of our tasks is to bring people together and understand how they can support each other to achieve shared goals. The focus of our measurement is on understanding and driving progress towards this, respecting and responding to the different priorities of people involved.
What is relevant to measure?
- What systems conditions are holding the problem of physical inactivity in place?
- Why are the system conditions the way they are?
- What can we do to shift the conditions?
- Why do we think this will make a difference?
- Have the conditions shifted?
- Have our actions contributed to the conditions shifting?
- What else might have contributed to the conditions shifting?
- What unintended consequences can we see as a result of our actions?
- Physical inactivity and the inequalities within this are complex. Social problems like this are influenced by a range of factors beyond the control of any single organisation. They do not respect organisational boundaries and can’t be fixed by individual interventions alone.
- We therefore have to operate and measure our work in ways that complement this context and support us to understand how we progress to achieving our shared goals.
How could we measure differently?
There is consistency emerging in conversations around the ‘what’ and the ‘why’ for our measurement, as outlined above. There is less consistency regarding the ‘how’.
There is a diverse mix of methods being adopted in places. Methods that are developed in the place, by the people doing the work. Innovating around things such as how to capture and make sense of changes to system conditions. How to capture and make sense of stories from people in the work. How to share and build momentum around what appears to be helping to affect change and stop doing what is not.
And there are some familiar approaches and methods being adapted to make them more suitable for use within whole systems approaches. Such as theories of change and pre and post intervention measurement exploring more deeply the unintended consequences of our efforts, the causal links between activities and outcomes and understanding why things may have happened. Or story-telling and case studies; focusing more on what happened, how and why, and giving people within communities a voice within measurement approaches, rather than being spoken on behalf of.
The way forward will likely involve a mixture of approaches. Adapting existing approaches as well as developing new ones to be sensitive to the complexity of physical inactivity. And it will require us to orientate our measurement more towards explanation and transferability of knowledge. To think about ‘what works, for whom, under what circumstances and why’.
How could we measure differently together?
- Measure to understand, explain and create transferable knowledge.
- Embrace complexity, be prepared to explore and expect the unexpected.
- Adopt a principles-based approach that encourages innovation, co-ownership and a shared understanding of why decisions are made and why actions are taken.
Principles for complexity friendly measurement
The principles explained
These are an attempt to provide a set of principles which can be explored by people within their measurement approaches. They are based on our current understanding and views regarding the common ingredients which help to make our measurement more fit for purpose and friendly to the complexity of physical inactivity.
We are interested in understanding perspectives on the principles. Do they resonate? What do you like about them? Are they helpful? Is anything missing? How do we explore these principles and share our learnings across people and places?
A shared vision that embraces different intentions for measurement
There is a distinct difference between a shared vision and a shared intention or purpose. A vision is what you want to create or accomplish and can often be something that is hard to disagree with, e.g. creating a healthier and more active nation. An intention is more of a mental state about what you will do and why, that is influenced by your personal values and how you see the world. We need to recognise and value different reasons for being involved in the work and what intentions people have for measurement. For some measurement will be about proving if something has worked. For others it will be about learning and understanding what progress has been made. Exploring the different intentions for our measurement efforts through the lens of our shared vision, helping us to measure what matters to people individually, is at the heart of complexity friendly measurement.
Complexity gives us the reason to innovate
This 2-minute video is a helpful summary of why we need to measure and learn in ways that are friendly to the complexity of what we are trying to achieve. If tackling physical inactivity and the inequalities within this needs different and more radical approaches then how we measure change and assess impact, and the basis for learning, should also shift with it. But it is not easy to change embedded ways of doing things. We can’t just propose new measurement approaches and expect everyone to be ready for them and embrace them. We have to help people understand complexity. To create a shared understanding of why we need to do things differently. We aren’t evolving measurement approaches because someone told us to. We are changing because too often we are measuring things that don’t help us to understand how we affect change and the impact of it, e.g. a greater focus on system conditions.
We are testing and adapting theories
A theory is how we think any idea is going to translate into the things we want to see and why. For example, influencing policies around road safety will lead to more people travelling by bike or walking, which will lead to more people being physically active, thereby reducing carbon emissions. The ‘If, Then, Because’ model is helpful here. IF we can test and adapt theories about what works to increase activity levels in our population, THEN we will create more sustainable and relevant opportunities for people to move more, BECAUSE we have a deeper, more nuanced understanding of ‘what works, for whom, under what circumstances and why’ within our population. We are constantly testing theories and updating them based on our learnings, e.g. theories based on a mixture of; observations, lived experiences and perspectives, and theories from literature.
Mixed methods that help to understand how and why
There are lots of different methods and approaches being used and tested (e.g., process tracing, contribution analysis, realist evaluation, ripple effects mapping, systems mapping, story-telling). And many more. They are methods which focus on understanding why and how, and following the threads and patterns of change. They support adaptation and iterative learning. Our methods are being developed and adapted in places by the people doing the work to understand how and why change is happening. This is about using different methods to capture different types of data. And more important than the actual data collection tool is what you are using the data for. If the purpose of our measurement is to support learning, improvement, adaptation and to seek deep explanation then it makes sense to use a variety of methods that can provide nuggets of data which you can combine to build a picture of the underlying nature of things. No one method will ever give you the full picture, but they can be combined to substantiate or refute your working theories. By sharing our experiences, we can learn with each other about how different methods are suitable for different contexts.
Explain and build knowledge around what is going on
The causes of social challenges such as homelessness and obesity are not always clear. They are not associated to one single factor. And therefore there is not one organisation who can address these influencing factors alone. This needs to be the basis for our measurement. To understand what conditions are holding the problem of physical inactivity in place and how we address these. Individually, organisationally and collectively we need to be able to explain what is happening around our efforts to tackle physical inactivity. To build knowledge around this that is accessible and applicable for each other. And to share and explore this together with a mindset of ‘lifting’ and applying the theories behind our learning in places, e.g. under what circumstances can you co-produce physical activity solutions with a deprived community? Focusing as much on this as the intervention itself.
Be accountable for illuminating learning
We need to become known for learning if we are serious about mobilising whole systems approaches around physical inactivity. For learning to be more ‘open-access’ so we get better at building from the learning of others. We should feel accountable to ourselves for learning and improving. And we need to be able to draw out from our learning what matters to people and not just what matters to ourselves. Eg, to talk to commissioners’ ‘head’ with meaningful stats, facts and data. Their ‘heart’ with powerful stories of change. And their ‘hands’ with tangible actions they can take. Having multiple outputs in multiple formats is important for translating the knowledge with different audiences.
Adopt deep curiosity together
Some of the factors holding the problem of physical inactivity in place will be embedded processes and behaviours. This requires us to be curious about why change is needed and how could it happen, e.g. The 5 Whys. To not fear the big and challenging changes. To seek, share and make sense. ‘For working and learning out loud connects us.’
“…What we loosely call knowledge, using terms like knowledge-sharing or knowledge capture, is just an approximation. We are not very good at articulating our knowledge, says knowledge management expert Dave Snowden: We always know more than we can say, and we will always say more than we can write down…”
Meet people where they are at
The readiness of people and places for new and alternative approaches to how we define and measure impact and progress will be influenced by a range of factors. From the ways in which people view the world, to behaviours, to expectations of ‘how things are done around here’. Each person and place may develop their thinking and sense of what is possible at different speeds. We need to meet people where they are at. As we try to ‘pull’ people across from unhelpful ways of measuring our work, we need to help people step into a safe, middle space. And for this to be a space where they can explore and become more open to change, before stepping into the new world of measurement.
People and place will shape it
People own what they help create. Continuing to develop a culture where the diversity of people and place is seen as an asset to learn with and from is key. To encourage local innovation around how to measure in ways that make sense to people and the place. To not look for set frameworks, but for measures that matter to the people doing the work and methods that enable them to be in the measurement. We need to encourage and ensure there is permission for people to shape how they will measure their collective efforts in their place.
Let us know what you think. Bring to life your successes. Challenge thinking and raise questions. Continue to connect with colleagues and share your views and learnings.
Many questions we know remain. This wasn’t about reaching definitive answers but about identifying how we keep progressing together towards more complexity friendly measurement approaches. A new approach to measurement presents a genuine chance to innovate and transform how people see its role in making a real impact on physical inactivity.
To help keep the conversations going in the coming months we are also interested in the following and if you have any ideas please do get in touch:
- Exploring local practice through a series of deep dive sessions into a range of measurement approaches and to consider the principles in action. Would you like to share your practice?
- Exploring views on the terminology used in this space (eg, measurement, evaluation, learning, impact, progress, conditions) and how we develop a shared understanding. How are you approaching the range of terminology?