Assessing Progress and Impact / Value
The purpose of the Regional Hub’s efforts is to create Outcomes, Impact and Value to various stakeholders. In a public sector or whole of society application (includes private, public and not for profit organizations as well as governments), Outcomes, Impact and Value to multiple stakeholders in both the short-term and long-term must be assessed and adaptions made on the basis of a balance among them.
Understanding progress in a complex collaboration aimed at improving mentally healthy campuses is a challenge. Yet understanding progress is critical to having the information necessary to choose alternative actions wisely – not only in your own organization, but as a contributor to the whole collaboration’s aspiration. Qualitative, Quantitative and narrative information are all needed to tell this story in a way that is actionable so that the Regional Hub can adapt as it learns through various stages.
One element of assessing progress involves descriptive processes – identifying what has been done over time. Even in a description one can discern the multiple dimensions of change in a complex environment, system or organization. Stories help to animate the concrete elements of a timeline, and this includes stories that illustrate hope and celebration, as well as stories that illustrate concern or despair.
Progress in how we understand the nuances of a collaboration’s complex terrain is also a marker, illustrated by increasing detail and nuance in various system maps that are updated through the stages of a Regional Hub’s trajectory. In individual projects, progress in illuminating the multiple dimensions of an intervention, delivery and organizational architecture as well as policy architecture are another marker.
A variety of methodologies can be used to assess progress and impact or value – here are five:
- Developmental Evaluation is an approach to assessment which emphasizes real time feedback and learning in emerging, messy and sometimes fast-moving environments and changes in people’s capabilities. Different types of developmental evaluations include Principle-focused evaluation and systemic learning evaluations that illuminate emerging delivery, management and policy architectures in early stage innovations.
- Utilization Focused Evaluation UFE can be used for different types of evaluation (formative, summative, process, impact) and it can use different research designs and types of data. Whichever type of evaluation, measuring only inputs and outputs leads to the challenge of having a ‘black box’ – not knowing why or why not the results were achieved. Responses to that challenge include Theory-based evaluation and Realist evaluation.
- Realist Evaluation answers “What works, for whom, in what respects, to what extent, in what contexts, and how?”, which is more helpful to decision-makers than a simple ‘did it work’ evaluation question. In order to answer the larger question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context.
- Outcomes Mapping was initially developed for the international development field where it is more appropriate to evaluate impact on the basis of behavioural change in the group that is then doing the work in their country, than on long-term changes in developing countries.
- Systemic Maturity Model assists multiple organizations to assess their own and the collective’s capacity to change. This approach embraces the challenge when a desired outcome e.g. mentally healthy campus, involves multiple dimensions of an environment, system or setting.