New Approaches to Evaluation

Understanding the impact and added value of interventions being implemented within defined geographical areas has long been recognised as a vital component of evidence-based policy-making. Understanding the counterfactual position (i.e. what would have happened anyway) is however, something which evaluators have continued to grapple with, particularly when evaluating the impact of area-based interventions. 

The use of comparative case studies to explore trends in policy-off and policy-on scenarios is the approach favoured by many evaluators, and is recognised as a feature of good practice in HM Treasury’s Magenta Book. Finding a suitable comparable case study can however be difficult, with few areas being sufficiently similar to enable direct comparison.

New research published by What Works[1] is seeking to develop a more refined approach to the identification of a comparison case study which could be applied to better understand the impacts of one-off interventions and events such as the movement of a major employer into an area.

The research undertaken by What Works Scotland[2] advocates the use of a synthetic control using data on past trends in several potentially comparable areas in order to provide a more robust basis for identifying impact. By drawing on trend data across a range of areas, there is less scope for analysis to be influenced by areas specific events or conditions. The approach is however only suitable when assessing the impact of major interventions which affect a significant proportion of an area or population; where there are no other events that affect the intervention area; and where impacts are only felt within the intervention area. 

The findings of the research are being considered by the ekosgen team to allow us to continue to build upon our extensive evaluation, impact assessment and data analysis work. The approach may have particular relevance to LEPs and Local Authorities; providing a robust quantitative method for assessing the impact of significant regional development policies, which can be used to give weight to more qualitative case study analyses of regional policies and interventions.

ekosgen regularly support clients to understand the impact and added value of projects, programmes and policies, including major capital investments such as The Sharp Project in Manchester and revenue schemes such as Greater Manchester’s 18-24 year old employment support programme.

For further details of our evaluation and impact assessment work or to discuss your support needs, please contact Carolyn Turnbull on carolyn.turnbull@ekosgen.co.uk  or 0845 644 5407

 

[1] https://www.gov.uk/guidance/what-works-network

[2] http://whatworksscotland.ac.uk/