A Rapid Evidence Assessment evaluating the impact and implementation of complex environmental policies on complex systems

Funded by: Defra


March 2014 to June 2014

Research project leader(s) at PSI

Robin Vanner

Ben Fagan-Watson

Background, summary and aims

Defra recognises that evaluation of the implementation and impact of its policies can be complicated. Environmental policies typically have impacts on very complex systems, and in response are often complex themselves. Complexity has been widely debated in a variety of policy and evaluation areas, and techniques have been developed to deal with it. There may be lessons that can be usefully transferred from policy areas with a longer tradition of evaluation. 

During research that the Policy Studies Institute conducted for Defra in 2012-14 (Scoping the remit and feasibility of a policy evaluation centre for Defra), in-depth interviews and workshops were conducted with Defra policy directors, social researchers, policymakers and social researchers from the Defra network. Issues raised by the participants included: polices that overlap both natural science and social science; the challenge of disentangling the impacts of specific policy interventions; Defra’s broad remit; some EU regulations restricting the flexibility of evaluation; the long timescales and broad geographical scope of Defra’s policies; and difficulties in establishing a robust counterfactual.

Project design

A Rapid Evidence Assessment (REA) will review literature relevant to the challenge of how to evaluate the implementation and impact of complex policies on complex systems within environmental policy areas. The core research question will be ‘What range of methods are available to evaluate the implementation and impact of complex environmental policies on complex systems?’ with sub-questions grouped around 1) Identifying the challenge and 2) Identifying possibly solutions. This REA will be delivered within four months through the following outputs:

  • The interim report will outline the research methodology, search strategy used (including a list of search terms, and the databases utilised), the criteria for including or excluding evidence, and a list of initial evidence as well as any research gaps identified.
  • The Final Report will synthesise findings from the project as a whole, document the approach taken, highlight key findings, draw out cross-cutting issues, and identify evidence gaps, uncertainties and any further research required.
  • A database will be provided with full details of each of the pieces of evidence included or excluded from the study. The database will lay out methodology, key findings, limitations etc. of each piece of evidence as specified in the tender.

These outputs will collate and synthesise evidence from a wide range of sources and present rigorously derived insights relevant to the development of environmental policy and evaluation. Key messages will be drawn out and tailored to different potential users of the work with Defra and more widely. We additionally hope to submit at least one paper based on the work to a relevant academic journal to ensure the long-term availability of the findings from the work.

Related publications / project outputs

What range of methods is available to evaluate the implementation and impact of complex environmental policies on complex systems?