Umbrella-review of meta-analyses: a methodological framework to support evidence-based policymaking

Policymakers should increasingly support their decisions through rigorous synthesis of the available scientific evidence. In several scientific fields, meta-analysis (MA) has become a gold standard quantitative method for evidence synthesis, and policymakers can potentially inform their decisions taking advantage of an increasing number of published MAs. However, MAs are heterogeneous, both in content and quality, and a systematic framework is needed to help scientific experts report the results and synthesize the outcomes, while assessing the quality and knowledge gaps in a rigorous and transparent manner. Here, we describe a methodological framework developed by the Joint Research Centre (JRC) of the European Commission (EC) for the assessment of the environmental, climate and productivity impacts of agricultural practices for policymakers in the context of the Common Agricultural Policy (CAP). This framework has three main aims: a) to provide answers for policy questions in the short term, using the most systematic and statistically-robust available evidence; b) to extract, classify and map available meta-analytical results for more in-depth analyses and data re-use (e.g., in modelling frameworks); and c) to build structured evidence libraries to initiate and facilitate a continuous science-to-policy interface on the topic. In this report, we provide details of the methods, stepwise-procedures and tools developed in this context, focusing on: 1. Framing policy question(s) using the Population-Intervention-Comparator-Outcome (PICO) process; 2. Defining the systematic search, screening and selection procedure of existing MAs; 3. Extracting and classifying the main results; 4. Assessing the quality of the evidence and the knowledge gaps; 5. Providing short-term semi-quantitative syntheses of the main outcomes; 6. Selecting, extracting and synthesizing the quantitative estimates of the effects; 7. Building an evidence library platform as an interface between policymakers and the scientific community.