What is a Systematic Review (SR)?
Systematic reviews (SR) are closely linked to the concept of rigorous impact evaluation (RIE). Put simply, a SR consists of many RIEs.
SRs synthesise the best available (rigorous) evidence on a specific research question. They have clear criteria for inclusion and exclusion of primary studies, an explicit and transparent search strategy, systematic procedures for data extraction, and offer a critical appraisal and analysis of included studies. By statistically integrating the results of a number of quantitative studies, a meta-analysis identifying average effects of interventions across large numbers of studies can be carried out.
What practitioners say
Experts from German Development Cooperation share their experience with a SR.
Henriette Sachse
Federal Ministry for Economic Cooperation and Development (BMZ)
Andrew Kaiser-Tedesco
Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH
What motivated you to undertake a SR?
Together with DEval, we had commissioned 3ie to develop an Evidence Gap Map (EGM) on peaceful societies. From that, we learned about evidence gaps with regards to social cohesion as well as gender equality and women’s empowerment in fragile contexts. As gender equality is a cross-cutting goal of every Transitional Development Assistance (TDA) project, we wanted to make sure that our projects had the necessary evidence to focus on the most effective approaches. Our former Head of Unit, Ralf Schröder, was a big source of motivation because he had just come back from the World Bank, where he had experienced firsthand the importance of learning from rigorous evidence and encouraged us to pursue this path. The SR offered us a way to get a broad overview of the results of relevant studies to inform our portfolio steering.
How has the SR benefited your work and informed decision-making?
The SR on gender changed the way we communicate with our implementing partners. Having a common basis of evidence and recommendations that everyone can easily access helps us to clearly present which approaches we’re looking to emphasize. This doesn’t mean we’re only using approaches that perform well in our review. Our partners need to be the final judges of what’s going to work best in their contexts. But we follow a guideline: “First consider the approaches supported by the strongest evidence and check if they fit the goals and context of the project. If they don’t, think more broadly.”
Could you share one particular insight you gained regarding the usefulness of the evidence product?
The most important insight we gained was about the relative effectiveness of approaches. The SR shows that we’re achieving some goals, such as resource empowerment, much better than others, such as changing norms or curbing gender-based violence. I think in international development, we often assume having effective answers, even when the evidence doesn’t support them or isn’t there. Many measures for changing norms, for example, are not that effective, or at least we don’t have the rigorous evidence indicating effectiveness. We should rather focus on the goals and approaches that we’re sure have traction, and work to improve other approaches so that we can effectively achieve our gender equality goals, as recently outlined in the feminist development policy.
What would you recommend others who are planning to develop a SR?
First, think about it systematically to make sure you are adding a review that fills a gap. EGMs are very helpful in this regard because they show the evidence clusters and gaps so clearly. Second, aim to produce a global public good by asking yourself what kind of review could be useful for others as well. The most obvious way to do this is by looking beyond just the contexts you’re operating in. This might even prove useful for your own project, as often promising interventions are transferable to completely new contexts. If your focus is too narrow, you might miss the opportunity to bring in a great new idea.