What did we analyse and how?

In an empirical research project DEval analysed the current state of rigorous impact evaluation (RIE) and evidence in German development cooperation (GDC). Learn about the research questions and data sources that we used in our research. 

 

Research Questions

  1. What is the status quo of (a) the initiation of RIE and (b) the take-up of RIE evidence within GDC?
  2. What are existing barriers to (a) the initiation of an RIE and (b) the take-up of RIE evidence within GDC?
  3. What are potential measures to overcome these barriers?

Interviews

We conducted 50 interviews from seven sub-groups among (a) international RIE experts and (b) a comprehensive set of stakeholders with RIE knowledge within GDC.

Further Information

For data analysis, we used an independent double coding approach, employing a tandem setup for increased reliability. We clustered codes, studied frequencies and carved out patterns, for example by comparing sub-groups.

We mapped all relevant stakeholder groups and corresponding organisations within GDC and complemented this with the stakeholder group of international RIE experts (see Figure). Second, we made a purposeful selection of organisations within each stakeholder group. The minimum requirement was for each stakeholder group to be represented by at least one organisation, while also considering the organisations’ size and relevance to GDC. We selected interviewees from these organisations who were able to cover as many different perspectives and thematic areas as possible with regard to the use of RIEs in GDC. We thus followed a diverse-case method within different strata as the sampling strategy [Seawright and Gerring, 2008]. In total, we conducted 50 interviews with stakeholders from seven sub-groups between July 2019 and February 2021.

Stocktaking

Using an online stocktaking survey to collect data of all RIEs that had been conducted with the involvement of GDC since 2014, we found 97 RIEs in total. The survey included questions about the RIE itself (e.g. its methodology), and about the evaluated development project.

Further Information

We sent out the survey to the evaluation departments of relevant organisations, such as BMZ, GIZ, KfW Development Bank, various development CSOs, foundations and to several research institutes and private consultancies. We complemented the survey with an online desk search for RIEs of relevant organisations. In most cases, project managers of the implementing organisation filled in the survey. Data collection ran from October 2020 to April 2021. 

In total, 142 RIEs were registered, of which 97 RIEs fulfilled the inclusion criteria and were analysed using descriptive statistics. Open responses were coded and clustered using content analysis.

Even though this was the most comprehensive search for RIEs in GDC to date, it is very likely that the 97 RIEs identified by our survey do not constitute the full number of RIEs in GDC between 2014 and 2020. Firstly, our desk research identified 18 potentially relevant RIEs for which no information was reported via our survey and for which we were unable to acquire further information. Secondly, the substantial effort necessary to detect 97 RIEs suggests that there may still be RIEs we do not know about.

Registration of RIE

The stocktaking survey page is still open and RIEs can be registered on an ongoing basis via
 

s2survey.net/RIE_registry/

Evidence survey

The evidence survey targeted all employees of GDC organisations receiving BMZ funds. 1 086 employees from at least 23 different GDC organisations participated in our evidence survey, providing information on their knowledge and use of rigorous evidence and their attitudes towards it.

Further Information


Institutional affiliation of respondents to the evidence survey

Source: evidence survey; question: "Where are you currently working primarily?" N=1,071. The number of respondents differs from the total sample as 15 respondents dropped out before answering this question.

Data analysis involved descriptive statistics as well as qualitative content analysis of open response fields.

The survey was distributed to the evaluation departments of 58 GDC organisations, requesting them to forward the survey link to all colleagues (except for administrative staff). Data were collected between March 2021 and April 2021. In sum, 1 086 respondents from at least 23 different organisations participated in the evidence survey and provided information. Dropout rate during survey completion was 39.9%, with 653 participants reaching the final page of the survey. Given the snowball procedure, we do not know exactly how many people were reached by the invitation to participate in our evidence survey. Therefore, it is not possible to calculate a response rate. The following numbers may help to put the number of employees per organisation and the reach of our evidence survey into perspective: BMZ has approx. 1 230 employees, GIZ has approx. 23 600 employees and KfW Development Bank has approx. 1 000 employees.

Review on international experiences with RIE

We commissioned a study on the institutionalisation of RIE implementation and evidence take-up among international peers in development cooperation (such as FCDO, J-Pal or Oxfam) to learn from international RIE experience.

Further Information

The report identifies organisational “archetypes”, collects best practices and common barriers and offers general recommendations for institutionalising RIE implementation and evidence take-up.

A consortium of the consulting firms Scio and Sattva carried out the study between April 2020 and December 2020. Based on the criteria of (a) similarity to GDC organisations, (b) reputation regarding RIE implementation or evidence take-up, and (c) learning potential for GDC, we selected the following institutions for in-depth analysis: Asian Development Bank (ADB), Agence française de développement (AFD), FCDO, J-PAL, MCC, Oxfam, USAID and the What Works Network (WWN).

Data collection was based on key informant interviews and accompanied by a literature search and analysis. In total, 29 semi-structured interviews were conducted. Interviews lasted one hour on average. A framework for institutionalisation guided the semi-structured interviews and subsequent data analysis. The framework covered different factors that potentially affected the systematic integration of the RIE approach into an organisation (e.g. strategic alignment and level of buy-in, funding and resources etc.).

Analysis of GDC portfolio and document analysis

We analysed the GDC portfolio and official project documents to check whether the global RIE evidence base is relevant to the GDC portfolio and to what extent the global RIE evidence base has already been utilised in GDC.

Further Information

Our analysis comprised four steps:

  1. Comparing German ODA flows with evidence from the 3ie development evidence portal (DEP): We mapped bilateral BMZ disbursements on the available rigorous evidence from the 3ie’s DEP. This comparison reveals the sectoral overlap between globally available rigorous evidence and German ODA.
     
  2. Screening 124 project documents of randomly sampled projects from GIZ and KfW Development Bank with regard to evidence take-up: Within the sector “Government & Civil Society”, we randomly drew five projects each from GIZ and KfW Development Bank for the period 2014–18. This resulted in a database of 124 documents, including, for example, project proposals and project evaluation reports. We screened all documents for their use of scientific evidence and two projects for non-scientific sources.
     
  3. Checking availability of relevant RIE evidence for sampled projects: We conducted a non-exhaustive keyword search to find thematically suitable RIEs, SRs and EGMs published after 2014 for each of the 10 projects to check whether the global RIE evidence base was, in fact, relevant to these specific projects. In addition, we screened abstracts of those studies that appeared to be of potentially high relevance to the sampled projects.
     
  4. Screening of project documents for two additional projects from KfW Development Bank and GIZ where RIEs had been conducted: We chose two additional projects in which RIEs had been conducted and screened all 67 project documents to see whether the RIEs were mentioned.

Literature search and further documents

The research process was accompanied by a review of the scientific literature as well as other secondary sources, including impact evaluation guidance documents, the 2019 GIZ RIE report, research papers, flagship reports and government documents.

Further Information

  • Impact evaluation guidance documents: We identified and screened more than a dozen official guidance documents by international development stakeholders that set out criteria for conducting RIE.
  • GIZ RIE report: Florian et al. (2019) published an internal review on RIE in 2019. This report examined RIEs conducted within GIZ between 2014 and 2018 and outlined conditions under which RIE have proven particularly useful for GIZ.
  • Research papers and flagship reports: We included insights on topics such as identifying relevant evidence, evidence take-up, experiences with RIE within GDC, research ethics and behavioural science.
  • Government documents: To analyse the contracting procedures between IOs and BMZ (see Box), we screened official BMZ guidance documents. The analysis aimed at examining forma requirements and options regarding RIE initiation and evidence take-up when planning and implementing bilateral projects.

 

In short: Procedures of GDC

The planning and implementation of bilateral projects follows mandatory procedural rules, which were revised in 2017 through a joint procedural reform (Gemeinsame Verfahrensreform, GVR) and are documented in internal guidance documents. These procedural rules specify which steps must be taken by BMZ and the respective IO when implementing a project.

To Top