Does Germany currently generate RIE evidence?
In an empirical research project DEval analysed the current state of rigorous impact evaluation (RIE) in German development cooperation (GDC). Here we answer questions, such as: How many RIEs have actors in GDC conducted? What designs were used in the RIEs? Is RIE initiation sectorally and regionally aligned with German development funding?
Below, you find 5 key findings:
Key finding 1: We found a total of 97 RIEs since 2014 – compared to the large volume of German ODA there is still substantial potential for conducting more RIEs (and more systematically) to increase the effectiveness of GDC.
Further Information
In total, we identified 97 RIEs fulfilling our criteria, of which respondents reported 47 RIEs as completed and 50 RIEs as still being in progress or having been approved (it is unclear whether all planned RIEs will be completed). Taking into account the increasing recognition of RIE among GDC stakeholders, we also expect the number of RIEs to increase in the future. However, the total number has to be interpreted with caution, given the fact that substantial numbers of these RIE were clustered within larger projects and given that we relied on self-assessments of methodological rigour.
23 RIEs were conducted as part of two GIZ global projects (Globalvorhaben) in the field of rural development. Within these global projects, the same or similar interventions were rigorously evaluated across different countries. Furthermore, an entrepreneurial training programme partially financed by a German CSO was rigorously evaluated in different contexts and at multiple points in time. This programme accounts for 13 RIEs.
Key finding 2: GIZ, KfW Development Bank and German civil society organisations (CSOs) have all been involved in RIEs (for example as project implementer or by financing the RIE).
Figure: RIE involvement by organisation
Further Information
Different stakeholders often have different roles within an RIE, such as implementing the project, or implementing the evaluation, which includes data collection and data analysis. Our interviewees stated that the typical stakeholder setup of an RIE comprises an IO or CSO that usually implements the project, together with their local implementing partners and a research institution or consulting firm that conducts or supports the impact evaluation. Involvement of local stakeholders usually takes one of three forms: 1) government departments in partner countries involved as coordinator or facilitator, 2) local CSOs or public institutions involved in the implementation of projects, and 3) local private companies carrying out data collection. For about 70% of RIEs at least one organisation from the partner country was reported to be involved in the implementation of the development project or the RIE evaluating it, leaving a share of approximately 30% of RIEs with no reported partner involvement.
In the relevant period, DEval has conducted four RIEs (DEval is included in “research institutions”).
Key finding 3: Most of the 97 RIEs were self-reported to be RCTs or to use a difference-in-differences (DiD) design, while other quasi-experimental designs were rarely used.
Further Information
GDC uses other quasi-experimental designs rarely or not at all, even though these designs often allow for RIE implementation at a lower cost than RCTs (for example when existing data can be used). The frequent use of DiD by those working in GDC is notable, compared to the global RIE evidence base, which we approximated using the 3ie DEP. RIEs in 3ie’s DEP use fewer DiD (17%) and more fixed effects (16%) and other quasi-experimental designs. The share of RCTs in 3ie’s DEP is 54% (3ie, 2021).
Of the RIEs in our stocktaking survey, 61% report that their RIE was accompanied by qualitative research.
Key finding 4: The sectoral distribution is not aligned with the distribution of German ODA flows or the global evidence base. This adds to our impression that RIE initiation in GDC does not yet follow an overarching strategy or systematic approach.
Figure: Sectoral comparison between stocktaking RIEs, German ODA flows and 3ie impact evaluations
Further Information
The sectoral distribution of GDC RIEs does not closely reflect the distribution of German ODA flows. For example, the sector “Government & Civil Society” accounted for 17% of BMZ-related, bilateral ODA disbursements between 2014 and 2018 – a much larger share than the 6% share of stocktaking RIEs.
There may be some differences between sectors with regard to how well RIEs can be implemented on average. However, the RIE experts in our interviews agreed that it is not a question of sector whether an RIE can be implemented or not (source: interviews). Instead, this can only be analysed and decided individually on a project-basis. RIEs can be implemented in all sectors – for example Funk et al. (2018) documented RIEs in GIZ and KfW development bank government projects, a sector that is sometimes considered as difficult for RIE implementation.
Similarly, the sectoral distribution of GDC RIEs also differs substantially from the distribution of RIEs in the global evidence base (which we approximate using 3ie’s Development Evidence Portal (DEP)).
Key finding 5: The regional distribution is not aligned with the distribution of German ODA flows or the global evidence base. Again, this adds to our impression that RIE initiation in GDC does not yet follow an overarching strategy or systematic approach.
Figure: Comparison of regional distribution between stocktaking RIEs, ODA flows and 3ie impact evaluations
Further Information
Sub-Saharan Africa (SSA) is the region in which RIEs of our stocktaking were by far the most frequently conducted (71). The countries in which RIEs of our stocktaking were conducted most frequently are Uganda (13), Ethiopia (6), Malawi (5) and Zambia (5).
Much as with the sectoral distribution, the regional distribution of GDC does not closely reflect the distribution of German ODA flows. For example, although SSA has a share of 71% of RIEs within GDC, it only receives 30% of German ODA. One potential explanation is that RIEs are often initiated by motivated individuals and not embedded in an overarching ODA-oriented strategy. Aligning RIEs regionally with ODA flows is one possible strategy for identifying when and where to implement RIEs. Other examples might be conducting RIEs in pilot projects or systematically targeting evidence gaps. However, we did not find any particular strategy for RIE implementation within GDC during our research.
Similarly, the regional distribution of GDC RIEs also differs substantially from the distribution of RIEs in the global evidence base, which we approximate using 3ie’s Development Evidence Portal (DEP).