What did we analyse and how?

For data analysis, we used an independent double coding approach, employing a tandem setup for increased reliability. We clustered codes, studied frequencies and carved out patterns, for example by comparing sub-groups.

We mapped all relevant stakeholder groups and corresponding organisations within GDC and complemented this with the stakeholder group of international RIE experts (see Figure). Second, we made a purposeful selection of organisations within each stakeholder group. The minimum requirement was for each stakeholder group to be represented by at least one organisation, while also considering the organisations’ size and relevance to GDC. We selected interviewees from these organisations who were able to cover as many different perspectives and thematic areas as possible with regard to the use of RIEs in GDC. We thus followed a diverse-case method within different strata as the sampling strategy [Seawright and Gerring, 2008]. In total, we conducted 50 interviews with stakeholders from seven sub-groups between July 2019 and February 2021.

Institutional affiliation of respondents to the evidence survey

Source: evidence survey; question: "Where are you currently working primarily?" N=1,071. The number of respondents differs from the total sample as 15 respondents dropped out before answering this question.

Data analysis involved descriptive statistics as well as qualitative content analysis of open response fields.

The survey was distributed to the evaluation departments of 58 GDC organisations, requesting them to forward the survey link to all colleagues (except for administrative staff). Data were collected between March 2021 and April 2021. In sum, 1 086 respondents from at least 23 different organisations participated in the evidence survey and provided information. Dropout rate during survey completion was 39.9%, with 653 participants reaching the final page of the survey. Given the snowball procedure, we do not know exactly how many people were reached by the invitation to participate in our evidence survey. Therefore, it is not possible to calculate a response rate. The following numbers may help to put the number of employees per organisation and the reach of our evidence survey into perspective: BMZ has approx. 1 230 employees, GIZ has approx. 23 600 employees and KfW Development Bank has approx. 1 000 employees.

Our analysis comprised four steps:

1. Comparing German ODA flows with evidence from the 3ie development evidence portal (DEP): We mapped bilateral BMZ disbursements on the available rigorous evidence from the 3ie’s DEP. This comparison reveals the sectoral overlap between globally available rigorous evidence and German ODA.
    
2. Screening 124 project documents of randomly sampled projects from GIZ and KfW Development Bank with regard to evidence take-up: Within the sector “Government & Civil Society”, we randomly drew five projects each from GIZ and KfW Development Bank for the period 2014–18. This resulted in a database of 124 documents, including, for example, project proposals and project evaluation reports. We screened all documents for their use of scientific evidence and two projects for non-scientific sources.
    
3. Checking availability of relevant RIE evidence for sampled projects: We conducted a non-exhaustive keyword search to find thematically suitable RIEs, SRs and EGMs published after 2014 for each of the 10 projects to check whether the global RIE evidence base was, in fact, relevant to these specific projects. In addition, we screened abstracts of those studies that appeared to be of potentially high relevance to the sampled projects.
    
4. Screening of project documents for two additional projects from KfW Development Bank and GIZ where RIEs had been conducted: We chose two additional projects in which RIEs had been conducted and screened all 67 project documents to see whether the RIEs were mentioned.