Demand
Diverging interests
Project implementers, policy makers, researchers and project partners have different interests and expectations, resulting in diverging motivations for implementing an RIE. Researchers, for instance, pursue scientific publications, whereas project implementers may be more interested in practical questions of project steering. The tension remains mostly unsolved within GDC to date. This barrier was primarily identified by respondents with RIE experience.
Potential solutions
Potential solution Ensure close collaboration between researchers and other stakeholders
To foster RIE initiation, it is helpful if researchers engage in close collaboration with all other stakeholders at the beginning of an RIE and, in particular, with project partners in partner countries. In this process, evaluators and researchers should present themselves as learning partners and supporters rather than auditors and “judges”. This approach can help emphasise the learning purpose of the RIE over that of accountability. Understanding stakeholder demands, and making use of and involving local capacities, can be concrete steps to generate relevant results, enhance support and build ownership for the RIE.
Potential solution Generate additional knowledge beyond the RIE
Generating additional knowledge beyond the results from the RIE can be an effective measure to foster RIE initiation. This might be descriptive information, needs assessments, cost calculations and process evaluations. First, answering these additional questions has an added value in itself. Second, the additional knowledge can be useful when interpreting the results of an RIE. Additional knowledge was either generated by additional data (often qualitative data within a mixed-methods design) or by extensive analysis of baseline and follow-up data from the RIE. This approach is increasingly followed by USAID and J-PAL as well as IPA and DIME.
Low priority given to RIE evidence
There is little (though increasing) demand for (RIE) evidence from different stakeholders, most importantly from donors (i.e. BMZ) and partners. This is accompanied by the observation of interviewees that other types of evidence (e.g. from process evaluations or expert opinions) are often ascribed equivalent or higher utility and quality.
Potential solutions
Potential solution Create awareness for RIE
To foster the initiation of RIEs within GDC, it is important to create awareness of RIE among GDC stakeholders. This can mean pointing out specific uses of RIE, particularly the causal attribution of impacts to interventions. It can also mean pointing to the potential of a solid impact communication to the public or to the fact that the costs of projects with low actual impacts are higher in the long run than costs associated with RIE. Creating and publicising positive RIE examples, emphasising the international relevance of RIE in the sectoral dialogue, and engaging in open and honest dialogue about ethical concerns can also contribute to awareness building. Awareness building among employees in leadership positions may be particularly beneficial as it signals support and creates demand.
Potential solution Create opportunities for RIE exchange
Creating opportunities for networking and exchange of RIE experiences within but also beyond their own organisation can be a potential measure to foster RIE initiation. This can take the form of peer exchange, research–practice exchange or policy–practice exchange, and includes reflections on practical implementation questions as well as existing evidence gaps.
Potential solution Demonstrate leadership commitment to RIE
Leadership commitment is a strong facilitator for RIE initiation within GDC and internationally. Specifically, such leadership commitment can be expressed by giving priority to evidence generation and take-up, by functioning as role models and explicitly using RIE findings for decision making or by rewarding evidence generation and take-up.
And when people realise that their evidence, which they have generated with a great deal of effort and time investment, is not being heard and that the political interests of the donor are always in the foreground, then they will very quickly come to the end of the line. Yes? So there must also be a willingness on the part of the donor to support evidence‑based decisions. And this tension between political decisions and evidence-based decisions must be balanced. (Int_9)
Potential solution Embed RIE in a strategy for evidence generation and take-up
Since take-up of results of an RIE is generally a core motivation for initiating an RIE, guaranteeing evidence take-up from the onset of an RIE is a relevant measure for its initiation. Evidence take-up in turn is facilitated when (rigorous) evidence is generated on operationally, strategically or politically relevant questions. In consequence, this means that such relevant questions must be defined prior to or at the beginning of an RIE. To implement this measure, a set of central questions with importance to GDC should be defined (e.g. within the scope of the core topic strategies or initiative topics) such that they can be systematically addressed with RIE and other evaluation approaches.
Potential solution Ensure early coordination of RIE and project cycles
To facilitate RIE initiation, it is important to systematically check at early stages in project cycles for the possibility of conducting an RIE (e.g. using an RIE evaluability assessment tool). This can be done by integrating RIE reflection points in existing process flows and in guidelines about how to create module proposals or brief statements. In practice, early coordination between the project cycle and a possible RIE could be achieved by involving RIE experts already in the project proposal or design phases (after commissioning), or proposing an RIE as a component of the commissioned project.
Potential solution Include RIE in the organisation’s vision, mission and strategy
To encourage RIE initiation it can be helpful for GDC organisations to commit to evidence-informed policy making in their strategies. They provide orientation and help anchor RIEs during changes in leadership. BMZ’s recent strategy document for evaluation in GDC is a good example of such an effort in that it explicitly encourages impact orientation and the role of accompanying research such as RIEs within GDC (BMZ, 2021).
Potential solution Pass formal legislation or requirement
Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. The United States’ Evidence Act, Mexico’s Social Development Law (White, 2019) or South Africa’s mandatory system of evidence assessment provide examples hereof. Instead of national legislation, RIE initiation could also be formalised by requirements passed by the respective government department or by binding organisational instructions. Our data shows that, whereas quota regulations or thresholds for RIE were largely deemed unhelpful, top-down requirements or clear donor signals were generally judged quite effective in establishing a more systematic use of RIE. In order to encourage the initiation of RIE, formal requirements could require project implementers to conduct: [?]
Potential solution Set incentives for RIE
Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).
No formal requirement for RIE
Our data shows a lack of formal requirement or procedural anchoring (including respective guidelines) for initiating RIE within GDC. Actually, the Federal Budget Regulations (Bundeshaushaltsordnung, BHO) formulates high requirements for performance assessments (Wirksamkeitskontrolle). Yet, these regulations do not require a specific evaluation approach. Hence, this regulation is not currently interpreted as necessitating RIEs.
Potential solutions
Potential solution Set incentives for RIE
Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).
Potential solution Pass formal legislation or requirement
Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. The United States’ Evidence Act, Mexico’s Social Development Law (White, 2019) or South Africa’s mandatory system of evidence assessment provide examples hereof. Instead of national legislation, RIE initiation could also be formalised by requirements passed by the respective government department or by binding organisational instructions. Our data shows that, whereas quota regulations or thresholds for RIE were largely deemed unhelpful, top-down requirements or clear donor signals were generally judged quite effective in establishing a more systematic use of RIE. In order to encourage the initiation of RIE, formal requirements could require project implementers to conduct:
RIEs in projects above a certain budget (threshold)
a number of RIEs per year in each sector (quota)
RIEs within (annually) changing thematic windows
RIEs in all pilot projects or in any project that is intended to be scaled-up
a standardised check for RIE feasibility and usefulness in existing process flows and respective internal guidance documents
Unclear benefit of RIE
There is a lack of incentives and (career) benefits for individuals for conducting RIEs. Implementing RIEs is reportedly generally not rewarded and comes hand in hand with additional personal effort.
A lack of clarity about the benefits of RIEs for the project, for example for operational project steering, can hinder the initiation of an RIE.
A lack of clarity and awareness about how to transfer or pool single RIE findings and use RIE for policy making, steering and strategy definition can also hinder RIE initiation.
Potential solutions
Potential solution Build RIE capacities
All data sources and all stakeholder groups name developing RIE capacities as a high‑priority measure. Capacity development measures should not only target decision makers and employees within GDC, but also include evaluation capacity development (ECD) in partner countries. At the same time, GDC can learn from partner countries like Mexico, Costa Rica and South Africa that have a long history of, and strong expertise in, evidence-based policy making. Developing RIE capacities should follow the systemic approach, targeting individuals, networks, institutions and the enabling environment. Topics can cover the following aspects: fundamental conceptual knowledge (“What is impact evaluation and what is it good for?”), methodological knowledge, RIE procurement and management, practical RIE implementation knowledge and knowledge on evaluation management and dissemination.
Potential solution Create awareness for RIE
To foster the initiation of RIEs within GDC, it is important to create awareness of RIE among GDC stakeholders. This can mean pointing out specific uses of RIE, particularly the causal attribution of impacts to interventions. It can also mean pointing to the potential of a solid impact communication to the public or to the fact that the costs of projects with low actual impacts are higher in the long run than costs associated with RIE. Creating and publicising positive RIE examples, emphasising the international relevance of RIE in the sectoral dialogue, and engaging in open and honest dialogue about ethical concerns can also contribute to awareness building. Awareness building among employees in leadership positions may be particularly beneficial as it signals support and creates demand.
Potential solution Create opportunities for RIE exchange
Creating opportunities for networking and exchange of RIE experiences within but also beyond their own organisation can be a potential measure to foster RIE initiation. This can take the form of peer exchange, research–practice exchange or policy–practice exchange, and includes reflections on practical implementation questions as well as existing evidence gaps.
Potential solution Embed RIE in a strategy for evidence generation and take-up
Since take-up of results of an RIE is generally a core motivation for initiating an RIE, guaranteeing evidence take-up from the onset of an RIE is a relevant measure for its initiation. Evidence take-up in turn is facilitated when (rigorous) evidence is generated on operationally, strategically or politically relevant questions. In consequence, this means that such relevant questions must be defined prior to or at the beginning of an RIE. To implement this measure, a set of central questions with importance to GDC should be defined (e.g. within the scope of the core topic strategies or initiative topics) such that they can be systematically addressed with RIE and other evaluation approaches.
Potential solution Generate additional knowledge beyond the RIE
Generating additional knowledge beyond the results from the RIE can be an effective measure to foster RIE initiation. This might be descriptive information, needs assessments, cost calculations and process evaluations. First, answering these additional questions has an added value in itself. Second, the additional knowledge can be useful when interpreting the results of an RIE. Additional knowledge was either generated by additional data (often qualitative data within a mixed-methods design) or by extensive analysis of baseline and follow-up data from the RIE. This approach is increasingly followed by USAID and J-PAL as well as IPA and DIME.
Potential solution Set incentives for RIE
Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).