What barriers hinder the initiation of rigorous impact evaluations (RIE)?

In an empirical research project DEval anlaysed which barriers hinder RIE initiation and which potential solutions could address those barriers. Please note that our potential solutions are called measures in our report. In our interactive illustration below, those barriers and solutions are organised along five critical steps for evidence generation:

1. Demand for evidence,  2. Awareness and capacity for evidence,  3. Feasibility for doing an RIE,  4. Resources to do an RIE,  5. Intention to do an RIE

 

Demand

Diverging interests

There is little (though increasing) demand for (RIE) evidence from different stakeholders, most importantly from donors (i.e. BMZ) and partners. This is accompanied by the observation of interviewees that other types of evidence (e.g. from process evaluations or expert opinions) are often ascribed equivalent or higher utility and quality.

Potential solutions
Potential solution Ensure close collaboration between researchers and other stakeholders

To foster RIE initiation, it is helpful if researchers engage in close collaboration with all other stakeholders at the beginning of an RIE and, in particular, with project partners in partner countries. In this process, evaluators and researchers should present themselves as learning partners and supporters rather than auditors and “judges”. This approach can help emphasise the learning purpose of the RIE over that of accountability. Understanding stakeholder demands, and making use of and involving local capacities, can be concrete steps to generate relevant results, enhance support and build ownership for the RIE.

Potential solution Generate additional knowledge beyond the RIE

Generating additional knowledge beyond the results from the RIE can be an effective measure to foster RIE initiation. This might be descriptive information, needs assessments, cost calculations and process evaluations. First, answering these additional questions has an added value in itself. Second, the additional knowledge can be useful when interpreting the results of an RIE. Additional knowledge was either generated by additional data (often qualitative data within a mixed-methods design) or by extensive analysis of baseline and follow-up data from the RIE. This approach is increasingly followed by USAID and J-PAL as well as IPA and DIME.

Low priority given to RIE evidence

There is little (though increasing) demand for (RIE) evidence from different stakeholders, most importantly from donors (i.e. BMZ) and partners. This is accompanied by the observation of interviewees that other types of evidence (e.g. from process evaluations or expert opinions) are often ascribed equivalent or higher utility and quality.

Potential solutions
Potential solution Create awareness for RIE

To foster the initiation of RIEs within GDC, it is important to create awareness of RIE among GDC stakeholders. This can mean pointing out specific uses of RIE, particularly the causal attribution of impacts to interventions. It can also mean pointing to the potential of a solid impact communication to the public or to the fact that the costs of projects with low actual impacts are higher in the long run than costs associated with RIE. Creating and publicising positive RIE examples, emphasising the international relevance of RIE in the sectoral dialogue, and engaging in open and honest dialogue about ethical concerns can also contribute to awareness building. Awareness building among employees in leadership positions may be particularly beneficial as it signals support and creates demand.

Potential solution Create opportunities for RIE exchange

Creating opportunities for networking and exchange of RIE experiences within but also beyond their own organisation can be a potential measure to foster RIE initiation. This can take the form of peer exchange, research–practice exchange or policy–practice exchange, and includes reflections on practical implementation questions as well as existing evidence gaps.

Potential solution Demonstrate leadership commitment to RIE

Leadership commitment is a strong facilitator for RIE initiation within GDC and internationally. Specifically, such leadership commitment can be expressed by giving priority to evidence generation and take-up, by functioning as role models and explicitly using RIE findings for decision making or by rewarding evidence generation and take-up.



 

And when people realise that their evidence, which they have generated with a great deal of effort and time investment, is not being heard and that the political interests of the donor are always in the foreground, then they will very quickly come to the end of the line. Yes? So there must also be a willingness on the part of the donor to support evidence‑based decisions. And this tension between political decisions and evidence-based decisions must be balanced. (Int_9)

Potential solution Embed RIE in a strategy for evidence generation and take-up

Since take-up of results of an RIE is generally a core motivation for initiating an RIE, guaranteeing evidence take-up from the onset of an RIE is a relevant measure for its initiation. Evidence take-up in turn is facilitated when (rigorous) evidence is generated on operationally, strategically or politically relevant questions. In consequence, this means that such relevant questions must be defined prior to or at the beginning of an RIE. To implement this measure, a set of central questions with importance to GDC should be defined (e.g. within the scope of the core topic strategies or initiative topics) such that they can be systematically addressed with RIE and other evaluation approaches.

Potential solution Ensure early coordination of RIE and project cycles

To facilitate RIE initiation, it is important to systematically check at early stages in project cycles for the possibility of conducting an RIE (e.g. using an RIE evaluability assessment tool). This can be done by integrating RIE reflection points in existing process flows and in guidelines about how to create module proposals or brief statements. In practice, early coordination between the project cycle and a possible RIE could be achieved by involving RIE experts already in the project proposal or design phases (after commissioning), or proposing an RIE as a component of the commissioned project.

Potential solution Include RIE in the organisation’s vision, mission and strategy

To encourage RIE initiation it can be helpful for GDC organisations to commit to evidence-informed policy making in their strategies. They provide orientation and help anchor RIEs during changes in leadership. BMZ’s recent strategy document for evaluation in GDC is a good example of such an effort in that it explicitly encourages impact orientation and the role of accompanying research such as RIEs within GDC (BMZ, 2021).

Potential solution Pass formal legislation or requirement

Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. The United States’ Evidence Act, Mexico’s Social Development Law (White, 2019) or South Africa’s mandatory system of evidence assessment provide examples hereof. Instead of national legislation, RIE initiation could also be formalised by requirements passed by the respective government department or by binding organisational instructions. Our data shows that, whereas quota regulations or thresholds for RIE were largely deemed unhelpful, top-down requirements or clear donor signals were generally judged quite effective in establishing a more systematic use of RIE. In order to encourage the initiation of RIE, formal requirements could require project implementers to conduct: [?]

Potential solution Set incentives for RIE

Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).

No formal requirement for RIE

Our data shows a lack of formal requirement or procedural anchoring (including respective guidelines) for initiating RIE within GDC. Actually, the Federal Budget Regulations (Bundeshaushaltsordnung, BHO) formulates high requirements for performance assessments (Wirksamkeitskontrolle). Yet, these regulations do not require a specific evaluation approach. Hence, this regulation is not currently interpreted as necessitating RIEs.

Potential solutions
Potential solution Set incentives for RIE

Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).

Potential solution Pass formal legislation or requirement

Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. The United States’ Evidence Act, Mexico’s Social Development Law (White, 2019) or South Africa’s mandatory system of evidence assessment provide examples hereof. Instead of national legislation, RIE initiation could also be formalised by requirements passed by the respective government department or by binding organisational instructions. Our data shows that, whereas quota regulations or thresholds for RIE were largely deemed unhelpful, top-down requirements or clear donor signals were generally judged quite effective in establishing a more systematic use of RIE. In order to encourage the initiation of RIE, formal requirements could require project implementers to conduct:

 



RIEs in projects above a certain budget (threshold)

 



a number of RIEs per year in each sector (quota)



 

RIEs within (annually) changing thematic windows

 



RIEs in all pilot projects or in any project that is intended to be scaled-up



 

a standardised check for RIE feasibility and usefulness in existing process flows and respective internal guidance documents

Unclear benefit of RIE

There is a lack of incentives and (career) benefits for individuals for conducting RIEs. Implementing RIEs is reportedly generally not rewarded and comes hand in hand with additional personal effort.

A lack of clarity about the benefits of RIEs for the project, for example for operational project steering, can hinder the initiation of an RIE.

A lack of clarity and awareness about how to transfer or pool single RIE findings and use RIE for policy making, steering and strategy definition can also hinder RIE initiation.

Potential solutions
Potential solution Build RIE capacities

All data sources and all stakeholder groups name developing RIE capacities as a high‑priority measure. Capacity development measures should not only target decision makers and employees within GDC, but also include evaluation capacity development (ECD) in partner countries. At the same time, GDC can learn from partner countries like Mexico, Costa Rica and South Africa that have a long history of, and strong expertise in, evidence-based policy making. Developing RIE capacities should follow the systemic approach, targeting individuals, networks, institutions and the enabling environment. Topics can cover the following aspects: fundamental conceptual knowledge (“What is impact evaluation and what is it good for?”), methodological knowledge, RIE procurement and management, practical RIE implementation knowledge and knowledge on evaluation management and dissemination.

Potential solution Create awareness for RIE

To foster the initiation of RIEs within GDC, it is important to create awareness of RIE among GDC stakeholders. This can mean pointing out specific uses of RIE, particularly the causal attribution of impacts to interventions. It can also mean pointing to the potential of a solid impact communication to the public or to the fact that the costs of projects with low actual impacts are higher in the long run than costs associated with RIE. Creating and publicising positive RIE examples, emphasising the international relevance of RIE in the sectoral dialogue, and engaging in open and honest dialogue about ethical concerns can also contribute to awareness building. Awareness building among employees in leadership positions may be particularly beneficial as it signals support and creates demand.

Potential solution Create opportunities for RIE exchange

Creating opportunities for networking and exchange of RIE experiences within but also beyond their own organisation can be a potential measure to foster RIE initiation. This can take the form of peer exchange, research–practice exchange or policy–practice exchange, and includes reflections on practical implementation questions as well as existing evidence gaps.

Potential solution Embed RIE in a strategy for evidence generation and take-up

Since take-up of results of an RIE is generally a core motivation for initiating an RIE, guaranteeing evidence take-up from the onset of an RIE is a relevant measure for its initiation. Evidence take-up in turn is facilitated when (rigorous) evidence is generated on operationally, strategically or politically relevant questions. In consequence, this means that such relevant questions must be defined prior to or at the beginning of an RIE. To implement this measure, a set of central questions with importance to GDC should be defined (e.g. within the scope of the core topic strategies or initiative topics) such that they can be systematically addressed with RIE and other evaluation approaches.

Potential solution Generate additional knowledge beyond the RIE

Generating additional knowledge beyond the results from the RIE can be an effective measure to foster RIE initiation. This might be descriptive information, needs assessments, cost calculations and process evaluations. First, answering these additional questions has an added value in itself. Second, the additional knowledge can be useful when interpreting the results of an RIE. Additional knowledge was either generated by additional data (often qualitative data within a mixed-methods design) or by extensive analysis of baseline and follow-up data from the RIE. This approach is increasingly followed by USAID and J-PAL as well as IPA and DIME.

Potential solution Set incentives for RIE

Next to formal regulations, setting incentives can enhance the initiation of RIEs, for example by including the initiation of RIEs or the take-up of evidence in annual staff targets, management board goals, only scaling up interventions that have been rigorously evaluated, or giving visibility to evidence champions. Giving visibility to evidence champions could be done by organising events in which best practices are presented, or by setting up project awards that honour evidence-generation through RIE. To incentivise researchers to conduct policy-relevant RIEs, tracking the policy influence of research as an important criterion for judging research quality can promote a cultural change over time (Smith et al., 2020).

Demand

Awareness and Capacity

Limited RIE capacity

Limited RIE capacity is a major barrier for RIE initiation among both project implementers and policy makers. A lack of RIE capacity makes it difficult for RIE commissioners (both at project and policy level) to anticipate what kind of insights and benefits RIEs can generate, how to commission them and how to conduct quality assurance. Due to limited RIE knowledge, RIEs are often not considered at the beginning of a project. However, this phase is critical to the implementation of a sound evaluation design within an RIE approach.

Potential solutions
Potential solution Build RIE capacities

All data sources and all stakeholder groups name developing RIE capacities as a high‑priority measure. Capacity development measures should not only target decision makers and employees within GDC, but also include evaluation capacity development (ECD) in partner countries. At the same time, GDC can learn from partner countries like Mexico, Costa Rica and South Africa that have a long history of, and strong expertise in, evidence-based policy making. Developing RIE capacities should follow the systemic approach, targeting individuals, networks, institutions and the enabling environment. Topics can cover the following aspects: fundamental conceptual knowledge (“What is impact evaluation and what is it good for?”), methodological knowledge, RIE procurement and management, practical RIE implementation knowledge and knowledge on evaluation management and dissemination.

Potential solution Create opportunities for RIE exchange

Creating opportunities for networking and exchange of RIE experiences within but also beyond their own organisation can be a potential measure to foster RIE initiation. This can take the form of peer exchange, research–practice exchange or policy–practice exchange, and includes reflections on practical implementation questions as well as existing evidence gaps.

Potential solution Develop guidance tools for RIE initiation

Having clear guidelines on when and how to initiate an RIE (i.e. an RIE evaluability assessment) can support the initiation of RIE. Such guidance tools should cover aspects of practical and methodological feasibility as well as the usefulness of RIE in a given project and context. This measure is particularly useful to foster a more systematic initiation of RIE, as international experiences, for example at USAID, Oxfam GB, FCDO and IDB suggest. A practical guidance tool, developed by the DEval RIE project, can be found here.

Potential solution Strengthen (existing) support structures for RIE

Strengthening support structures for RIE, which can either provide direct support or link those looking for support with external structures (e.g. researchers or consultancies), is a highly relevant measure to facilitate RIE initiation. Such support structures can either be internal or external and can offer services such as project-based consulting, design clinics or matchmaking workshops. GIZ and KfW Development Bank have recently established such support structures (Bräuer et al, 2020: 80; KfW, 2021).

Awareness and Capacity

Feasibility

Methodological challenges

Methodological challenges when setting up an RIE are one of the most important factors hindering RIE initiation in GDC. Challenges regarding methodological feasibility include:

“contamination” of the control group (meaning that a fraction of the control group also receives the treatment even though it should not) and spill-over effects (meaning that positive effects of the treatment among the treatment groups have positive knock-on effects on the control group)

availability of, access to and quality of data

lack of statistical power due to few available units of observation (“small or medium N”) and

difficulties in constructing treatment and control groups.

Potential solutions
Potential solution Allow for pragmatic RIE solutions

The willingness of project teams, administrative staff and researchers to find pragmatic solutions to challenges during RIE implementation can be a facilitating factor for initiating RIE. Finding pragmatic RIE solutions can mean realising an RIE at short notice and generating findings more quickly (e.g. through real-time data analysis or rapid evaluations) or deviating from the experimental “gold standard”, while pursuing high‑quality statistical analyses (e.g. using quasi-experimental designs). It can also mean collecting context-relevant data remotely or exploiting synergies regarding data.

Potential solution Build RIE capacities

All data sources and all stakeholder groups name developing RIE capacities as a high‑priority measure. Capacity development measures should not only target decision makers and employees within GDC, but also include evaluation capacity development (ECD) in partner countries. At the same time, GDC can learn from partner countries like Mexico, Costa Rica and South Africa that have a long history of, and strong expertise in, evidence-based policy making. Developing RIE capacities should follow the systemic approach, targeting individuals, networks, institutions and the enabling environment. Topics can cover the following aspects: fundamental conceptual knowledge (“What is impact evaluation and what is it good for?”), methodological knowledge, RIE procurement and management, practical RIE implementation knowledge and knowledge on evaluation management and dissemination.

Potential solution Develop guidance tools for RIE initiation

Having clear guidelines on when and how to initiate an RIE (i.e. an RIE evaluability assessment) can support the initiation of RIE. Such guidance tools should cover aspects of practical and methodological feasibility as well as the usefulness of RIE in a given project and context. This measure is particularly useful to foster a more systematic initiation of RIE, as international experiences, for example at USAID, Oxfam GB, FCDO and IDB suggest. A practical guidance tool, developed by the DEval RIE project, can be found here [?]

Potential solution Strengthen (existing) support structures for RIE

Strengthening support structures for RIE, which can either provide direct support or link those looking for support with external structures (e.g. researchers or consultancies), is a highly relevant measure to facilitate RIE initiation. Such support structures can either be internal or external and can offer services such as project-based consulting, design clinics or matchmaking workshops. GIZ and KfW Development Bank have recently established such support structures (Bräuer et al, 2020: 80; KfW, 2021).

Uncoordinated timing between project and RIE

Lack of harmonisation in the timing of the project and the RIE is a critical barrier to RIE implementation. RIE and projects usually have different time horizons (e.g. RIEs can only be contracted after the project has been formally approved). This substantially reduces incentives to conduct RIEs, because RIE cannot be financed from project funds beyond the project duration. In addition, it may be unclear whether RIE results can still be utilised for the current project cycle.

Potential solutions
Potential solution Create awareness for RIE

To foster the initiation of RIEs within GDC, it is important to create awareness of RIE among GDC stakeholders. This can mean pointing out specific uses of RIE, particularly the causal attribution of impacts to interventions. It can also mean pointing to the potential of a solid impact communication to the public or to the fact that the costs of projects with low actual impacts are higher in the long run than costs associated with RIE. Creating and publicising positive RIE examples, emphasising the international relevance of RIE in the sectoral dialogue, and engaging in open and honest dialogue about ethical concerns can also contribute to awareness building. Awareness building among employees in leadership positions may be particularly beneficial as it signals support and creates demand.

Potential solution Ensure early coordination of RIE and project cycles

To facilitate RIE initiation, it is important to systematically check at early stages in project cycles for the possibility of conducting an RIE (e.g. using an RIE evaluability assessment tool). This can be done by integrating RIE reflection points in existing process flows and in guidelines about how to create module proposals or brief statements. In practice, early coordination between the project cycle and a possible RIE could be achieved by involving RIE experts already in the project proposal or design phases (after commissioning), or proposing an RIE as a component of the commissioned project.

Potential solution Pass formal legislation or requirement

Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. The United States’ Evidence Act, Mexico’s Social Development Law (White, 2019) or South Africa’s mandatory system of evidence assessment provide examples hereof. Instead of national legislation, RIE initiation could also be formalised by requirements passed by the respective government department or by binding organisational instructions. Our data shows that, whereas quota regulations or thresholds for RIE were largely deemed unhelpful, top-down requirements or clear donor signals were generally judged quite effective in establishing a more systematic use of RIE. In order to encourage the initiation of RIE, formal requirements could require project implementers to conduct:
RIEs in projects above a certain budget (threshold)
a number of RIEs per year in each sector (quota)
RIEs within (annually) changing thematic windows
RIEs in all pilot projects or in any project that is intended to be scaled-up
a standardised check for RIE feasibility and usefulness in existing process flows and respective internal guidance documents

Feasibility

Resources

Limited time and financial resources

A lack of resources is one of the primary barriers to RIE initiation. Lack of time was mentioned as the most relevant factor hindering RIE initiation among respondents to the evidence survey, even more so than financial resources. In concrete terms, issues mentioned were high workloads or lack of dedicated RIE staff or time allotments. Further, RIEs are relatively expensive compared to other types of evaluations. Therefore, they often require additional funding or a reallocation of funds from planned project activities.

Potential solutions
Potential solution Allocate financial resources and time to RIE

Allocating larger shares of the available working time and sufficient personnel to monitoring and evaluation, including RIE, is one of the most frequently mentioned potential measures to foster RIE initiation within GDC. Further, additional financial resources were among the most relevant potential measures to facilitate RIE initiation. Financial resources can be provided by topping up project budgets (without taking away resources from planned activities), partnering with other donors or by establishing a central fund or support programme for RIE that directs financial resources into strategically relevant topics.

Potential solution Allow for pragmatic RIE solutions

The willingness of project teams, administrative staff and researchers to find pragmatic solutions to challenges during RIE implementation can be a facilitating factor for initiating RIE. Finding pragmatic RIE solutions can mean realising an RIE at short notice and generating findings more quickly (e.g. through real-time data analysis or rapid evaluations) or deviating from the experimental “gold standard”, while pursuing high‑quality statistical analyses (e.g. using quasi-experimental designs). It can also mean collecting context-relevant data remotely or exploiting synergies regarding data.

Resources

To Top