What barriers hinder the use of existing RIE evidence?

In an empirical research project DEval anlaysed which barriers hinder the use/take-up of RIE evidence and which potential solutions could address those barriers. Please note that our potential solutions are called measures in our report. In our interactive illustration below, those barriers and solutions are organised along six critical steps for evidence generation:

1. Demand for evidence,  2. Availability of evidence,  3. Awareness and capacity for evidence,  4. Resources to use evidence,  5. Evidence is accessible and useable  6. Intention to do an RIE

Demand

Low priority of evidence for decision-marking

There is little (though increasing) demand for (RIE) evidence from different stakeholders, most importantly from donors (i.e. BMZ) and partners. This is accompanied by the observation of interviewees that other types of evidence (e.g. from process evaluations or expert opinions) are often ascribed equivalent or higher utility and quality.

Potential solutions
Potential solution Create awareness for evidence take-up

Creating awareness for RIE, SRs and EGMs is central to increasing take-up of rigorous evidence. When creating awareness for the take-up of global evidence, it is particularly important to address the widespread belief that the global RIE evidence base does not lend itself to the GDC portfolio. Our analysis of 3ie’s DEP found that it contains a large number of studies that are indeed potentially relevant to the GDC portfolio.

Potential solution Create opportunities for RIE exchange

Creating opportunities for RIE exchange can be a potential measure to increase evidence take-up, as direct, personal interaction is often more effective in fostering evidence take-up than disseminating written documents. Exchange can happen as peer-exchange, research-practice-exchange or policy-practice-exchange. The yearly BMZ-funded conference of the “Poverty reduction, equity and growth network (PEGNet)” is an example of such an event where research and practice meet to present and discuss new RIEs.

Potential solution Demonstrate leadership commitment to RIE

Leadership commitment and giving priority to evidence-informed decision making is an important measure to foster evidence take-up. Leadership commitment can be expressed by giving priority to evidence generation and take-up, by functioning as role models and explicitly using RIE findings for decision making or by rewarding evidence generation and take-up.

Potential solution Include the take-up of rigorous evidence in the organisation’s lead strategy

Organisations should commit to evidence-informed decision-making in their lead strategy to facilitate the take-up of self-generated and global RIE evidence. Such strategies provide orientation and help anchor RIEs during changes in leadership. BMZ’s recent strategy document for evaluation in GDC is a good example of such an effort in that it explicitly encourages impact orientation and the role of accompanying research such as RIEs within GDC (BMZ, 2021).

Potential solution Pass formal legislation or requirement

Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. In concrete terms, this could be done by explicitly including the obligation to draw on rigorous evidence in existing process flows and respective guidance documents. Referring to scientific studies in the formal reporting documents would allow stakeholders, including BMZ employees, to verify the lines of reasoning. Formal requirements could also regulate the dissemination of results. Formal legislation and requirements for evidence take-up are judged heterogeneously. Although an explicit requirement can potentially be highly effective, there is also a chance that a requirement would not be practically feasible or not taken seriously. Further, formal requirements need to be combined with other measures such as resources and capacity development to be effective.

Potential solution Set incentives for evidence take-up

Next to formal regulations, setting incentives can enhance evidence take-up, for example by including evidence take-up in annual staff targets, or giving visibility to “evidence champions” who push for evidence take-up. This could be done, for example, by granting evidence awards, such as the Africa Evidence Leadership Award (Africa Evidence Network, 2021). To incentivise researchers to conduct evidence syntheses, it would be beneficial to recognise SRs as an adequate contribution to a PhD thesis.

No formal requirement for the take-up of rigorous evidence

There is no explicit requirement or formally structured process to use the global RIE evidence base for policy making, programming and project implementation. Though the BMZ guidance documents implicitly require the take-up of self-generated evidence, which can include RIE evidence, there is no requirement to make use of the global RIE evidence base.

Potential solutions
Potential solution Pass formal legislation or requirement

Formal legislation or requirements at the national level can be potential measures to increase RIE initiation. In concrete terms, this could be done by explicitly including the obligation to draw on rigorous evidence in existing process flows and respective guidance documents. Referring to scientific studies in the formal reporting documents would allow stakeholders, including BMZ employees, to verify the lines of reasoning. Formal requirements could also regulate the dissemination of results. Formal legislation and requirements for evidence take-up are judged heterogeneously. Although an explicit requirement can potentially be highly effective, there is also a chance that a requirement would not be practically feasible or not taken seriously. Further, formal requirements need to be combined with other measures such as resources and capacity development to be effective.

Potential solution Set incentives for evidence take-up

Next to formal regulations, setting incentives can enhance evidence take-up, for example by including evidence take-up in annual staff targets, or giving visibility to “evidence champions” who push for evidence take-up. This could be done, for example, by granting evidence awards, such as the Africa Evidence Leadership Award (Africa Evidence Network, 2021). To incentivise researchers to conduct evidence syntheses, it would be beneficial to recognise SRs as an adequate contribution to a PhD thesis.

Potential solution Systematically consult the RIE evidence base when designing projects

A systematic consultation of the global and self-generated RIE evidence base early on when designing projects can foster take-up of global RIE evidence. A systematic anchoring could happen by integrating reflection points in existing process flows and in guidelines about how to create module proposals or brief statements. Those reflection points can ensure that state-of-the-art RIE evidence supports the proposed project. AFD, IDB, Oxfam GB, WWN and J-PAL perform such reflective learning sessions. Similarly, USAID, Oxfam GB and FCDO reported having long-term formalised partnerships with universities and think tanks to provide inputs for project design using global evidence.

Demand

Availability

Limited relevance of the intervention or the research question

The take-up of RIE results can be hindered when the intervention evaluated or the research questions asked are not sufficiently relevant. The perception of relevance can however differ across stakeholders. In practice, at organisational and policy level, evidence take-up was hindered for interventions of little political or strategic relevance (e.g. when political priorities had changed over time) or low innovation potential. On project implementation level, questions supporting short-term project steering were perceived as most relevant.

Potential solutions
Potential solution Ensure close cooperation between researchers and other stakeholders

Close cooperation between researchers and other stakeholders is a central measure to enhance take-up of self-generated and global RIE.

Potential solution Generate additional knowledge beyond the RIE

Generating additional knowledge beyond the results from the RIE can be an effective measure to increase evidence take-up. This might be descriptive information, needs assessments, cost calculations and process evaluations. First, answering these additional questions has an added value in itself. Second, the additional knowledge can be useful when interpreting the results of an RIE. Additional knowledge was either generated by additional data (often qualitative data within a mixed-methods design) or by extensive analysis of baseline and follow-up.

Potential solution Identify interventions and evaluation questions of operational, strategic or political relevance

Identify interventions and evaluation questions of operational, strategic or political relevance.

Unclear transferability (external validity) of single RIEs

Limited external validity of single RIEs is a frequently mentioned barrier to the take-up of global RIE evidence. In practice, it was often assumed that results from RIEs conducted in one context and on one population were not transferable to other contexts or populations and therefore existing RIEs – even if the same intervention was looked at – were not relevant. This is a prominent barrier both at the project level and for strategic decision making. At the strategic level, in particular, general lessons are of interest rather than context-specific insights.

Potential solutions
Potential solution Build RIE take-up capacities

Capacity development is a frequently mentioned potential measure to increase the take-up of both self-generated and global RIE evidence. Capacity development for self-generated and global evidence should comprise:


Self-generated RIE evidence:


Trainings for interpreting the results of RIEs and their integration into project planning and policy advice can be particularly helpful to facilitate their actual take-up.


Global RIE evidence base:


Capacity development measures for increasing the take-up of global RIE evidence should focus on how to find, assess, interpret and integrate relevant RIEs and evidence synthesis products into project planning and policy making. For researchers and evaluators, the focus should be on the adequate communication of findings. For policy makers and programme implementers, capacity development measures should be very practical and tailored to needs and working realities (e.g. formats should be short).

Potential solution Synthesise and further condense the body of evidence

Synthesising the global evidence base is the most relevant measure to increase take-up of global evidence. SRs, followed by EGMs, were mentioned as effective synthesis tools next to portals of the WWN, such as the teaching and learning toolkit from the Education Endowment Foundation.1 Unfortunately, such high-quality synthesis tools, including SRs and EGMs, are largely unknown in GDC and their production requires time. A synthesis strategy of several years that gives priority to a certain sector each would hence be helpful. Evidence syntheses could even be coordinated internationally in order to provide the entire development community with these products in a cost-effective manner.

 

Availability

Awareness and Capacity

Scarce and unsystematic dissemination of findings

Results of self-conducted RIEs are often scarcely and unsystematically disseminated within the organisation and beyond. In practice, for most of the RIEs no dissemination concept existed. Hence, dissemination activities of self-generated RIE findings were mostly driven by the motivation of individual GDC employees or researchers.

Potential solutions
Potential solution Build a network of knowledge

Build a network of knowledge

Potential solution Establish a comprehensive and active dissemination strategy

Establish a comprehensive and active dissemination strategy

Potential solution Present findings in an illustrative way

Present findings in an illustrative way

Awareness and Capacity

Resources

High staff turnover

High staff turnover is a frequent barrier to the take-up of self-generated evidence. Employees with intrinsic motivation for evidence-based programming, who had initiated an RIE, were sometimes not working in the respective project anymore when the RIE was completed. This results in a lack of responsibility and ownership of take-up of RIE findings once they become available.

Potential solutions
Potential solution Build RIE take-up capacities

Build RIE take-up capacities

Potential solution Define roles and responsibilities

Define roles and responsibilities

Limited time resources

Being structurally overworked and having no time to search, read, assess, and process the existing evidence is one of the most frequently mentioned barriers. Even research-oriented employees, who have a good evidence-literacy and know where to find relevant scientific evidence, reported not having time to engage with the existing evidence base.
Yes, one of the main obstacles is certainly time. We work under extreme time pressure. And I think that many colleagues would like to use much more evidence and take much more time to read studies and so on when proposing projects. That’s what everyone actually says. (Int_9)

Potential solutions
Potential solution Build a network of knowledge

Build a network of knowledge

Potential solution Demonstrate leadership commitment to RIE

Demonstrate leadership commitment to RIE

Potential solution Present findings in an illustrative way

Present findings in an illustrative way

Potential solution Provide evidence on a demand-driven basis

Provide evidence on a demand-driven basis

Potential solution Synthesise and further condense the body of evidence

Synthesise and further condense the body of evidence

Unclear responsibility for ensuring evidence take-up at the organisational level

Unclear responsibilities hinder the take-up of rigorous evidence at the organisational level. Interviewees from BMZ consider themselves to be generalists and argued that the IOs should be on top of the latest scientific evidence. In contrast, employees of IOs reported that they were no experts in research but in project implementation and that keeping track of the latest and growing body of scientific evidence within broad and multiple topics would not be possible due to their high workload. International experts argue that it is neither the core competence of policy makers nor of project implementers to keep track of the scientific evidence. Though sporadic university cooperation exists, researchers are not an integral part of the GDC system, and explicit positions of “applied researchers”, “chief economists”, “chief scientific advisors”, “scientific knowledge brokers” or research departments do not exist in GDC organisations.
[…] I find it’s very unfair to project teams to expect them to be experts in research. Researchers are expert in research. That’s the truth of it. Research evolves very quickly. There’s so many studies at all times. You have to keep up with the literature. You have to read and be informed. And researchers are professionals of evidence. And so the idea that somebody who works in development should know all the evidence and be able to incorporate that evidence in their own project I think is unfair. (Int_Exp_1)

Potential solutions
Potential solution Build a network of knowledge

Building a network of knowledge brokers is by far the most relevant potential measure for increasing take-up of global evidence. Knowledge brokers can take on many of the measures that foster take-up of global evidence, including education and training, production or commissioning of evidence synthesis products or illustrative presentation of findings. There are already examples of existing structures of knowledge brokers at FCDO or NORAD (Evans et al., 2021). Knowledge brokers should keep a close relationship to the potential end-users, thereby gaining a good understanding for whom different pieces of evidence might be relevant and what the most pressing evidence questions are.

Potential solution Create awareness for evidence take-up

Creating awareness for RIE, SRs and EGMs is central to increasing take-up of rigorous evidence. When creating awareness for the take-up of global evidence, it is particularly important to address the widespread belief that the global RIE evidence base does not lend itself to the GDC portfolio. Our analysis of 3ie’s DEP found that it contains a large number of studies that are indeed potentially relevant to the GDC portfolio.

Potential solution Provide evidence on a demand-driven basis

Demand-driven provision of evidence is a highly relevant potential measure to foster take-up of global RIE evidence. Such a demand-driven model is, for instance, implemented in Demark, Norway and Sweden. They have publicly funded units that actively and regularly engage with policy makers to discuss findings and identify evidence needs for policy making and programming (White, 2019). Other examples are the West Africa Capacity-building and Impact Evaluation (WACIE) Helpdesk and the countries of South Africa, Uganda and Zimbabwe, which also have rapid response services embedded within their governments. The timely provision of evidence is a precondition for the success of this measure, meaning that either rapid syntheses are required or that the demand is expressed early.

Resources

Access and Usability

Missing knowledge management

The take-up of self-generated RIE evidence at an organisational level can be hindered by the lack of an easily accessible knowledge management system for completed, ongoing and planned RIEs. Our data shows that RIEs were mostly initiated by decentral units (e.g. in partner countries) and central units rarely kept track of the initiated RIEs and their results. The lack of such a system not only hinders the take-up of RIE evidence and sharing of lessons learnt but also makes project implementers and policy makers ask the same impact evaluation questions over and over again. Organisational knowledge management systems are particularly relevant for organisations with high staff turnover.

That’s why the question for us is: Where is our institutional memory? […] Where is it actually stored in BMZ? It can’t be that we keep commissioning the same thing and no one knows. […] People are hired and have to commission things in [country X]. How are they supposed to know whether it works or not, what the GIZ writes down? Absurd. […] It’s funny, isn’t it? No one asks. Everyone thinks that whoever is hired here can do it. (Int_35)

Potential solutions
Potential solution Improve (intra-)organisational knowledge management systems

Improving knowledge management systems is a potential measure to improve take-up of self-generated RIE evidence at the organisational level. This can take the form of a platform or unit that collects and displays data, results and experiences from RIEs that have been generated in GDC in an easily accessible and user-friendly way. KfW Development Bank has recently launched an internal database for their general evaluations that gives quick access to KfW Development Bank’s evaluation reports. GIZ is considering developing a similar tool. BMZ has anchored the development of an RIE database in its BMZ 2030 reform processes, which can be found here.

Access and Usability

Intention

Lack of culture of learning from failure

The lack of a culture of learning from failure is a frequently mentioned barrier for evidence take-up. Respondents perceived pressure by BMZ and felt the need to report positive results of projects for domestic political communication. Head of projects are required to meet 10 to 15 indicators in a three years project cycle, which is often not realistic. In addition, interviewees mentioned the challenge of dealing with negative results from RIEs. At root there is the fear of not getting additional funding (in which practitioners have a personal interest as it secures the job) if indicators are not fulfilled or the evaluation shows negative results.

Potential solutions
Potential solution Create awareness for evidence take-up

Creating awareness for RIE, SRs and EGMs is central to increasing take-up of rigorous evidence. When creating awareness for the take-up of global evidence, it is particularly important to address the widespread belief that the global RIE evidence base does not lend itself to the GDC portfolio. Our analysis of 3ie’s DEP found that it contains a large number of studies that are indeed potentially relevant to the GDC portfolio.

Potential solution Strengthen a culture of learning from failure

Strengthening a culture of learning from failure can support take-up of self-generated RIE evidence. This also includes the open and unbiased interpretation of evaluation results by researchers and evaluators. Respondents to the evidence survey asked for independent and professional interpretations and recommendations that are unpolished and do not please the commissioner or any particular stakeholder group.

Intention

To Top