What is Rigorous Impact Evaluation (RIE)?
On this page, you learn what rigorous impact evaluation (RIE) is all about and why it is useful.
In a nutshell, RIE is an evaluation approach that uses a control or comparison group to evaluate whether an intervention works or not. But let’s be more specific:
RIE comprises a set of different evaluation designs (experimental and quasi-experimental designs). It is an approach that allows the causal attribution of a change in an outcome of interest, for example household income, to a specific intervention, for example microloans. To do so, it is necessary to compare what actually happened with what would have happened without the intervention, the so-called “counterfactual situation”.
In our example, the counterfactual situation would be to compare the income of households that received a microloan intervention to what the income of the very same households would have been if they had not received the microloan. Without such a counterfactual, we cannot say with certainty, that it was our intervention that caused the impact and not some other external factors. We might not be certain that the household income changed due to the microloans and not due to lower taxes or better infrastructure that happened at the same time like the microloan.
Of course, it is logically impossible to observe the same households receiving and not receiving the intervention. This is why this counterfactual situation is approximated constructing a so called “control or comparison group”. That group is constructed in a way that it is as similar to the intervention group as possible. To construct such a group experimental and quasi-experimental designs come into play.
In an experimental design, also called randomised controlled trial (RCT), we use randomisation to assign our observation units either to 1) the intervention group (that “receives” the intervention) or 2) to the control group (that does not “receive” the intervention). In our example, our observation units would be households, but they could also be something like schools, villages or children under five.
Random assignment of a sufficiently large number of observational units (e.g. households) ensures that prior to the intervention the two groups are on average identical in terms of their observable and non-observable characteristics. The only dimension, where the groups differ, is whether or not they received the intervention. Thereby, any difference in the outcome between the two groups after the intervention can be attributed to our intervention.
Quasi-experimental designs do not use randomisation but instead apply other specific study designs or statistical methods to estimate the counterfactual situation. Quasi-experimental designs include regression discontinuity designs, different matching techniques, difference-in-differences estimation, interrupted time series, instrumental variable approaches and fixed effects models. They can also often be applied when the intervention has already started, whereas RCTs must be prepared before the intervention has started.
Recent discussions in the evaluation literature emphasise that it is important to understand not only whether an intervention has an effect. It is also important to determine the way in which it produces the effect and under what circumstances the effect occurs (the so-called causal mechanism driving the effect; see Schmitt (2020) for an overview). RIE are therefore often most insightful if they are based on theory and are combined with qualitative components (White, 2009).
Systematic reviews (SR) and evidence gap maps (EGM) are closely connected to the concept of RIE. They are explained here (SR) and here (EGM).
For a quick introduction to RIE you can also watch our video:
What practitioners say
Experts from German Development Cooperation share their experience with RIE.
Ann-Kathrin Hentschel and Jelena Stojanovic
Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH
What motivated you to conduct an RIE?
To promote labour-market oriented technical and vocational education and training (TVET), we wanted to investigate how graduates of modernized dual TVET programmes that, among others, put a stronger focus on active participation in company training transition from school to work compared to their peers. Therefore, we decided to compare labour market outcomes of those graduates with two comparison groups of graduates from non-modernized programmes: graduates that had attended TVET programmes at the same schools but in different thematic areas and graduates at different TVET schools but in comparable thematic areas.
How has the RIE benefited your work and informed decision-making?
The main benefit of the RIE was that it showed that the project was on the right track. Students of modernised TVET programmes held higher quality jobs, more often got their first job in the company where they did their practical training, earned higher salaries, and were more likely to report that they use the skills and knowledge acquired during TVET training. Being the first rigorous study conducted for TVET profiles with dual training elements in Serbia, it constituted a strong foundation for decision making of the partners as well as the promotion of TVET with dual training elements.
Could you share one particular insight you gained regarding the usefulness of the evaluation?
The RIE showed that compared to their peers, the graduates of the TVET profiles that are implemented with strong company involvement, held better-paid jobs and are more likely to work in the area they were trained in. This evidence underlined the assumption that TVET with dual elements is more oriented to the needs of the labour market and that it allows for a smoother transition from school to work. This insight might not have been possible without having a RIE, as a less rigorous tracer study would have only monitored employment effects of those graduating from TVET profiles with dual elements and not their peers.
What would you recommend others who are planning to conduct a RIE?
Conducting an RIE gives projects an opportunity to gain a better understanding of which interventions lead to a desired outcome. To make the most of this opportunity, I would recommend to design the RIE from the very beginning, that is in the project design phase. Choosing the methodology and explaining its implications to collaborators, e.g. the need to collect data for both modernized and comparison programmes takes time. Overall, the continuous involvement of partners and partner organisation, e.g. through a working group, was crucial for the success of our RIE. Furthermore, rigorous approaches may need to be included in the budget plan as required human and financial resources often exceed the usual scope of the project’s results-based monitoring. Finally, we learned that there are a lot of aspects to be considered with regard to the right way of motivating and interviewing study participants, particularly of the comparison group.