What is Rigorous Impact Evaluation (RIE)?
On this page, you learn what rigorous impact evaluation (RIE) is all about and why it is useful.
In a nutshell, RIE is an evaluation approach that uses a control or comparison group to evaluate whether an intervention works or not. But let’s be more specific:
RIE comprises a set of different evaluation designs (experimental and quasi-experimental designs). It is an approach that allows the causal attribution of a change in an outcome of interest, for example household income, to a specific intervention, for example microloans. To do so, it is necessary to compare what actually happened with what would have happened without the intervention, the so-called “counterfactual situation”.
In our example, the counterfactual situation would be to compare the income of households that received a microloan intervention to what the income of the very same households would have been if they had not received the microloan. Without such a counterfactual, we cannot say with certainty, that it was our intervention that caused the impact and not some other external factors. We might not be certain that the household income changed due to the microloans and not due to lower taxes or better infrastructure that happened at the same time like the microloan.
Of course, it is logically impossible to observe the same households receiving and not receiving the intervention. This is why this counterfactual situation is approximated constructing a so called “control or comparison group”. That group is constructed in a way that it is as similar to the intervention group as possible. To construct such a group experimental and quasi-experimental designs come into play.
In an experimental design, also called randomised controlled trial (RCT), we use randomisation to assign our observation units either to 1) the intervention group (that “receives” the intervention) or 2) to the control group (that does not “receive” the intervention). In our example, our observation units would be households, but they could also be something like schools, villages or children under five.
Random assignment of a sufficiently large number of observational units (e.g. households) ensures that prior to the intervention the two groups are on average identical in terms of their observable and non-observable characteristics. The only dimension, where the groups differ, is whether or not they received the intervention. Thereby, any difference in the outcome between the two groups after the intervention can be attributed to our intervention.
Quasi-experimental designs do not use randomisation but instead apply other specific study designs or statistical methods to estimate the counterfactual situation. Quasi-experimental designs include regression discontinuity designs, different matching techniques, difference-in-differences estimation, interrupted time series, instrumental variable approaches and fixed effects models. They can also often be applied when the intervention has already started, whereas RCTs must be prepared before the intervention has started.
Recent discussions in the evaluation literature emphasise that it is important to understand not only whether an intervention has an effect. It is also important to determine the way in which it produces the effect and under what circumstances the effect occurs (the so-called causal mechanism driving the effect; see Schmitt (2020) for an overview). RIE are therefore often most insightful if they are based on theory and are combined with qualitative components (White, 2009).
Systematic reviews (SR) and evidence gap maps (EGM) are closely connected to the concept of RIE. They are explained here (SR) and here (EGM).