Strong Designs: Experiments
The designs listed on this page require an experienced evaluator to carry them out. They are also likely to be expensive and time consuming. They are, however, the strongest type of evaluation which means there is more confidence about whether the changes seen were due to the intervention.
There are different types of experimental design but all involve the random allocation of participants into those who receive the intervention, and those who do not.
Randomised Controlled Trial (RCT)
A randomised controlled trial (RCT) with two groups of participants is a very robust evaluation option. One group of participants receives the intervention (the intervention group) and another group of participants (the control group) receives no intervention.
Individual participants are randomly assigned to their groups (intervention or control) so that both groups are representative of the target population. Both groups are tested before and after the intervention takes place. The randomisation of participants to intervention and control groups makes it an experimental design.
Strengths
- An RCT helps to ensure that any difference found between the groups is due to the absence or presence of the intervention, and not other factors such as individual personality traits or geographical location.
Weaknesses
- RCTs are complex and expensive to set up.
- There can be ethical issues to address. For example is it fair for one person to receive the intervention while a person in similar circumstances does not? One way to overcome ethical concerns is to offer the control participants the successful intervention at a later stage so that they are not disadvantaged by being in the control group. On the other hand, if the intervention makes things worse, were the participants disadvantaged in any way?
- It can be difficult to retain contact with your participants over time and your findings may be skewed depending on the characteristics of those who 'drop out'.
- The pre-test questions may, to some degree, affect the post-test results as the pre-test questions may prime the participants, educate them about the issue under study, or they may think that the researchers are expecting different answers in the post-test.
Cluster randomised controlled trial
A cluster RCT is where whole groups of participants are randomised, instead of individual participants being randomised to intervention or control groups. It is suitable for ETP interventions delivered to institutions such as schools, rather than to individuals.
The procedure is similar to that for RCTs but the sample size calculations and the way the data are treated vary. You will need expert statistical support if you use this design. For example, an intervention could be aimed at learner drivers. To form a sample, instead of randomising individual learner drivers to the intervention or control group, driving schools and independent instructors in the local area could be randomised instead. For example: 10 driving instructors (the cluster) could be randomly allocated to the intervention group so that all drivers learning with them form the intervention group. Another 10 driving instructors (another cluster) are randomly selected to the control group, so that all drivers learning with those 10 instructors form the control group.
Strengths
- This method recognises that individuals within a given group or institution will all receive a similar experience and so the data collected should be treated statistically differently to data collected from individuals who have been recruited for an intervention.
Weaknesses
- The disadvantage of cluster RCT's is that the individuals within groups are likely to be similar by way of their membership of the clusters. For example, all drivers with one particular instructor are likely to have received very similar advice and this advice may affect the delivery and impact of the intervention.
- It can be difficult to retain contact with your participants over time and your findings may be skewed depending on the characteristics of those who 'drop out'.
After intervention only with a control group
In an after intervention only evaluation with a control group, one measurement is taken of each group of participants. One group of participants receives the intervention (the intervention group) and another group of participants (the control group) receives no intervention. The measurements are taken after the intervention has been delivered.
Individual participants are randomly assigned to their groups (intervention or control) so that both groups are representative of the target population. With a randomised control group an after only design is a robust experimental approach.
Strengths
- If participants are randomly allocated to the intervention or control group (randomised) then any differences between the two groups to begin with will have been minimised to some extent. This means that as the two groups were similar to begin with, and any differences observed after the intervention has taken place, are likely to have been caused by the intervention.
- As you are only taking one measurement from each group, you do not need to retain contact with participants for a repeat survey.
- Without taking measurements before the intervention takes place the evaluation will take less time so there is less chance of participants 'dropping out' and biasing the results. For instance, those people who drop out may be a different type of person to those who do not.
- With randomised groups stronger statistical tests can be conducted in the data analysis.
Weaknesses
- Even with randomisation, without a before measure you cannot be confident that the two groups really were equivalent to begin with, i.e. that the only difference between them was the presence or absence of the intervention.
- Assigning participants to intervention and control groups is not always possible, or desirable, in real-life situations.
- Also, not all factors which may affect the intervention outcomes can be controlled for.