Type of Evaluation
Evaluation design means the type of evaluation that you choose to do, and also the timing of when you take measurements, e.g. before and after the road safety project takes place. It does not refer to discussions of data collection methods such as surveys or interviews. For more information on methods please see the help page on Ways to Collect Data.
The first question you should ask yourself when thinking about what evaluation design to use is: What is the purpose of the evaluation?
Spend time thinking about why you are evaluating, and how you want to use the results of the evaluation. Do you want to:
- Improve your intervention?
- Demonstrate the effectiveness of your intervention?
- Both improve and demonstrate effectiveness?
Improving the Intervention
An evaluation to identify how to improve a road safety project or intervention usually takes place when the project is first set-up. The intervention may not have started yet or a few ideas may have been trialled.
To understand how to improve the project you need to examine how the intervention is carried out, including its inputs and outputs. Inputs are all of the resources that go into the road safety project; such as money, staff time or expertise. Outputs are anything that is directly created as a result of the inputs; such as a leaflet or a number of presentations that are delivered. Describe these inputs and outputs in your logic model or project plan. You are, therefore, investigating the day-to-day mechanics of how the intervention is prepared and carried out.
Although the main focus of this type of evaluation is to investigate the day-to-day mechanics of how the intervention is prepared and carried out in order to identify improvements, ideally you would also examine some outcomes. Outcomes are not to be confused with outputs (although the similarity in their name doesn't help matters). Outcomes are more about the end-game and are usually the reason why the intervention is taking place. The outcomes in road safety education, training or publicity projects are usually:
- To increase awareness or improve knowledge
- To improve attitudes
- To improve behaviours
However, it is likely that you will only be able to measure short-term outcomes in this type of evaluation. This means measuring knowledge, attitudes or intended behaviour change, immediately or soon after the intervention has taken place. The advantage of examining some outcomes is that you can establish weaknesses in the project's content which will require improvement. For example, if a particular message in an educational workshop is not well remembered then it will need to be improved.
Measuring Effectiveness
Does the road safety project work?
When there is a focus on measuring effectiveness, the project may have been running for a while, or effectiveness may now be being considered after some initial testing on the design and delivery of the project.
This type of project will need to measure outcomes. For road safety education, training or publicity projects, outcomes are usually:
- To increase awareness or improve knowledge
- To improve attitudes
- To improve behaviours
At minimum these outcomes should be measured in the short-term. This may mean having the audience take part in the evaluation immediately after they have been involved in the intervention. Ideally, however, there should also be some further follow up, to see if any effects of the intervention are sustained over time.
In addition to measuring outcomes, ideally processes will be examined. These are the day-to-day mechanics of how the intervention is prepared and carried out. This investigation can provide clues as to why the project was, or was not, successful. For example, the outcomes may reveal that students found it difficult to remember a specific topic area. The process information may indicate, however, that workshop leaders found it difficult to cover all of the material in the time allocated.
Common ways to measure outcomes
Outcome evaluation designs are the different ways in which you can collect data to measure if the intervention has been effective in improving knowledge, attitudes and/or behaviours. They are used when you would like to put a number on how effective a project was (also known as quantitative designs, quant relating to quantity or number). For example, road safety attitudes were improved by 20%.
Outcome evaluation designs fall into three main categories:
- Experimental (most robust)
- Quasi-experimental
- Non-experiment (least robust)
The category that will be chosen relies on a number of factors. The evaluator should conduct the most robust (or strongest) type of evaluation they can, within the resources that are available. show more >
Experimental designs are the most robust but they also require the most resources. They should be undertaken by an experienced evaluator and are likely to be the most expensive and time-consuming designs. Non-experimental designs are the least robust, which means it is difficult to know if any benefits were really due to the road safety intervention. They are, however, often the cheapest and easiest to do. Non-experimental designs, therefore, may be a good starting point for those who are not confident in evaluation. Quasi-experimental methods are often chosen because they can achieve some level of robustness but tend to be cheaper and easier to run than experimental designs.
Experimental designs involve two groups of participants – the group receiving the intervention, and a control group who do not. The researcher randomly allocates participants to one of the two groups (this is known as ‘randomisation.). As they are randomly selected it means that each potential participant has an equal chance of being in either the intervention group or the control group. This randomisation helps to minimise selection bias, i.e. a bias behind what group the researcher or the audience put themselves into. For example, those already concerned with safety are more likely to put themselves forward to go to a road safety workshop. This means there will already be a difference between those who attend a workshop and those who do not, even before the workshop begins.
Quasi-experimental designs are like an experiment in that they have two groups of participants: one who receives the intervention, and one who does not. However, participants are not randomly allocated to the groups. For this reason the non-intervention group is called a 'comparison' group instead of a control group. A comparison group is a convenience group of participants who share some of the characteristics of those receiving the intervention.
For example: a comparison group could be the same year group of school children as those receiving the intervention, but from a different school in a similar geographical environment.
If you deduct the total change in the control or comparison group, from the total change in the intervention group, you will find the amount of change that can be attributed to the intervention (or in other words: the net change).
Non-experiment designs involve neither a control nor a comparison group. They consist only of participants who receive the intervention. This type of design is a lot quicker, cheaper and easier to conduct but it is also the least robust. Without a control or comparison group you cannot say for certain that any change found, from the before and after measurements, was caused by the intervention. With a control or comparison group you can see the change that would have happened, had the intervention not been present.
Doing Both: Making improvements and Measuring Effectiveness
It is likely that the purpose of an evaluation, will be to examine both where improvements can be made to the design and delivery of the project, and to measure effectiveness.
If this is the case both the day-to-day mechanics of how the intervention is prepared and carried out and the outcomes (changes to knowledge, attitudes and/or behaviours) will need to be examined in detail. The sections above (Improving the Intervention and Measuring Effectiveness) provide more information on what each part requires.
Evaluation Design Exercise
For an exercise on evaluation design, suitable for an evaluation training event, please see our Workshop Materials page.