What should I measure?
The measurements taken in an evaluation will depend on what you are looking to achieve from it. To demonstrate if a road safety project has been effective – i.e. there has been an improvement in knowledge, attitudes and/or behaviour – then measurements of these outcomes need to be a part of the evaluation. Alternatively, if the evaluation seeks to determine where improvements to the delivery of the road safety project can be made then outputs should be measured.
Measuring Effectiveness - Outcomes
Outcomes are the changes that result from a road safety project - a change in knowledge, attitudes or behaviour, for example. They are not the same as outputs (see below).
Outcomes for each road safety intervention should be defined to understand the project’s purpose. They should relate to the project's aims and objectives so by measuring outcomes you will be able to determine if the aims and objectives have been achieved. Your outcomes should form a part of your Road Safety Project Plan (Logic Model).
Casualty Reduction
It is unlikely that you will be able to link your particular road safety project to a reduction in road casualties. It is, therefore, not advised to try to measure the effectiveness of your programme through casualty reduction rates because:
- You cannot pinpoint whether, and how much of, the reduction was due to your project rather than other external factors e.g. a national road safety adverts, changes in weather, another local road safety project or improvements in vehicle design.
- Local statistics usually change so little from year to year it's difficult to tell whether the change that occurred, in the community you targeted, was greater than what would be expected by chance.
To link a particular intervention to changes in casualty reduction it is likely that the road safety project and the evaluation would need to be conducted on a large scale over a long period of time and run under the conditions of a scientific trial. These types of project are likely to require a great deal of resources.
Ways to Measure Effectiveness
Rather than casualty reduction it is therefore necessary to identify proxy outcome measures that can be expected to contribute to a reduction in casualties. Outcomes that you should seek to measure include:
- Changes in attitudes e.g. against drink-driving;
- Changes in knowledge and understanding of road safety; and
- Changes in behaviour, e.g. through the skills obtained through cycle training.
These measures should be specific and reflect the project's educational objectives - for example, measuring children’s ability to find a safe route as an outcome of a child pedestrian training programme.
In turn, such changes in behaviour could be expected to lead to a reduction in casualty or accident rates.
But, it is usually very difficult to separate the effects of an ETP project from the influence of other factors. For example, as children grow older they also gain experience, and are likely to be exposed to road safety messages from other national or local interventions as well as the specific project in which they've been involved. Multiple outcome measures will increase the reliability of the findings and should be used to overcome these difficulties.
Before Measurements
It is important to measure the existing level of behaviour, attitudes, knowledge or skills, before the project is implemented. The amount of change after the project is compared against the before measurements.
Short and Long Term Outcomes
Short term outcomes are likely to be the effects you see immediately after the intervention. In education programmes or publicity campaigns these are likely to be changes in awareness levels, knowledge and understanding or attitude change. In training programmes there may be changes in skill development that can be observed.
It is also a good idea to measure more long term outcomes. These are the effects of the project over time. At this point, the audience will have had time to put what they have learned into practice, and you can test if they have done so. You can also measure if any changes in knowledge or attitudes have been sustained over time.
Improvements to Delivery - Outputs
An output is something produced by the project (for example, good practice guidelines), the number of people exposed to an intervention, or an activity conducted during the project to achieve the objectives (for example, an education workshop).
Measuring outputs might consist of reporting that a number of training courses had been conducted.
Measuring outputs can help to make improvements to the mechanics of how a road safety project is delivered. For example, by measuring the uptake of a road safety education programme this will highlight where there is a short-fall in the number of schools you would like to target.
There is a tendency for those who are inexperienced in evaluation to measure outputs to try to state whether a road safety project was effective. For example, someone might claim that their education workshops in schools were successful because they were delivered to 500 schools. This does not tell you, however, whether the children are safer as a result. The workshop leader, for example, could have decided not to talk about road safety at all.