Lessons Learnt

Doing Evaluation

The following real-life case study is an example of the value of collecting monitoring data. It also illustrates the benefit of conducting a process evaluation.

Understanding Responses and Discovering Best Practice

In response to a spate of serious and fatal crashes involving young people in our area, all 20 Secondary Schools were sent a flyer, via e-mail and post, offering a "free" 90 minute practical refresher driving session for 6th formers who had already passed their driving test.

Heads of 6th form were asked to make students aware of the free offer (up to them how they did this) and to consider including it as a session during the school day/PSHE/tutorial, although the sessions could be taken after school or at weekends. The flyer and letter were copied to the Head teacher and Chair of Governors of 20 schools. There was sufficient funding to enable 100 sixth formers to take part for free. There were about 5,000 sixth formers attending the 20 schools in total, of whom about 500 were thought to have passed their driving test at the time the offer was made.

Through monitoring responses from each of the 20 schools, we found that there had been no contact back from 15 of the schools. Four schools said they would advertise/promote it in the common room, and one was very keen and asked for further information.

Our records showed that a total of 35 students participated in the refresher session. We knew that 30 of those (86%) came from the one school that asked for more information. The other 5 students responded independently from 3 schools.

Knowing this we set about finding out why only one school responded so positively to the offer. What was special about that school? And what was it they did which resulted in the much higher take up?

We were interested in how we could improve the delivery of the intervention, in order to improve the outcomes. It was evident from the lack of response and the lack of active participation from so many schools (recorded through our monitoring data), that there was a problem with delivery.

Click here to go back to the top of the page.We found that there were three factors involved:

• Following receipt of the flyer, the school that was so supportive had asked whether an RSO could come to their 6th form tutorial time for 10 minutes and "sell it" rather than just putting up a notice in the 6th form common room. We agreed that an RSO would do this. He was a former police traffic officer. The "personal" delivery factor played a part.
• The Head of 6th form at this school also "sold" the offer to her students encouraging participation by emphasising that it was not only beneficial as a life-skill, but also that it would look good on their personal statement when applying to University.
• The 6th form timetable at this school enabled some flexibility for one day in the term when students could participate in a range of activities not in the usual curriculum.

We asked the other schools, one of which had 800 students in the 6th form, why the response was so poor. The results showed that lack of time was main reason. The Head of 6th form at the supportive school thought that flexibility within the timetable was workable at all 6th forms, and the poor response was more to do with a lack of publicity and effort on the part of the sixth form heads.

So we now know that instead of just distributing flyers and assuming that Head teachers and Heads of 6th forms would promote the offer, we would have to provide further support and contact, to make it easier for school staff to act on. For instance, we could suggest ways for schools to promote the offer, using the examples above of including it in their UCAS personal statements; and making use of the days allowed for extra-curricular activities.

We also now know that the presence of a "real person" explaining the benefits of the offer - the 'personal factor', could generate a more positive reaction in this instance, and critical to the intervention's success, even though our usual stance is not to provide RSO's as "speakers" at schools. We decided to look at this further, in future evaluations.

Using Logic Models for the First Time

The following paragraphs were written by Matt Pickard on behalf of the Regional Fatal 4 Team at Derbyshire County Council. He comments on the benefit of using a logic model for their intervention, despite it being a daunting prospect at first!

I must admit to some trepidation over doing the logic model as it seemed more difficult than it actually was. I'm sure that that was because the whole project considered evaluation as a key part, and we had worked through aims, objectives etc very thoroughly, even if it wasn't done exactly in the E-Valu-it terminology.

I can now see the real value in doing this first or at an early stage to refine the intervention and plan it logically. The real problem is convincing people of this. I can now honestly say that working though the logic model isn't as hard as it first seems, and it does give you focus on what's needed. Later on it makes the process and reporting much easier as you have already thought through what's needed. Time spent on the logic model is an investment in the long run, not an imposition.

Lessons Learnt from an Early Experience of Evaluation and Interpreting Results

Thanks to Dorset Road Safe ETP Working Group, for the following account of an early attempt at evaluation. The account highlights learning points.

We thought that road safety practitioners would like to hear about our early experience of trying to evaluate an ETP publicity intervention and our interpretation of the results. The intervention was planned before “E-valu-It” came into being and we would do things very differently now.

Our road safety partnership pooled their resources to pay for several full-colour whole page road safety adverts that appeared in a free magazine aimed at parents of primary school aged children. The magazine is issued six times a year and has a local circulation of some 50,000 copies. It is distributed by 92% of all primary and first schools in the partnership area. The partner agencies had a long-standing working relationship with the editors of the magazine and many road safety and sustainable school travel adverts had been placed in the same magazine in previous years. No evaluation on the effectiveness of the advertising had been carried out before.

In this particular case, over a period of 12 months, a whole page advert was placed in three different editions of the magazine. A different road safety theme was chosen for each edition timed to coincide with the DfT road safety THINK calendar of activity. The themes chosen for the three adverts were:

• Making independent journeys – "cycling & walking"
• Christmas Drink-Drive
• Be Safe Be Seen

The total cost of placing the adverts was £3,000 which was shared equally between the 7 partners.

It was after the second advert was placed that a decision was made by the partner agencies to try to establish the effectiveness of the advertising. We wanted to find out whether parents could recall seeing any of the road safety adverts that had appeared in previous editions of the magazine over a 12 month period. We also wanted to find out if parents could recall what the road safety messages/themes were and whether their behaviour had changed in any way having seen them. The opportunity was also used within the survey to establish brand awareness of two different logos - one for the overall road safety partnership and one that was being used widely for an ongoing large-scale local enforcement/education campaign.

It was agreed by all partners that a simple survey published in the magazine would be the most appropriate method of gathering information. Some of the questions would be “closed” with tick box answers and others “open” although space was limited for “open” answers. The last of the three paid-for adverts would appear on the back page of the magazine and the reader survey would appear on an early inside right-hand page of the same edition.

The magazine editors agreed to provide free copy space for the reader survey and provided a competition incentive with a single prize of a free return channel ferry crossing for a vehicle and up to four passengers for the winner picked at random from entries received. Entrants were asked to complete the survey and post it back at their own expense to the magazine’s offices.

We spent about 30 hours of officer time planning and developing the intervention and 12 hours on the evaluation.

Results:

A total of 65 completed survey forms were received (0.13% response rate).

Our interpretation of the results and lessons learnt:

We were very disappointed with the level of response, with only 65 out of a potential 50,000 readers completing the survey. A response rate of just 0.13%.

We assumed we would receive a greater number of responses considering the value of the prize incentive. A web site link to complete the survey and enter the competition would probably have generated a greater response. Parents would not have had to complete a survey, cut it out and post it at their own expense in order to enter the competition. We assumed that placing the survey on an early right hand inside page would attract the reader's eye.

Around half the respondents said they recognised the partnership logo. The logo actually appeared on the back page of the magazine supporting the whole page full colour road safety advert. Public exposure of the logo has been in place for about 2 years.

This provides some evidence that local people will recognise a logo if there is sufficient exposure.

The 80% awareness figure for the local largescale campaign logo mirrored the results from other recent local surveys. Public exposure of the logo has been in place for about 18 months.

This was an encouraging result. Frequent blanket exposure of branding appears to work in terms of recall. It is possible however, that some people answer positively because they want to please (or to win the prize!).

Almost half the respondents said they could not recall seeing any road safety adverts in the magazine in the past 12 months. This was disappointing as two different themed whole page road safety adverts had appeared in previous editions of the magazine over the previous year and the third advert was on the back page of the edition that carried the reader survey.

We didn’t know whether this was the first time the reader had seen the magazine. We don’t know if the back page is a good place for an advert as we didn’t ask readers where in the magazine they remember seeing the ad.

Almost half the respondents cited road safety themes that were not included in any of the adverts – in particular, speed and mobile phone use.

This may be because people are exposed to other external road safety advertising via different mediums and have been so for a long time so they assume this is what the adverts would have been about. The local campaign has focused a good deal on illegal mobile phone use and the increasing number of drivers caught for speeding offences. These were not themes in the magazine advertising.

Around half the respondents said they had altered their behaviour after having seen one of the adverts. Half said they had not. Of those who said their behaviour had changed, around two thirds related to road safety themes that did not appear in the adverts (Speed awareness and mobile phones/texting).

As we don’t know what their behaviour was before seeing the adverts it is impossible to know if the responses are true or not. The fact that a large proportion mentioned behaviour change related to themes not in the adverts may again be because people are exposed to other external road safety advertising via different mediums and have been so for a long time so they assume this is what we want to hear.

A fifth of respondents correctly identified that the adverts were placed by the Partnership. More than half thought they were government adverts.

The Partnership logo appeared with the advert on the back page of the magazine edition where the survey was published so some readers may have seen this before completing the survey.

Summary:

We now realise that without any specific aims or SMART objectives for our intervention, it is impossible to draw any conclusions about the outcomes. We weren’t really sure what it was we were trying to evaluate as we didn’t have a project plan or an evaluation plan in place. Each partner had slightly different priorities in terms of road safety theme so we tried to accommodate these by spreading the themes across different editions of the magazine rather than focusing on just one theme and repeating it in every edition. We also didn’t use our data analysis, road safety strategy, strategic casualty reduction assessment or intervention gap analysis work to decide on which common theme should be used in the adverts.We didn’t establish a logic model with our aims, inputs, outputs, assumptions, external factors, outcomes or long term impacts. If we had, we might have been able to redesign the intervention or decide not to carry it out at all.