Completed E-valu-it Question Sets
A screen shot of the E-valu-it Toolkit showing the Questionnaire section.
The following is an example of a completed question set taken from the E-valu-it Toolkit. These questions are designed to inform the evaluation recommendations for your intervention and the users' responses to these questions are transferred into their report template.
Only to be used in conjunction with the E-valu-it Toolkit.
Questions and Answers
Q1. What is the local road safety issue that your intervention is intended to address? (For example: Parents parking on zig-zag lines outside primary schools, or young people not wearing rear seat belts).
Answer: Adult car users not wearing seatbelts.
Q2. Why have you chosen to focus on this specific issue? (i.e. how can you demonstrate that there is a need for an intervention). Please select all that apply and provide details of your selection(s) in the box provided.
a) Anecdotal observation
b) Systematic observation
c) Research and evaluation reports
d) Complaints from the public
e) Local knowledge
f) Traffic speed data
g) Traffic volume data
h) Recorded traffic offences
i) Demographic data
j) Public consultation
k) There is no evidence yet
l) Other
Answer: a) Anecdotal observation, b) Systematic observation, c) Research and evaluation reports, e) Local knowledge, g) Traffic volume data.
Q3. What elements does your ETP intervention include? Please select all that apply and provide details of your selection(s) in the box provided.
a) Large scale presentation (e.g. Theatre in education)
b) Small scale presentation (e.g. Presentation to a classroom of school children)
c) Training courses (e.g. Older driver workshops)
d) Stands at public events or in public places
e) Poster campaign
f) Leaflet campaign
g) Web-based publicity (e.g. YouTube video clip)
h) Website
i) E-learning
j) Bus-back advertising
k) Diversionary measure (e.g. Speed awareness)
l) Radio advertising
m) TV advertising
n) Cinema advertising
o) Self-selecting training (e.g. Refresher driver training)
p) One-to-one advice and training
q) SMS messaging
r) Other
Answer: e) Poster campaign, j) Bus-back advertising, k) Diversionary measure (e.g. Speed awareness), l) Radio advertising.
Wide-scale publicity/advertising using posters, bus-backs, radio, telephone kiosks and adverts in local newspapers.
Q4. What information do you have to support your chosen intervention? (i.e. why do you think your particular intervention approach will be effective in addressing the specific issue identified in question 3.) Please select all that apply and provide details of your selection(s) in the box provided.
a) Continuation of existing practice
b) Replication of an intervention practice elsewhere
c) Research and evaluation reports
d) Road user audit report
e) Professional/expert advice
f) There is no information yet
g) Other
Answer: b) Replication of an intervention practice elsewhere.
Positive results from another shire authority where a combined mass publicity and enforcement project achieved significant results.
Q5. Have you identified aims for your intervention?
Yes
No
Answer: Yes.
Q5 a. What and who do you hope to change by your intervention? Your aim should relate to a measurable outcome. For example: Your aim could be to reduce the number of parents (the who) stopping on zig-zag lines outside primary schools (the what), or to increase rear seat belt wearing (the what) by under 25 year old passengers (the who).
Answer: Increase the percentage of adult car occupants wearing seatbelts.
Q5 b. By when do you hope to achieve your aim(s)? For example: A reduction in the number of parent driven vehicles stopping on zig-zag lines outside primary schools by July 2011.
Answer: An increase in adult seatbelt wearing rates by 31st March 2011.
Q6. Have you identified objectives for your intervention?
Yes
No
Answer: Yes
Q6 a. Please state your objectives. Your objectives are the ‘how’ of your intervention – how you intend to achieve your aims. For example: ‘To increase awareness by distributing a leaflet to 200 parents on the reasons why they should not park on the zig-zag lines’.
Answer: To observe a 5% increase, from 90 to 95%, in the number of adults wearing seat belts by March 2011.
To achieve a 20% positive change in the attitudes of adults towards the likelihood of being caught by the police for not wearing a seat belt, by March 2011.
Q7. Is your ETP intervention delivered in conjunction with: (please select one)
a) Engineering interventions
b) Enforcement interventions
c) Neither engineering nor enforcement interventions
d) Both engineering and enforcement interventions
Answer: b) Enforcement interventions.
Q8. Does your intervention consist of a single type of intervention or is it a programme of interventions? (please select one)
a) Stand-alone intervention (e.g. just bus-back advertising)
b) Programme of interventions (e.g. bus-back advertising as well as radio advertising)
Answer: b) Programme of interventions.
Q9. Which group of road users do you want to influence with your intervention? (i.e. which group of road users’ behaviour do you specifically want to change?) This is your target group. For example: To raise awareness amongst HGV drivers of blind spots and left turns to reduce cyclist casualties you would influence HGV drivers to benefit cyclists. Please select all that apply.
a) Car drivers (all)
b) Car drivers (learner)
c) Car drivers (novice)
d) Car drivers (experienced)
e) Car passengers
f) Motor cyclists – large machine (all)
g) Motor cyclists – large machine (learner)
h) Motor cyclists – large machine (novice)
i) Motor cyclists – large machine (experienced)
j) Motor cyclists – small machine (all)
k) Motor cyclists – small machine (learner)
l) Motor cyclists – small machine (novice)
m) Motor cyclists – small machine (experienced)
n) Van drivers or van passengers
o) Business drivers (non-HGV)
p) Business drivers (HGV)
q) Fleet managers
r) Public Service Vehicle Drivers
s) Agricultural vehicle drivers
t) Taxi drivers
u) Mobility scooter users
v) Cyclists
w) Horse riders
x) Bus users
y) Pedestrians
z) Age 0-5
A1) Age 6-10
A2) Age 11-15
A3) Age 16-24
A4) Adults 25-54
A5) Adults 55+
A6) Adults with learning difficulties
A7) All ages
A8) Black and Minority Ethnic
A9) Male and Female
A10) Males only
A11) Females only
Answer: a) Car drivers (all), b) Car drivers (learner), c) Car drivers (novice), d) Car drivers (experienced), e) Car passengers, n) Van drivers or van passengers, o) Business drivers (non-HGV), p) Business drivers (HGV), q) Fleet managers, r) Public Service Vehicle Drivers, s) Agricultural vehicle drivers, t) Taxi drivers, x) Bus users, a7) All ages, a9) Male and Female.
Q10. Which group of road users do you want to benefit with your intervention? (i.e. which group of road users’ safety do you specifically want to improve?). For example: You want to benefit cyclists by reducing the number of cyclist casualties through educating HGV drivers. Please select all that apply.
a) Same as previous question
b) Car drivers (all)
c) Car drivers (learner)
d) Car drivers (novice)
e) Car drivers (experienced)
f) Car passengers
g) Motor cyclists – large machine (all)
h) Motor cyclists – large machine (learner)
i) Motor cyclists – large machine (novice)
j) Motor cyclists – large machine (experienced)
k) Motor cyclists – small machine (all)
l) Motor cyclists – small machine (learner)
m) Motor cyclists – small machine (novice)
n) Motor cyclists – small machine (experienced)
o) Van drivers or van passengers
p) Business drivers (non-HGV)
q) Business drivers (HGV)
r) Fleet managers
s) Public Service Vehicle Drivers
t) Agricultural vehicle drivers
u) Taxi drivers
v) Mobility scooter users
w) Cyclists
x) Horse riders
y) Bus users
z) Pedestrians
A1) Age 0-5
A2) Age 6-10
A3) Age 11-15
A4) Age 16-24
A5) Adults 25-54
A6) Adults 55+
A7) Adults with learning difficulties
A8) All ages
A9) Black and Minority Ethnic
A10) Male and Female
A11) Males only
A12) Females only
Answer: a) The same as Question 9.
Q11. Approximately how many people will be exposed to your intervention? For example: A leaflet drop to 5000 local homes publicising older driver refresher courses. Think about how you arrived at your figure; for instance did you ask your Local Authority’s Statistical Analysis Team to identify the districts with the greatest population of retired residents? This helps you to target your intervention. (Please enter a number).
Answer: 200,000
Q12. Of those, how many do you realistically expect to actively engage with the intervention? For example: 100 driving residents out of the 5000 homes targeted will attend the refresher course (Please enter a number).
Answer: 20,000
Q13. Of those actively engaged how many do you expect to influence? For example: 95 of the 100 who attend, will alter the way they drive in one or more ways (Please enter a number).
Answer: 10,000
Q14. Start date of the intervention: (optional) (dd/mm/yy)
Answer: 01/01/10
Q15. End date of the intervention: (optional) (dd/mm/yy)
Answer: 31/12/10
Q16. What do you expect the total financial cost of the intervention to be?
Answer: £75,000
Q16 a. Does this financial cost include staff time?
Answer: Yes
Q16 b. What proportion of the total budget for the intervention are you expecting to spend on its evaluation? (Please select one)
a) Zero %
b) 1-4 %
c) 5-10 %
d) More than 10%
e) Don’t Know
Answer: b) 1-4%
Q17. Who is going to lead your evaluation? (optional) (Name and role)
Answer: Robert Smith – Road Safety Manager.
Q18. Does anyone within your team, or a partner agency, have experience with statistical analysis? (optional) (Please select one)
Yes
No
Answer: Yes
Q19. What is the purpose of the evaluation? (Think about what you intend to use the results for). (Please select one)
a) To improve the intervention
b) To show the effectiveness of the intervention
c) Both
Answer: b) To show the effectiveness of the intervention.
Q20. What stage are you at in planning your intervention? (Please select one)
a) Planning
b) Planning but not delivered
c) Delivering
d) Delivered
Answer: c) Delivering.
Q21. How do you intend to measure progress towards your aim? (You can select more than one)
Option 1: Monitoring Indicator
Option 2: Intermediate Indicator
Option 3: Long-term Outcome
Answer: Option 1: Monitoring Indicator, Option 2: Intermediate Indicator, Option 3: Long-term Outcome.
Q21 (1a). Please describe the data you will use for monitoring: For example: Number of parents who return a slip to say that they have received and read the leaflet.
Answer:
Monitoring the number of times the radio adverts are played, and the number of people estimated to hear those adverts, using listening statistics.
The number of bill board and bus back adverts displayed over a six week period.
The number of people who can recall seeing or hearing the adverts.
Q21 (1b). How can the data tell you if you have achieved your aim? For example: Previous research has shown that parents do not fully understand why they should not stop on the zig-zags. Therefore to reduce zig-zag parking a leaflet containing the reasons for not stopping on the zig-zags will be distributed. The monitoring data of how many parents have returned the slip cannot tell you if you have actually reduced zig-zag parking but it can tell you if parents are likely to be more aware of the issue and thus less likely to stop on the zig-zags.
Answer:
Previous research shows that 60% of adults feel there is little chance of being caught by the police for not wearing a seat belt. Therefore the billboard, busback and radio adverts communicating the new police enforcement campaign, will challenge this perception. Monitoring the number of adverts and the number of people exposed to them will not tell us if attitudes have changed or if people wear their seat belt more but it will tell us if drivers are likely to be more aware of the increased police enforcement, and therefore the increased likelihood of being caught.
Q21 (1c). What is your baseline for your monitoring indicator? For example: In previous years only 50% of parents have returned a slip to say they have received the leaflet. If this is a brand new intervention you may not have any baseline data yet.
Answer:
This is a brand new intervention so there is no baseline data yet for the number of people who can recall the advert.
Q21 (1d). By how much do you hope to change the indicator you are monitoring? For example: To increase the number of parents returning a slip to say they have received a leaflet, to 90%.
Answer:
We hope that 80% of those surveyed in Newtown will correctly recall hearing or seeing the advert(s) but without baseline data we cannot say by how much this will have changed.
Q21 (2a). Please describe the data you will use for your intermediate indicator. For example: A survey of parent knowledge on the reasons for not stopping on the zig-zags.
Answer:
Routine surveys of knowledge on the legal consequences of not wearing a seatbelt, and on the perceived likelihood of being caught by the police for this offence. Surveys to be conducted repeatedly over the six-week period.
Q21 (2b). How can the data tell you if you have achieved your aim? For example: Parents’ awareness of the reasons for not stopping on the zig-zags may influence their behaviour so it can be expected that this intermediate change will result in the final aim of reduced parking on zig-zags.
Answer:
Awareness of the legal consequences of not wearing a seatbelt and of increased enforcement may influence their behaviour so this intermediate change will result in the final aim of increased seatbelt wearing.
Q21 (2c). What is your baseline for your intermediate indicator? For example: Before they received the leaflet, parents scored, on average, 60% in a survey on the reasons for not stopping on the zig-zags. If this is a brand new intervention you may not have any baseline data yet.
Answer:
90% were aware of the legal consequences of not wearing a seatbelt following previous surveys.
60% of those who don’t wear a seatbelt felt there was little chance of being caught and prosecuted for the offence.
Q21 (2d). By how much do you hope to change the situation you are measuring in the intermediate indicator? For example: An increase from 60% to 90% in parent knowledge on the reasons for not stopping on the zig-zags.
Answer:
An increase from 90% to 95% in awareness of legal consequences of not wearing a seatbelt and a reduction from 60% to 40% in those who felt there was little chance of being caught and prosecuted for the offence.
Q21 (3a). Please describe the data you will use for your long-term outcome. For example: The number of cars stopping on the zig-zags between 8 and 9am and 3 and 4pm.
Answer:
The percentage of adults observed wearing a seatbelt.
The percentage of adults receiving a fixed penalty notice for seatbelt offences. This is a proxy measure.
Q21 (3b). What is your baseline for your long-term outcome? For example: On average, before parents received the leaflet, 40 cars per day were stopping on the zig-zags between 8 and 9 am and 3 and 4pm. If this is a brand new intervention you may not have any baseline data yet.
Answer:
On average, 92% of adults have been observed, by police, wearing a seatbelt on selected routes during peak rush hour times.
On average 38% of officer issued fixed penalty tickets were for seat belt offences.
Q21 (3c). By how much do you hope to change the long-term outcome you are measuring? For example: The number of cars stopping on the zig-zags between 8 and 9 am and 3 and 4 pm, will decrease from 40 to 20 cars per day.
Answer:
The percentage of adults wearing seatbelts will increase by 5% from 92% to 97%.
The percentage of fixed penalty tickets issued by the police for seatbelt offences will reduce from 38% to 20%.
Only to be used in conjunction with the E-valu-it Toolkit.