Presentation on theme: "Standardisation of In – Car Assessments Draft final report to Forum May 2007 JOHN HUNTER."— Presentation transcript:
Standardisation of In – Car Assessments Draft final report to Forum May 2007 JOHN HUNTER
Background Forum has talked about standardisation for many years There has been progress as a result of meetings and the Greenwich course, but no agreement on how best to achieve this goal TRL Report confirmed lack of uniform approach
AIMS Establish the range of methodologies used to record Details and Outcome of in – car assessments throughout Forum Establish the basis / rationale for the approach followed in each Centre Analyse the methodologies objectively Make recommendations
METHOD Questionnaire sent to all centres seeking 1 Recording Sheets 2 Their basis? 3 Factors recorded / scored on sheets 4 Relationship between records and decision on IN CAR outcome 5 Influence of other factors on final outcome of DRIVING Assessment 6 Any other observations / points?
Measuring and Recording Qualitative Semi-quantitative –Ordinal Scales - Adjacent scores either better or worse - Intervals between points variable Resistance to use – Why?
Why Measure? Aide memoire Evidence Bad records, poor defence; No records, no defence (MDDUS) Audit / Research
What to Measure? FACTORS should be Observable Recordable Relevant SCALES should be Sensitive Specific (Acceptable) TERMINOLOGY should be Accurate and Valid
RESULTS – 1 Scales used in Recording sheets Interval In-Car Clinic Overall after Return Outcome 5-point 1 1 4-point 7 5 1 3-point 3 4 2-point 1 None / 10 7 11 not specified
RESULTS – 2 Basis for sheets used Developed from DSA Test Cognitive Assessment based on Mavis workshop 1999 and Level 3 ADI Experience Progressive development of original Forum sheet (1980s) Other Mixture of above
VALIDITY Does the scale measure what it is supposed to? Elements - Face - Construct (theoretical basis) - Content (convergent, discriminant, concurrent, predictive)
Content Validity Do the components cover all aspects of the factor being assessed? Legal basis for driving assessment is the European Driving Directive 2006/126/EC - Specified Skills - Higher Level Road Safety Considerations Provides criteria against which recording sheets can be evaluated
EDD 2006 / 126 / EC Specified Skills Clutch Road position –Junctions Gears and Roundabouts Accelerator Lane Discipline Brake Traffic Lights Steering Road Signs Signals Car Control Observation Distance from other cars Anticipation Priority (X-roads & Speed Junctions) Mirrors Defensive & Social Driving Behaviour
Evidence that Specified Skills have been Assessed Factors Recorded (n) No. of Centres 20 2 19 3 18 1 12 – 16 5 7 – 11 4 None specified 2
2006 / 126 / EC Higher Driving Skills Recognise Traffic Dangers Have Command of Vehicles so as - NOT to create dangerous situations, and - To react appropriately if they occur Comply with Traffic Regulations Retain full use of Faculties needed for Safety Ensure Safety of All Road Users
RESULTS – 3 Higher Level Driving Skills (A) Take Appropriate Action 10 Multi-tasking / Divided Attention 10 Lane Selection / Discipline 9 Emergency Stop 9 Retention of Information / Instructions 7 Spatial Awareness / Clearance of Stationary Vehicles 7 Route planning / Navigation using Road Signs 6
RESULTS – 4 Higher Level Driving Skills (B) Decision Making 5 Speed Regulation 5 Road Sign Recognition / Recall 5 Note of Assessors Safety Actions 5 Forward Planning 3 Merge with traffic from Right or Dual Carriageway 3 Others (including Figure of 8 Manoeuvre)
Basis for Decisions - Results Patterns of Problems 11 (trends, consistency) Attribution to clinical / cognitive factors 7 Critical / Dangerous Incident 6 Overall Impression 6 Potential for Improvement 4 Mixture of Above 9 Scores Totalled 2
Influence of Other Factors on Final Decision of Driving Assessment Med Prog Cog SAU Fam Almost Never 1 3 10 Seldom 1 1 5 3 Often 10 6 4 6 4 Almost Always 5 11 12 2 (Others? Yes 9, No 8) Med – Medical Prog – Prognosis Cog – Cognitive SAU – Static Assessment Unit Fam – Family Views
CONCLUSIONS Study has confirmed, and given an idea of the extent of, the lack of standardisation of the In - Car part of Driving Assessment Some concepts have been misapplied Forum needs to address these shortcomings soon Many of the deficiencies can be remedied quickly and easily, but some will require changes in attitude and approach There are examples of good practice which can be built upon (and adapted if necessary) so they can be adopted and applied in all Centres
How Best To Go Forward? Consider having a NUMBER of Proformas for the different stages of the In – Car Ax A method of assessing and recording the Specified Skills should be easy to standardise and accept The elements of a systematic approach to assessing and recording Higher Level Skills are known but need to be agreed Outcome categories should be agreed
Recording In – Car Performance ASSESS CONTROL SKILLS and DRIVING BEHAVIOUR Using 3,4 or 5 Point scales The Specified Skills can be recorded directly, but The Higher Skills require agreement on terminology and method of evaluation
TRIP Methodology 4 – Point scale used to record performance, sometimes broken down into different elements, at specific points on the routes e.g. safe probably safe probably unsafe unsafe
Comments on TRIP - 1 Already used in 5 Centres for On – Route recording Each Centre has its favoured scale which it applies to everything being assessed BUT Is this appropriate for all parameters? TRIP forces decisions to be made on borderline performance BUT Is this always the right thing to do?
Comments on TRIP - 2 Some cases are genuinely in middle Observer bias (hard or easy markers) (particularly important when patterns are being used as basis for decisions) Unexpected occurrences are not scored, even if highly relevant or decisive TRIP was developed as a tool for research on groups of people, but in DA, we need to be get correct decision in individual cases
Comments on TRIP - 3 Can we consider adapting TRIP to take account of these concerns? RECOMMENDATIONS - Use 5 – point Scale - Use different terminology where necessary for different parameters - Have section on forms for recording unexpected incidents and assessor interventions
In Clinic Summary of TRIP Findings Profile of Parameters used on this form should be the same as on In – Car Form The Profile of scores should be transposed to this Summary Sheet PLUS a space for describing Critical Incidents
Content of On – Route Measures Suggestions for Higher Level Skills Parameter Extremes of Scales Vehicle Control Unsatis-------------Satis Orientation in Traffic Unsure ------------Sure Actions in Traffic Obstructs-------Courteous Reactions in Traffic Delayed---------Normal Accuracy of Reaction Poor------------Correct Speed Behaviour (1) Too fast-------------Normal Speed Behaviour (2) Too slow-----------Normal
Content of On – Route Measures Suggestions for Higher Level Skills Parameter Extremes of Scales Distance (forward) Too close----------Satis Distance (side) Too close----------Satis Lane Discipline Unsatis-------------Satis Merge with Traffic® Unsafe-------------Safe Emergency Stop Delayed--------Normal Risk behaviour Aggressive--Defensive
Content of On – Route Measures Suggestions for Higher Level Skills Parameter Extremes of Scales Attention Wanders----------Normal Divided Attention Distractable------Normal Anticipation Limited-------Foresighted Forward Planning Unsatis---------------Satis New Learning Poor-------------Normal Memory (tasks) Poor--------------Normal
Content of On – Route Measures Suggestions for Higher Level Skills Parameter Extremes of Scales Roundabouts Unsafe---------------Safe Decision making Indecisive------Decisive Route Navigation Unable-----------Correct Critical Events Dealt with---------Reacted (specify) by Assessor Appropriately Others?
Outcome The OUTCOME should be described as 1.Satisfactory 2.Minor problems, probably satisfactory 3.Review 4.Probably unsatisfactory 5.Unsatisfactory Need for adaptations / tuition etc should be separate from Outcome
Final Thoughts Summary of Recommendations for standardisation of content and process of Recording Sheets