9-1 Using SafetyAnalyst Module 4 Countermeasure Evaluation

9-2 Why the Countermeasure Evaluation Tool? Goal of SafetyAnalyst is to help agencies determine how funds can be spent in the most cost-effective manner to improve safety Results of Module 4 can be used to: –Plan safety improvement programs –Update AMFs for use within Module 3 – Economic Appraisal and Priority Ranking

9-3 When to Use the Countermeasure Evaluation Tool The same countermeasure has been implemented at a number of sites The agency wants to assess how effectively the countermeasure performed: –Did it improve the overall safety performance at the respective sites? –Did it reduce a specific target accident type? –Was it cost-effective?

9-4 What Information Is Needed Locations of improved sites Countermeasure(s) to evaluate for each site –Year of implementation Site characteristics ADTs (before and after improvement) Yearly accident counts (before and after) Safety performance functions (SPFs)

9-5 General Steps for Countermeasure Evaluation Step 1 – Create a list of sites to be evaluated Step 2 – If necessary, add implemented countermeasure information to sites Step 3 – Select type of analysis Step 4 – Review results report

9-6 Step 1 – Create a List of Sites to Be Evaluated Site list to include only locations where a specific countermeasure has been implemented Sites should be of the same site type Desirable to include as many sites as possible –Statistically significant results are more likely with a greater number of sites

9-7 Countermeasure Evaluation Minimum Number of Sites Likelihood of finding statistically significant effects due to a countermeasure increases as the number of evaluation sites increase General rule of thumb: –Recommend 10 to 20 sites with 3 to 5 years of BEFORE accident data and 3 to 5 years of AFTER accident data

9-8 Step 2 – Add Countermeasure Information to Sites The countermeasure implemented at the evaluation sites must be identified before the evaluation can proceed: –The Data Management Tool (or Implemented Countermeasure Tool) is used to add this data to SafetyAnalyst –Countermeasure name and date may be imported with the tool or entered manually

9-9 Step 2 – Add Countermeasure Information to Sites The following information is needed for the implemented countermeasure at each site: –Countermeasure name –Construction start date –Construction end date –Major reconstruction flag –Construction cost (optional)

Step 3 – Select Type of Analysis The user must choose between a project analysis or a countermeasure analysis For a project analysis, the implemented countermeasure file is searched for all projects that include site list sites for selection by the user For a countermeasure analysis, a similar search returns individual countermeasures found for sites in the site list 9-10

9-11 Countermeasure Analysis This is the primary analytical technique The user must select which countermeasure(s) are being evaluated Multiple selections may be evaluated individually or as a countermeasure combination Only those sites in the site list with matching implemented countermeasure(s) will be included in the analysis

Project Analysis This is a secondary analytical technique that provides an estimate of safety improvement across all sites and countermeasures entered in a project Individual projects are evaluated separately Results are not reported with statistical certainty 9-12

9-13 Step 3 – Select Statistical Analysis Test for percent change in accident frequencies –Empirical Bayes (EB) method Test for change in proportion of target accident types –Wilcoxon signed rank test Both evaluations are available for both analysis types

9-14 Accident Severity Level User has a choice to perform the effectiveness analysis based upon one or more of the following: –Total accidents –Fatal and severe injury accidents –Fatal and all injury accidents –Property-damage-only accidents

9-15 Accident/AADT History The user must specify the time periods, or number of years of data, that will be used in the analysis: –Years before construction –Years after construction –Exclude years prior to major reconstruction –Buffer period for construction

Individual Site Selections Accident/AADT History is individually adjustable for each site being evaluated Site details may be reviewed to include/exclude site from the analysis SPF to be used in calculations may be adjusted Construction cost may be reviewed and updated 9-16

9-17 Accident Attribute Selection Evaluations can consider specific collision types related to: –Accident type and manner of collision –Vehicle turning movement –Light condition –Weather condition –Roadway surface condition –Others

9-18 Step 4 – Review Results Report Two types of output reports are generated during a countermeasure evaluation, in accordance with the two basic types of before-after evaluations. Summary information is provided for: –Input parameters –Effectiveness table –Site characteristics for individual sites –Accident statistics for individual sites

9-19 Output Measures for Percent Change in Accident Frequency Effectiveness (% change) Statistical Significance Effectiveness (Odds Ratio) Variance (Theta) Standard Error (Theta) Standard Error (E) Test Statistic

9-20 Output Measures for Change in Proportions of Target Accidents Simple average proportion BEFORE Simple average proportion AFTER Simple average difference (After- Before) Number of sites included in the statistical analysis

9-21 Output Measures for Change in Proportions of Target Accidents Estimated median treatment effect Selected (nominal significance level) Lower confidence limit of median treatment effect Upper confidence limit of median treatment effect Summary of statistical significance

Benefits of Using Module 4 Complex before-after analyses simple to specify and conduct Useful for HSIP reporting Update AMFs for use in Module 3 economic analyses 9-22

Questions 9-23