Charles Fay, Sr. Program Officer Big Data Meets Computer Vision Dec. 7, 2012 Accelerating solutions for highway safety and performance SHRP 2 Strategic Highway Research Program

 Like challenges?  Then you should be excited by SHRP 2 NDS  ~ 4 petabytes of data that need to be post-processed  ~ 1 million hrs of video  ~ 3000 subjects, 5 million trips, >18 million miles driven, 4 billion GPS points  Real world - automotive conditions (daylight variance; nighttime IR); low quality cameras & images  Data compressed(H264) and saved at 15 Hz  PII (personal identifiable information) & protection of privacy  Patience with getting access to data- working out details 

Advisors to the Nation on Science, Engineering, and Medicine To: "investigate, examine, experiment, and report upon any subject of science or art - whenever called upon to do so by any department of the government” Transportation Research Board (TRB) is one of six major divisions Est National Academy of Sciences

Content What’s the Problem(s)? Preview video data Naturalistic Driving Study (NDS) Roadway Information Database(RID) FHWA Exploratory Advanced Research Program Goal today: promote interest in mining these data o making these data more usable Ultimately saving lives/ reducing severity of injury

Public Health & Highway Safety: Crashes leading cause of death for ~ 4-34 year old (US)* ~ 40,000 total deaths in US/year* ~ million injuries /yr in US Estimated costs: $230 billion/ yr in US Driver behavior has been identified as the major factor in 90% -95% of roadway crashes (know very little about behavior ) Major issue around the world; Naturalistic driving studies in EU, China, Australia; others in development- way of the future wot.motortrend.com

Computer Vision:  Before analysts can use the full NDS dataset – more usable form – that is where you come in  Lots of data from ~ 3000 participants ▪ ~ 4 petabytes; 1 million hrs video + other sensor data; 5 million trips; > 18 million miles  Saved video poorer quality relative to what you are used to analyzing.  PII (personal identifiable information) & data access (working out details-patience please) ▪ recording continuously: GPS; face video

DRIVER RELATED  Driver behavior  Distraction  Head pose  Eye gaze  Fatigue/drowsiness  Mobile device use  Hand position  Foot/pedal CONTEXT RELATED  Traffic signal state  Roadside information  Weather, pavement conditions  Bike/ Pedestrian  Other vehicles (brake lights) & traffic

“Your challenge should you choose to accept… …working on post processing these data in an efficient manner to gain meaningful information” kmnnz.wordpress.comhttp://kellypuffs.wordpress.com

Benefits of the Study (safety related) These data are not available – one of a kind database(s): decades of use Almost Everyone (OEMs, DOTs, researchers) eager to get hands on these data Intelligent / automated/connected vehicles & transportation Improved understanding of baseline driving behaviors:  Trip characteristics  Driver performance profiles  Adherence to laws and basic safety practices Improved understanding of unsafe behaviors and traffic events:  Assess circumstances and motivations for speeding, red light running, etc.  Deconstruct crashes and near-misses and examine causality  How do driver, vehicle, roadway, and environmental factors influence behavior and impact crash risk? Improved ability to develop safety countermeasures for:  Education and training  Roadway design and traffic engineering  Vehicle design  Regulation and enforcement  Ability to direct countermeasures at driver subgroups

Camera Image Samples Center stack – Pedal Interactions; hands Forward View - color Right-Rear View Driver Face – Rotated for max pixel efficiency Periodic still cabin image, permanently blurred for passenger anonymity (child safety seat use?) What can be done post-processed? Video 15Hz; H 264 compression

11 480x360 Scaled full to 480x x360 Scaled full to 360x240 or cropped at 360x240 and Rotated 90 degrees 360x120 Scale Vertical by ¼ horizontal by 1/2 360x120 Crop 25% off top and Bottom then Scale by 1/2

SHRP2 Naturalistic Driving Study (NDS) & Roadway Information Database (RID) NDS RID

NDS Data RID (GIS) (DAS GPS is Link) Existing Data characterize the environment in which the participant/ DAS operates: roadway, crash, safety campaigns, laws, traffic, weather, work zones… linked to roadway segment New Roadway Data Collected and QA ~ 1950 DAS ~3000 participants ~ 5 million trips Passenger Car, Van, SUV, Pickup

Six NDS Data Collection Sites across the U.S. One Coordinator NDS Data WA Data Collection IN Data Collection NC Data Collection NY Data Collection PA Data Collection FL Data Collection

NDS Data  Driving data: from instrumentation on vehicle  Driver data: from questionnaires, tests  Vehicle data: vehicle inspection; CANbus-vehicle network  Crash data: detailed investigation of selected crashes  Will include both restricted and non-restricted data requiring various levels of access  Restricted data: that which may be used to identify a participant, such as face video or GPS. Requires high level of physical and electronic security, data access agreements, ethics review, oversight. Working on specifics for data access (remote enclave(s) being considered)  Non-restricted data can be disseminated more widely via web access, summarized data sets, numerical variables

Data Acquisition System (DAS) 18

 Multiple Videos  Machine Vision  Eyes Forward Monitor  Lane Tracker  Accelerometer Data (3 axis)  Rate Sensors (3 axis)  GPS  Latitude, Longitude, Elevation, Time, Velocity  Forward Radar  X and Y positions  X and Y Velocities  Cell Phone  ACN, health checks, location notification  Health checks, remote upgrades  Illuminance sensor  Infrared illumination  Passive alcohol sensor  Incident push button  Audio (only on incident push button)  Turn signals  Vehicle network data  Accelerator  Brake pedal activation  ABS  Gear position  Steering wheel angle  Speed  Horn  Seat Belt Information  Airbag deployment  Many more variables… DAS Overview

total vehicle years

Thru Lane: 1 (21’) Thru Lane: 1 (12’) Accel. Lane: 1 Thru Lane: 2 (11’) Deccel. Lane: 1 Thru Lane: 1 (12’) Left Turn Lane: 1 Thru Lane: 1 (14’) Deccel. Lane: 1 Flush Paint. Flush Paint. Flush (Painted) 2’ Mix/Combo0’ Mix/Combo3’ Mix/Combo 2’ Mix/Combo N/A Grade, Cross Slope Unpaved Shoulder: N/A Rumble Strips: N/A Lighting: N/A Flush Paint. Flush Paint. Flush (Painted) N/A Thru Lane: 1 (12’) Thru Lane: 1 (11’) Right Turn: 1 Thru Lane: 1 (12’) 3’ Mix/Combo 4’ Mix/Combo N/A Paved Shoulder Median Lanes Paved Shoulder Median Lanes

Horizontal Curvature: Radius, Length,PC, PT,Direction Grade Cross Slope/ Super Elevation Lane in terms of the number, width, and type ( turn, passing, acceleration, car pool, etc…) Shoulder type/curb; paved width if exists Intersection location, number of approaches, and control (uncontrolled, all-way stop, two-way stop, yield, signalized, roundabout). Ramp termini are considered intersections Posted speed limit sign and location (R2-4 Series) Median presence(Y/N), type (depressed, raised, flush, barrier) Rumble Strip presence(Y/N) location (centerline, edgeline, shoulder) Lighting presence( Y/N) FHWA determining if additional data types will be processed (e.g., All MUTCD signs; barriers - TBD)

Route Name DirectionChainageState Collection Date Front ROW Images

#ItemPriority 1Crash Data 1 2Traffic Information - AADT 8Aerial Imagery 9Speed Limit Data 10Speed Limit Laws 11Cell phone and text messaging laws 12Automated enforcement laws 13Alcohol-Impaired and Drugged Drivers laws 14Graduated driver licensing (GDL) laws 15State motor cycle helmet use laws 16Seat Belt Use laws 5Local Climatological Data (LCD) NOAA 17Cooperative Weather Observer/Other Sources 4Winter Road Conditions (DOT) 2 3Work Zone Information 18Traffic Data - Continuous Counts (ATR) 19Traffic Data -Short Duration Counts 21Changes to existing infrastructure condition 22Roadway Capacity Improvements 6Nonrecurring Congestion 3 20Automated Enforcement 7Travel Time Data 23Innovative Treatments 4 25Recurring Congestion

25

 Working on providing data from 24 individuals  ~ 45 min per driver  Variety of facial features  Glasses/sunglasses  Daytime/ nighttime conditions  IRB; consent form allow data to be shared for research purposes  May need your IRB approval- most likey expedited review

 Exploratory Advanced Research Program  Video analytics workshop: 10/10-11/2012  summary report by January 2013   

 Charles Fay  