1. Slide 1 ILLINOIS - RAILROAD ENGINEERING Development of Machine Vision Technology for Inspection of Railroad Track John M. Hart*, J. Riley Edwards, Steven V. Sawadisavi, Esther Resendiz* Christopher P. L. Barkan, Narendra Ahuja* Railroad Engineering Program and *Beckman Institute University of Illinois at Urbana-Champaign AAR Technology Scanning Meeting - 24 September 2009

2. Slide 2 ILLINOIS - RAILROAD ENGINEERING Presentation Outline Background Prioritization of Inspection Tasks Current Camera Views Initial Work on Turnout Inspection Data Collection Algorithm Development Lateral View Processing Over-the-Rail View Processing Future Work and Inspection Tasks Summary and Conclusions

3. Slide 3 ILLINOIS - RAILROAD ENGINEERING Background Inspection of track and track components is a critical, but labor intensive task resulting in large annual operating expenditures There are limitations in speed, quality, objectivity, and scope of the current inspection methods Machine vision provides a robust solution to facilitate more efficient and effective inspection of many track components

4. Slide 4 ILLINOIS - RAILROAD ENGINEERING Machine Vision System Outline Image Acquisition System Machine Vision Algorithms Railroad Track Data Analysis System Pertinent Information for Railroads

5. Slide 5 ILLINOIS - RAILROAD ENGINEERING Top Track Related Accident Causes from 2001-2005 (Class I Mains & Sidings)

6. Slide 6 ILLINOIS - RAILROAD ENGINEERING Current Component Inspection Priorities Rail anchors –Missing –Shifting –Inappropriate patterns Cut spikes –Missing –Raised –Inappropriate patterns Turnouts –Missing bolts –Missing cotter pins –Damaged switch and frog points

7. Slide 7 ILLINOIS - RAILROAD ENGINEERING Future Component Inspection Priority Future inspection tasks: –Turnouts (continuation of current work) –Crossing diamonds –Crib ballast / curve breathing Other inspection possibilities (may require additional camera views): –Measuring tie spacing –Detecting insulated joint slippage –Monitoring wayside rail lubricator performance –Recording rail manufacturing markings –Determining thermite weld integrity –Monitoring track circuit bond wire condition –Inspecting grade crossing surface condition –Measuring flangeway width and depth

8. Slide 8 ILLINOIS - RAILROAD ENGINEERING Current Camera Views: Over-the-rail View Used for measuring the distance between the spike head and the base-of-rail and verifying spiking patterns Also used to measure crib ballast level with respect to the top of the tie

9. Slide 9 ILLINOIS - RAILROAD ENGINEERING Current Camera Views: Lateral View Optimal view for measuring the distance between the rail anchors and the edge of the ties and verifying anchor patterns Used to verify spike heights measured in the over-the-rail view Conducive to panorama generation

10. Slide 10 ILLINOIS - RAILROAD ENGINEERING Turnout Background - FRA Data Analysis According to the FRA (Federal Railroad Administration) Accident database, the top three causes of turnout-caused derailments are: –Broken or Worn Switch points –Broken Switch –Switch point gap Switch points account for almost 13% of all rail accidents Frogs account for about 3%, in particular, worn or broken frog points

11. Slide 11 ILLINOIS - RAILROAD ENGINEERING Turnouts: Areas of Focus Switch –Distance of switch points from stock rail when open and closed –Missing bolts and cotter pins –Worn points Frog –Worn or broken point Guardrail –Gauge check clearance

12. Slide 12 ILLINOIS - RAILROAD ENGINEERING Field Data Collection Initially used handheld cameras to take images –Obtained insights on later challenges, such as variability in component appearance Developed Video Track Cart for field data acquisition –Used on low density track for verifying camera views and experimenting with lighting Optimizing lighting is a challenging task –Reviewed lighting solutions from existing machine-vision systems

13. Slide 13 ILLINOIS - RAILROAD ENGINEERING Video Track Cart Equipped with two camera mounts for the over-the-rail view and a gauge-side lateral view Geared tripod heads for precise adjustment of camera views Deep-cycle marine battery supplies power for all electronics Rugged laptop resistant to environmental conditions

14. Slide 14 ILLINOIS - RAILROAD ENGINEERING Initial Field Video Data Collection Visit to Advanced Transportation Research and Engineering Laboratory (ATREL) in Rantoul, IL - Tested on track panel Monticello Railway Museum (MRM) visits –Improved lateral view and over-the-rail view mounts –Manually measured and catalogued track defects for algorithm verification

15. Slide 15 ILLINOIS - RAILROAD ENGINEERING NS Field Video Data Collection Collected 20 videos on NS track west of Champaign, IL using both camera views Components captured: Turnout, panelized concrete grade crossing, continuously welded rail (CWR), tangent and curved track, anchors, and cut spikes Captured videos under a variety of lighting conditions

16. Slide 16 ILLINOIS - RAILROAD ENGINEERING Lighting Considerations Reviewed lighting systems of similar machine-vision systems –University of Central Florida’s track inspection system Lasers and timed strobe lights Sun shields –ENSCO’s joint bar inspection system High-powered xenon lights No sun shields –Georgetown Rail’s Aurora system Lasers with light filters on cameras Lighting solutions are unique to each system’s data collection requirements, with our system requiring even illumination over a large area of track

17. Slide 17 ILLINOIS - RAILROAD ENGINEERING Track Component Isolation Delineate ties, ballast, and rail using edge detection Incorporate texture classification to make edge detection robust –Extract patches of texture from a hand-labeled training image –Compare new texture patches against known textures Use Gabor filter outputs to classify patches from new video frames Original image Edge image (base-of-rail highlighted) Tie patches Ballast patches Rail patches

18. Slide 18 ILLINOIS - RAILROAD ENGINEERING Base-of-Rail Delineation The original image, where we will isolate the base of the rail

19. Slide 19 ILLINOIS - RAILROAD ENGINEERING Base-of-Rail Delineation Strong horizontal edges are detected (candidate edge is shown)

20. Slide 20 ILLINOIS - RAILROAD ENGINEERING Base-of-Rail Delineation Surrounding pixels above and below candidate edge are examined

21. Slide 21 ILLINOIS - RAILROAD ENGINEERING Base-of-Rail Delineation Ballast below edge Base of the rail is confirmed

22. Slide 22 ILLINOIS - RAILROAD ENGINEERING Component Detection Detect tie plates using similar edge/texture method –Edge detection, verified with texture classification Hypothesize the spike locations (and missing spike locations) using prior knowledge of tie plate structure –Spikes (and missing spikes) are located with template filters –Spike height is measured relative to the base of the rail Original Image Delineated components Spikes and missing spikes

23. Slide 23 ILLINOIS - RAILROAD ENGINEERING Component Detection: Anchors Detect anchors –Use prior knowledge of anchor location (since tie plate and rail bottom have been delineated) –Find parallel edges to achieve anchor detection Robust to changes in anchor orientation Isolated anchors

24. Slide 24 ILLINOIS - RAILROAD ENGINEERING Defect Detection Results Individual components are located and measured Tie with shifted anchor

25. Slide 25 ILLINOIS - RAILROAD ENGINEERING Defect Detection Results Individual components are located and measured Tie with shifted anchor

26. Slide 26 ILLINOIS - RAILROAD ENGINEERING Defect Detection Results - Shifted Anchor Defect detected if anchor displacement with respect to the edge of the tie exceeds a threshold Tie with shifted anchor

27. Slide 27 ILLINOIS - RAILROAD ENGINEERING Component Detection on Field Video Field data video

28. Slide 28 ILLINOIS - RAILROAD ENGINEERING Panoramic Generation

29. Slide 29 ILLINOIS - RAILROAD ENGINEERING Track Panorama Visualization of cumulative track components using panoramic stitching Tie / Tie plate delineation on test panorama Component detection on test panorama

30. Slide 30 ILLINOIS - RAILROAD ENGINEERING Performance Lateral-view video from NS track (54 consecutive ties) Tie and anchor detection rates (precision and recall)

31. Slide 31 ILLINOIS - RAILROAD ENGINEERING Performance Lateral-view video from NS track (54 consecutive ties) Tie plate hole and spike localization success

32. Slide 32 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection - Original Frame Original frame Locate track components from over-the-rail video

33. Slide 33 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection - Rail Location Locate rail by spatial self-similarity across consecutive frames Locate rail

34. Slide 34 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection - Texture Classification Classify texture in frame as ballast/non-ballast Detect ties with tie filter Locate ties with filter Tie Filter

35. Slide 35 ILLINOIS - RAILROAD ENGINEERING Localize tie and tie plate using edge boundaries Ballast detection used as preliminary step Over-the-rail Inspection - Tie and Plate Location Delineate ties with gradients

36. Slide 36 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection - Spike Location Locate spike and spike hole with spatial template Locate spike and tie plate hole

37. Slide 37 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection Results Inspected frame

38. Slide 38 ILLINOIS - RAILROAD ENGINEERING Over-the-rail Inspection Video Field data video

39. Slide 39 ILLINOIS - RAILROAD ENGINEERING System Implementation Future system likely to be mounted on either: –Track geometry car –Rail defect detector car –High-rail vehicle Possible initial implementation on defect detector car –Inspections occur with acceptable frequency –Crews better trained with advanced inspection equipment –Already equipped for massive data storage and upload The system will be initial adapted for experimental testing high- rail vehicle

40. Slide 40 ILLINOIS - RAILROAD ENGINEERING Future Work Plan for the remainder of 2009 –Continue testing spike and anchor algorithms on field- acquired video –Begin lighting experimentation on Video Track Cart –Complete study on methods to inspect for turnout defects –Develop an approach for detecting turnout components –Record additional videos at MRM and on other local tracks Approach for 2010 –Investigate cameras which can run at higher speeds and in a greater variety of environmental conditions –Develop and test algorithms for turnout defects –Begin adaptation of acquisition system for trials on a track vehicle

41. Slide 41 ILLINOIS - RAILROAD ENGINEERING Future Component Inspection Discussion Future inspection tasks: –Turnouts (continuation of current work) –Crossing diamonds –Crib ballast / curve breathing Other inspection possibilities (may require additional camera views): –Measuring tie spacing –Detecting insulated joint slippage –Monitoring wayside rail lubricator performance –Recording rail manufacturing markings –Determining thermite weld integrity –Monitoring track circuit bond wire condition –Inspecting grade crossing surface condition –Measuring flangeway width and depth

42. Slide 42 ILLINOIS - RAILROAD ENGINEERING Acknowledgements Research Sponsors –AAR Technology Scanning Program –NEXTRANS Center Industry Assistance –David Davis of TTCI –Technology Scanning Committee –CN –UP –NS –BNSF –Monticello Railway Museum 42

43. Slide 43 ILLINOIS - RAILROAD ENGINEERING UIUC Multidisciplinary Research Team Computer Vision and Robotics Laboratory: Narendra Ahuja, Professor, Beckman Institute John M. Hart, Senior Research Engineer, Beckman Institute Esther Resendiz, Graduate Student, Electrical and Computer Engineering Railroad Engineering Program: Chris Barkan, Associate Professor, Director of Railroad Engineering Program Riley Edwards, Lecturer, Railroad Engineering Program Luis Fernando Molina Camargo, Graduate Research Assistant Matt Toussaint, Undergraduate Research Assistant Randy Nolden, Undergraduate Research Assistant Brennan Caughron, Undergraduate Research Assistant

44. Slide 44 ILLINOIS - RAILROAD ENGINEERING Appendix Additional Turnout Data

45. Slide 45 ILLINOIS - RAILROAD ENGINEERING Turnouts: Switch Point Inspection Rail point closed tightly Chipped Gap on open rail point is wide enough for wheel flange Missing cotter pins

46. Slide 46 ILLINOIS - RAILROAD ENGINEERING Spring Frog Inspection Missing cotter pins Gauge Springs working Tightly fit

47. Slide 47 ILLINOIS - RAILROAD ENGINEERING Bound Manganese Frog (BMF) Inspection Gauge 6” of point missing worn down 5/8” Flangeway depth Worn tread point 6” max 4” defect