1. ForeSight: Mapping Vehicles in Visual Domain and Electronic Domain Dong Li, Zhixue Lu, Tarun Bansal, Erik Schilling and Prasun Sinha Department of Computer Science and Engineering The Ohio State University 1

2. Need for Targeted Communication OK, but who are you? What’s in front? Are you talking to me? Hey you at the back -- Your lights are off! I am overtaking you, don’t change lane! 2

3. Today’s Solutions Unicast: Hand Gestures, Eye Contact Requires parties to see each other Broadcast: Honk, Shout Disturbs others Agitates/annoys both parties 3

4. Tomorrow’s Technology: One Possibility Broadcast using Smartphone/DSRC Honking/shouting in the electronic domain Would cause sensory overload for drivers 4

5. Fundamental Problem in Targeted V2V Communication Who is the sender/receiver? Sender: What is the receiver’s unique address? Receiver: Which vehicle sent message to me? 5

6. To match vehicles in visual and electronic domains. Objective At the same time Decrease matching time Increase accuracy Generate less network traffic VID: Visual ID assigned by camera (e.g., red/yellow/blue box) EID: Electronic ID of the vehicle (e.g., IP/MAC address) 6

7. Available Features FeaturesAccuracy & Uniqueness Vehicle ColorNot always unique GPSNot accurate enough Vehicle ImageNot unique, environment dependent Plate numberUnique, but hard to read Relative SpeedMay not be accurate …. A unique set of features known to both vehicles is desired. 7

8. Main Idea If single feature is unreliable, can we use multiple features to do matching? System requirement: Camera, GPS Receiver and Radio Radio: communication Smartphone, DSRC Camera: identify vehicles Smartphone, Vehicle Security Driving Recorder Camera 8

9. Challenges Feature Inaccuracy E.g., A blue vehicle might be observed as black. Heterogeneous Capability Vehicles may not have smartphone, camera, radio, or may not be running our solution. Distributed in Nature Each vehicle only knows limited information 9

10. Vehicle Matching Process Matching vehicles based on similarity Estimate similarity between vehicles Weight Features Cluster {VIDs, EIDs} Obtain VIDs & EIDs Get VIDs from cameraGet EIDs from radio 10

11. Visual Matrix (from Video-Analysis) VIDN Features V1V1 f 11 f 12 …f 1N V2V2 f 21 f 22 …f 2N...……….… VMVM f M1 f M2 …f MN Vehicles Observed through Camera VID : Visual ID (camera assigns visual IDs to the observed vehicles) 11 VID only has local meaning (cannot be used by neighbors)

12. Electronic Matrix (from Electronic Messages) EIDN Features E1E1 f 11 f 12 …f 1N E2E2 f 21 f 22 …f 2N...……….… EKEK f K1 f K2 …f KN IDs received through WiFi/DSRC EID: Electronic ID (IP address, MAC address, etc.) 12

13. Create Similarity Matrix E1E1 E2E2...EKEK V1V1 …0.84...… V2V2 …… VMVM VIDFeatures V1V1 f 11 f 12 …f 1N V2V2 f 21 f 22 …f 2N...……….… VMVM f M1 f M2 …f MN EIDFeatures E1E1 f 11 f 12 …f 1N E2E2 f 21 f 22 …f 2N...……….… EKEK f K1 f K2 …f KN Electronic Matrix EVisual Matrix V Similarity Matrix S S = V  E T 13

14. Adaptive Weight (AW) Algorithm The Problem: How to combine different features to get the similarity value between two cars? The Intuition: Features with diversity values are important. E.g., color provides no information if the cars have the same color The Solution: Define Feature Distinguishability: the probability that any two observed vehicles are different based on this feature Similarity of two vehicles: weighted mean of the feature distinguishability values. 14

15. different lane, different color EbEb EaEa Matching with Similarity Matrix 15 V1V1 me V2V2 Visual Domain Electronic Domain 0.99 0.5 0.01 Steps Assign VIDs Receive EIDs Calc. Similarity Remove low similarity links different lane, similar color same lane, different color same lane, similar color

16. Matching with Similarity Matrix Greedy Matching Maximal Matching 16 Weighted Bipartite Graph Matching Problem EaEa EbEb 0.9 0.5 VIDsEIDs V2V2 EaEa EbEb 0.9 0.5 VIDsEIDs V2V2 Greedy matching is preferred. V1V1 V1V1

17. Clustering Vehicles Global distinguishability not required Nearby cars need to be distinguished Cluster the cars into smaller groups based on feature distance. Apply the AW algorithm within clusters 17 Clustering

18. Experiment Driving in freeway & local drive with 3 cars Using smartphone to collect GPS, video Experiment result 18 Vehicles with same color leads to low precision

19. Simulation 19 Using SUMO + NS3 Modeled the visibility of neighboring cars Modeled car detection prob., color detection accuracy, etc. ForeSight significantly improves the matching performance! 0.23 0.18

20. Case Studies: Improve GPS Each vehicle estimates its location with Its own GPS measurement Neighbors’ estimation of its location (assistance from Nbrs.) 20 Interesting Observations: When a car’s GPS error low, it is more likely to be matched by more neighbors. The match error increases as the number of neighbors increases: dense traffic makes matching more unreliable. High vehicle density

21. Case Studies: Reduce Disturbance Application: Send message to vehicles that are in front but has a slower speed Compare Broadcast, GPS and ForeSight 21 SchemesNotified Vehicles Ground Truth1021 Broadcast29 x 1021 GPS2706 (95% recall) ForeSight1141 (95% recall)

22. EbEb E1E1 E3E3 EaEa E1E1 E2E2 Future Work: Conflict Resolving Conflicts may Appear Matching result computed by different vehicles Matching result at different time Possible Solution Collaboration between neighbors 22 EbEb E1E1 E3E3 E1E1 EaEa E2E2 EID VID