1. Soccer Video Analysis EE 368: Spring 2012 Kevin Cheng

2. To detect and track key features needed to interpret events in a Soccer game from a video Goal

3. Clip

4. Player DetectorField Detector Ball Detector Overview Frame Pre- Processing Input Image

5. Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask Field detector is “trained” on a set of images with known RGB values, using a 16x16x16 color map

6. Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask For each frame, using the field color map, each pixel is labeled as “field” or “non-field”

7. Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask The largest region is considered where the field lies. A mask is created from the convex hull of the region.

8. Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask Using a canny edge detector and hough transform, we attempt to extract the field lines to help extract information of the field location

9. Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector Using the first frame as a keyframe, we determine the mean RGB values of the 2 teams. This is done by taking a histogram of the field, and finding the 2 highest bins that are NOT labeled as ‘field’ by the field map.

10. Player Detector To find potential player positions, we use a SIFT keypoint detector to narrow down the search regions Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider

11. Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector

12. Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector Since a single player can produce multiple SIFT keypoints, we implement a minimum distance connectivity map to remove redundant keypoints.

13. Keypoint#1234567 10133.090294.6414318.0277661.91123 164.7938 2133.0902038.47077115.1173167.9047 137.186 394.6414338.47077076.65507132.8157 131.5295 418.02776115.117376.65507072.13876 153.675 561.91123167.9047132.815772.1387600225.6103 661.91123167.9047132.815772.1387600225.6103 7164.7938137.186131.5295153.675225.6103 0 Connectivity Map A connectivity map is created were the (i,j) entry is the pixel distance between keypoint i, and keypoint j. This matrix is square and symmetric

14. Keypoint#1234567 10133.090294.6414318.0277661.91123 164.7938 2133.0902038.47077115.1173167.9047 137.186 394.6414338.47077076.65507132.8157 131.5295 418.02776115.117376.65507072.13876 153.675 561.91123167.9047132.815772.1387600225.6103 661.91123167.9047132.815772.1387600225.6103 7164.7938137.186131.5295153.675225.6103 0 Total0110110 All entries larger than 40 are tagged as duplicates. Here we see that keypoints (2,3) should be merged, and that (5,6) are duplicates. Connectivity Map

15. Keypoint#124567 10133.090218.0277661.91123 164.7938 2133.09020115.1173167.9047 137.186 418.02776115.1173072.13876 153.675 561.91123167.904772.1387600225.6103 661.91123167.904772.1387600225.6103 7164.7938137.186153.675225.6103 0 Connectivity Map Systematically, we remove duplicates starting from the highest connected keypoint.

16. Keypoint#12457 10133.090218.0277661.91123164.7938 2133.09020115.1173167.9047137.186 418.02776115.1173072.13876153.675 561.91123167.904772.138760225.6103 7164.7938137.186153.675225.61030 Connectivity Map Systematically, we remove duplicates starting from the highest connected keypoint.

17. Keypoint#12457 10133.090218.0277661.91123164.7938 2133.09020115.1173167.9047137.186 418.02776115.1173072.13876153.675 561.91123167.904772.138760225.6103 7164.7938137.186153.675225.61030 Connectivity Map The remaining keypoints are passed along as the player positions detected in the current frame

18. Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Tracking Since it cannot be assumed that we detect all players in every single frame, we implement a tracking algorithm. This algorithm once again utilizes a connectivity map to match current detections with history element.

19. 1234567891011 10542.04326.60224.26435.08127.9452.30356.31161.00265.56437.06 253.17500.1285.17178.68389.41103.711310.21117.68229.45395.82 3224.26322.32109.900211.01119.95179.68132.0663.6877.96219.15 4357.3195.8457.13133.0678.25245.94312.211196.45122.60101.85 5435.08123.17117.39211.010320.86390.4179.25274.08188.4460.34 6543.071.05216.48323.37124.19421.1502.13197.82383.61281.00106.16 7129.12419.11205.57119.40319.991.20105.59244.1764.14139.43314.02 8163.00380.60165.5261.70272.0965.60120.61193.462113.38276.19 9437.06105.15110.70219.1560.34315.03396.80102.60278.16175.130 10323.58218.473.01107.01120.09203.55283.1458.00164.4868.41113.71 History Now Player Tracking The (i,j) entry compares the pixel distance between the i th current detection and j th history element Moving from left to right, history elements with their closest current keypoints in a one-to-one matching

20. Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Tracking For each player, we assign a counter that keeps track of its detection history: – If the history element is matched with a current element, we increment the counter by 2 (max of 15). – If a history element does NOT find a match in the current frame, we decrement the counter by 1 (minimum of 0). – Only players whose counter is greater than 5 are displayed. – If a player’s counter drops below 0, it is removed from the history

21. Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Detector The ball detector is based on corner detection since a small round object should have high “cornerness”.

22. Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Detector To filter out all other corners in the image, we only look at corners with white RGB values. Another source of white corners in the image are the white field lines. So we also mask out corners that lie on the detected hough transform.

23. Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Tracking

24. Strengths: – Has a > 90% detection and tracking rate of – Built in robustness to false-positives and false-negatives Weaknesses: – A little bit jittery from frame to frame due to keypoint based detection. – Will falsely assign referees onto a team. Results: Player Detector

26. Strengths: – Maintains track of ball very well Weaknesses: – Requires to see the ball in every frame – Partial ball obscuring could cause tracker to latch onto another local maximum. Results: Ball Detector

28. Timing Analysis 2.63 seconds per frame. Field Detector: 46.6% (1.15 s/f) –.27 s spent in applyMap – Likely due to poor MATLAB programming. – Canny edge and Hough transform also big contributors Player Detector: 15% (.40 s/f) – SIFT detector:.28 s/f Ball detector: 10.9% (.27 s/f)

29. Field Detection: – Add ability to detect where in the field you are using field lines and features. Player Detection and Tracking: – SIFT is a large computational bottleneck. Are there more viable alternatives? Ball Detection and Tracking: – Improve handling of cases where ball is obscured for a short amount of time. Analysis and interpretation of detected events. Further Work