Soccer Video Analysis EE 368: Spring 2012 Kevin Cheng

To detect and track key features needed to interpret events in a Soccer game from a video Goal

Clip

Player DetectorField Detector Ball Detector Overview Frame Pre- Processing Input Image

Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask Field detector is “trained” on a set of images with known RGB values, using a 16x16x16 color map

Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask For each frame, using the field color map, each pixel is labeled as “field” or “non-field”

Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask The largest region is considered where the field lies. A mask is created from the convex hull of the region.

Field Detector Harris Corner Detector Input Image RGB Map Assignment Input Image Convex Hull of Largest Region Hough Transform Hough Mask Field Color Map Creator Field Mask Using a canny edge detector and hough transform, we attempt to extract the field lines to help extract information of the field location

Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector Using the first frame as a keyframe, we determine the mean RGB values of the 2 teams. This is done by taking a histogram of the field, and finding the 2 highest bins that are NOT labeled as ‘field’ by the field map.

Player Detector To find potential player positions, we use a SIFT keypoint detector to narrow down the search regions Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider

Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector

Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Detector Since a single player can produce multiple SIFT keypoints, we implement a minimum distance connectivity map to remove redundant keypoints.

Keypoint# Connectivity Map A connectivity map is created were the (i,j) entry is the pixel distance between keypoint i, and keypoint j. This matrix is square and symmetric

Keypoint# Total All entries larger than 40 are tagged as duplicates. Here we see that keypoints (2,3) should be merged, and that (5,6) are duplicates. Connectivity Map

Keypoint# Connectivity Map Systematically, we remove duplicates starting from the highest connected keypoint.

Keypoint# Connectivity Map Systematically, we remove duplicates starting from the highest connected keypoint.

Keypoint# Connectivity Map The remaining keypoints are passed along as the player positions detected in the current frame

Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Tracking Since it cannot be assumed that we detect all players in every single frame, we implement a tracking algorithm. This algorithm once again utilizes a connectivity map to match current detections with history element.

History Now Player Tracking The (i,j) entry compares the pixel distance between the i th current detection and j th history element Moving from left to right, history elements with their closest current keypoints in a one-to-one matching

Harris Corner Detector Input Image SIFT Keypoint Detector Input Image Keypoint Team Assignment Redundancy Removal Player Tracking Player Positions Team Color Decider Player Tracking For each player, we assign a counter that keeps track of its detection history: – If the history element is matched with a current element, we increment the counter by 2 (max of 15). – If a history element does NOT find a match in the current frame, we decrement the counter by 1 (minimum of 0). – Only players whose counter is greater than 5 are displayed. – If a player’s counter drops below 0, it is removed from the history

Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Detector The ball detector is based on corner detection since a small round object should have high “cornerness”.

Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Detector To filter out all other corners in the image, we only look at corners with white RGB values. Another source of white corners in the image are the white field lines. So we also mask out corners that lie on the detected hough transform.

Harris Corner Detector Input Image Harris Corner Detector Input Image Hough and White Mask Gaussian Weighting Max Value Ball Position Ball Tracking

Strengths: – Has a > 90% detection and tracking rate of – Built in robustness to false-positives and false-negatives Weaknesses: – A little bit jittery from frame to frame due to keypoint based detection. – Will falsely assign referees onto a team. Results: Player Detector

Strengths: – Maintains track of ball very well Weaknesses: – Requires to see the ball in every frame – Partial ball obscuring could cause tracker to latch onto another local maximum. Results: Ball Detector

Timing Analysis 2.63 seconds per frame. Field Detector: 46.6% (1.15 s/f) –.27 s spent in applyMap – Likely due to poor MATLAB programming. – Canny edge and Hough transform also big contributors Player Detector: 15% (.40 s/f) – SIFT detector:.28 s/f Ball detector: 10.9% (.27 s/f)

Field Detection: – Add ability to detect where in the field you are using field lines and features. Player Detection and Tracking: – SIFT is a large computational bottleneck. Are there more viable alternatives? Ball Detection and Tracking: – Improve handling of cases where ball is obscured for a short amount of time. Analysis and interpretation of detected events. Further Work