1. Automatic Camera Calibration for Image Sequences of a Football Match Flávio Szenberg (PUC-Rio) Paulo Cezar P. Carvalho (IMPA) Marcelo Gattass (PUC-Rio)

2. reference points object points Juiz Virtual

3. reference points object points Juiz Virtual

4. Proposed algorithm Computation of the planar projective transformation Camera Calibration Filtering to enhance lines Next image in the sequence Detection of long straight-line segments First image of the sequence Line recognition Line readjustment Computation of the initial planar projective transformation

5. Filtering to enhance lines The Laplacian of Gaussian (LoG) filter is applied to the image with threshold. Gaussian filter Laplacian filter

6. Detection of long straight line segments Segmentation is done in the image to locate long straight line segments candidate to be field lines. This procedure is divided in two steps: Eliminating pixels that are not in a straight line. Determining straight lines segments.

7. Eliminating pixels that are not in a straight line The image is divided, by a regular grid, in rectangular cells.

8. For each of theses cells, the eigenvalues, 1  2 of the covariance matrix, given below, are computed. If 2 = 0 or 1 / 2 > M (given) then eigenvector of 1 is the predominant direction else the cell does not have a predominant direction Eliminating pixels that are not in a straight line

9. Cells with pixels forming straight line segments: Eliminating pixels that are not in a straight line

10. Determining straight line segments The cells are traversed in such a way that columns are processed from left to right and the cells in each column are processed bottom-up. Each cell is given a label:  If there is no predominant direction in a cell, discard it.  Otherwise check the three neighbors to the left and the neighbor below the given cell. If any has a predominant direction similar to that of the current cell, then it receives the label of that cell; otherwise, a new label is used for the current cell.

11. Determining straight line segments Group the cells with the same label Merge the groups that correspond to segments that lie on the same line. At the end of the process, each group provides a straight line segment. 

12. Field lines recognition From the set of segments, the field lines are detected and the field is recognized.  Model-based recognition method [Grimson90] Set of restrictions F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 f1:f1: f2:f2: Interpretation Tree F1F1 F6F6 F2F2 F3F3 F4F4 F5F5 F7F7 Model f1f1 f2f2 f3f3 f4f4 f5f5 Visualization f6f6 f7f7 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2 F1F1 F7F7  F6F6 F5F5 F4F4 F3F3 F2F2

13. The node {f 1 : F 1, f 2 :F 6, f 3 :F 3 } is discarded because it violates the restriction: The line representing F 6 must be between the lines representing F 1 and F 3. Field lines recognition Discarding nodes F1F1 F6F6 F2F2 F3F3 F4F4 F5F5 F7F7 Model f1f1 f2f2 f3f3 f4f4 f5f5 Visualization f6f6 f7f7

14. Field lines recognition Choosing the best solution F1F1 F6F6 F2F2 F3F3 F4F4 F5F5 F7F7 Model f1f1 f2f2 f3f3 f4f4 f5f5 Visualization f6f6 f7f7 In general, there are several feasible interpretations; We choose the one where the sum of the length of the matched segments is maximum. f 1 :  f 2 : F 3 f 3 :  f 4 : F 1 f 5 : F 6 f 6 : F 4 f 7 : F 7 f 1 :  f 2 :  f 3 : F 3 f 4 : F 1 f 5 : F 6 f 6 : F 4 f 7 : F 7 WINNER

15. F7F7 f1f1 f2f2 f3f3 f4f4 f5f5 Visualization f6f6 f7f7 F1F1 F6F6 F2F2 F3F3 F4F4 F5F5 F7F7 Model Field lines recognition Final result f 1 :  f 2 : F 3 f 3 :  f 4 : F 1 f 5 : F 6 f 6 : F 4 f 7 : F 7 f1f1 f2f2 f3f3 f4f4 f5f5 Visualization f6f6 f7f7 F1F1 F6F6 F2F2 F3F3 F4F4 F5F5 Model f 1 :  f 2 : F 3 f 3 :  f 4 : F 1 f 5 : F 6 f 6 : F 2 f 7 : F 5 or

16. Computation of the initial planar projective transformation A planar projective transformation corresponding to the recognized lines is found (using points of intersection and vanishing points as reference points).  points of intersection vanishing points

17. Line readjustment tolerance of the line readjustment

18. Line readjustment reconstructed line tolerance image points new located line The new located line is obtained by a least square method

19. Computation of the final planar projective transformation After relocating the field lines, a better reconstruction of the field can be obtained.

20. Camera calibration Camera is calibrated using Tsai’s method for reconstruction of elements not on the plane of the field.

21. For the first image, we apply the camera calibration process proposed. In order to optimize the process from the second image on, we take advantage of the previous image calibration. The final planar projective transformation for the previous image is used as a initial transformation for the current image. Working with a sequence of images Computation of the planar projective transformation Camera Calibration Next image in the sequence Line readjustment

22. Result s First scene Last scene The artificial sequence First scene Last scene The real sequence

23. Result s Computer: Pentium III 600MHz The sequence of test has 27 frames The time of processing: 380 milliseconds (< 900 milliseconds needed to real-time)

24. Results (accuracy) Field’s Points Correct Coordinates Reconstructed Coordinates Error xyzuvuv 105.068.000.0081.707216.58481.731215.9720.612 88.513.840.00230.11778.133228.74777.5251.499 88.554.160.001.236183.4630.424183.1970.854 99.524.840.00259.039134.206258.566133.8150.614 99.543.160.00146.690174.826146.067174.4840.711 105.030.340.00269.817155.102269.629154.6970.446 105.030.342.44270.921181.066270.215180.8630.735 105.037.662.44224.101194.645223.291194.4070.845 105.037.660.00223.405170.271223.082169.8760.510 Average Error 0.696 Tab. 1 - Comparison between the correct and reconstructed coordinates for the first scene.

25. Results (accuracy) Field’s Points Correct Coordinates Reconstructed Coordinates Error xyzuvuv 105.068.000.0097.167205.94096.791205.5850.517 88.513.840.00243.88366.434243.54966.0220.530 88.554.160.0016.101173.17415.655172.6230.709 99.524.840.00273.344124.029273.125123.7150.382 99.543.160.00160.672164.798160.366164.4210.486 105.030.340.00284.160145.173283.992144.9140.309 105.030.342.44285.241171.290284.886171.0900.407 105.037.662.44238.127184.768237.744184.5380.447 105.037.660.00237.462160.349237.252160.0630.355 Average Error 0.452 Tab. 2 - Comparison between the correct and reconstructed coordinates for the last scene.

26. Conclusions The algorithm presented here has generated good results even when applied to noisy images extracted from TV. The algorithm can be used in widely available computers (no specialized hardware is necessary). Processing time is well below the time needed for real-time processing. Extra time can be used, for example, to draw ads and logos on the field.

27. Future work Investigate processes for smoothing the sequence of cameras by applying Kalman filtering. Develop techniques to track other objects moving on the field, such as the ball and the players. Draw objects on the field behind the players, to give the impression that the players are walking on them.