1. 1 5. Video Object Tracking and Processing To achieve augmented reality, computer generated graphics should be shown together with the live video In addition, the generated graphics should be changed according to the live video contents Hence we need to be able to understand the contents of the live video References: 1.Gonzalez, Rafael C., Digital image processing, Prentice-Hall, 2002. 2.The Mandelbrot DAZIBAO, “Lesson 5 – The HSV Colorspace”, http://www.mandelbrot-dazibao.com/HSV/HSV.htm. 3. Ogre Tutorials, http://www.ogre3d.org/wiki/index.php/Ogre_Tutorials

2. 2 Project Requirements In this project, a few color balls are randomly placed on the racing track You are required to develop computer programs that can detect the color balls in the live video obtained from the camera installed on the robot car Computer graphics should be generated to cover the areas in the video where the balls are found The size and position of the graphics should be adjusted according to the distance and relative direction between the balls and the robot car (i.e. the camera) Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

3. 3 Procedure for Detecting the Color Balls Grab a frame of the live video Convert the video data from RGB color space to HSV color space HSV color space is more close to human description to colors Select all pixels in the image that matches with the color of the ball Assume there are no other objects in the video that have the same color and similar shape Remove the isolated pixels (optional) Assume noise in the images may have similar color as the ball but exists in the form of isolated pixels Locate the center of the remaining pixels Assume the remaining pixels all belong to the ball of interested Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

4. 4 RGB Color Space For the current digital displays, each pixel is made of a Red, a Green and a Blue sub-pixel Each sub-pixel is activated by a byte of data, i.e. a number between 0 and 255 With these three bytes, the computer can generate 256x256x256 = 16,777,216 combinations If R = x, G = y and B = z, we can plot an RGB cube as follows: While we have a huge amount of possibilities, it is also tricky to define a precise color by adjusting the three components: If R=G=B=0, we have black If R=G=B=255, we have white If R=G=255 and B=0, we have yellow But how about the others???

5. 5 HSV Color Space HSV codification provides an intuitive method for color selection. Closer to human perception: The blend of the three components is defined by a single parameter called "Hue" The "Saturation" parameters selects how grey or pure the color will be The "Value" parameter defines the brightness of the color The HSV Color Space can be visualized as a cone Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

6. 6 RGB to HSV Conversion r – value of Red g – value of Green b – value of Blue max – the greatest of (r, g, b) min – the smallest of (r, g, b) h ∈ [0, 360) is the hue angle in degrees, and s, v ∈ [0, 100] are the saturation and value Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

7. 7 Why HSV? The color of an object as shown in the video can differ quite significantly from what is seen using naked eyes Many environmental factors can affect the color of an object in the video Exposure of the camera White balance setting Ambient lighting Reflection of the spot lights around Theoretically, an object of red color should have the RGB value of (255,0,0) In practice, one can never find such value although the object is really red in color when seeing it with naked eyes Some tolerance should be allowed if we want to detect an object of red color in the video. But the problem is how to set this tolerance Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

8. 8 Why HSV (cont)? We should consider an object in the video as red in color if it is “rather red” to “very red” Besides, a red object should be considered as red no matter it is pictured in a dark environment or a bright environment How to set such a tolerance? By using the HSV codification Hue (H) – Define a set of Red color Saturation (S) – Define how Red it is Value (V) – Define how bright it is Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

9. 9 Why HSV (cont)? Result of using the Paint.NET In RGB color space, for every R value, there can be a set of G and B values that will give you a red color Difficult to adjust if a set of Red colors are to be defined In HSV color space, H defines a set of Red colors S defines the saturation V defines the brightness A possible set of values for Red can be H 335 S > 45 V > 60 In HSV color space, H defines a set of Red colors S defines the saturation V defines the brightness A possible set of values for Red can be H 335 S > 45 V > 60 Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

10. 10 Original Video Frame A blue ball and a red ball are put in front of the camera Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

11. 11 Select the color We set a tolerance as follows: ?? < H < ??; S > ??; V > ?? We compare the HSV value of every pixel of the video frame to see if it falls within the tolerance Yes – mark it as green No – no action No matter how good is your estimation, there must be some pixels selected not belong to the object of interest They are considered as noise Noise Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

12. 12 Denoising Noise on the image is often considered as isolated dots or islands It can be reduced by using a simple denoising algorithm as follows For every 5x5 block, if most pixels have color falls within the tolerance, consider them as a part of the object For this block, only a few pixels are green. Hence they are noise For this 5x5 block, all of them have been marked as green. They are a part of the object Should carry out the above test for the whole image For this 5x5 block, all of them have been marked as green. They are a part of the object Should carry out the above test for the whole image

13. 13 Denoising Result The above algorithm should be carried out pixel-by-pixel for the whole image As a result, most of the noise are removed Depends on the actual environment, the denoising performance may not be as good as that Need long time fine- tuning Noise are removed Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

14. 14 Locate the object The location of the ball can be easily estimated by summing up all marked pixels row- by-row and column-by-column and look for their peaks The peak For each column, if a marked pixel is found, add 1 For each row, if a marked pixel is found, add 1 … …

15. 15 Restriction of the algorithm Certainly, the algorithm can only detect one blue ball If there are multiple blue balls, multiple peaks will result Become very difficult to determine which peak corresponds to which blue ball The difficulty also applies to the situation that there are other large objects that are also blue in color Multiple balls with different color is not a problem One can detect balls one color at a time Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

16. 16 Replace the detected object with 3-D graphics To achieve augmented reality, we need to combine graphics with live video If the position of the object is detected, we can replace it with 3-D graphics, such as fire generated by Ogre particle system Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

17. 17 Create a fire in Ogre // The particle "Examples/Smoke" is in the file // Example.particle // Assume mParticleSys and mParticleNode are two private // variables of EIE330projectApp mParticleSys = mSceneMgr->createParticleSystem("Smoke", "Examples/Smoke"); mParticleNode = mSceneMgr->getRootSceneNode() ->createChildSceneNode(); mParticleNode->setVisible(false); // Default - invisible mParticleNode->setPosition(0, 0, -140); // Set the position of the fire mParticleNode->attachObject(mParticleSys); : mParticleSys->setDefaultDimensions(220, 220); // Set the size of the fire

18. 18 Resize the fire Big fire! Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

19. 19 Changing the size and orientation of graphics For a real object on the screen, its size and orientation will change with the change of the camera position If a 3-D graphic is used to replace a real object on the screen, its size and orientation should also change with the change of the camera position Camera is closer to the object Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

20. 20 Changing the size and orientation of graphics Camera is moved to the right hand side of the screen Camera is moved to the left hand side of the screen Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

21. 21 How do we know the relative direction and distance of camera? By using the markers Each ball and the robot car will have a marker associated with it In Lab 6, you are required to use the markers’ data to estimate the relative distance and direction between the blue ball and the camera Need to replace the blue ball with a 3-D object that will change in size and orientation according to the position of the camera marker Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun PC Camera

22. 22 Task for Lab 5 Modify the function TextureSystem::UpdateTexture() to implement the following: Change the color space Select the pixels with red and blue color Denoise the selected pixels Detect the position of the red ball Create a fire at the position of the red ball on the screen

23. 23 Task for Lab 6 (1 st part) Change the size and orientation of an Ogre head with respect to the relative position of the camera, which is given in the first part of Lab 6 Procedure Create a Ogre head in EIE330ProjectApp::createScene() Implement the function EIE330ProjectApp::placingObject(float distance, float direction); // change the size and orientation of the Ogre head // based on the input parameters Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun

24. 24 Task for Lab 6 (2 nd part) The blue ball will be replaced by a 3-D object of your choice on the screen By making use of the markers’ data, estimate the relative distance and angle between the blue ball and the camera Modify the function placingObject() such that it will change the size and orientation of the 3-D object based on the estimated relative distance and angle marker PC Camera

25. 25 More instructions For Lab 5 and Lab 6, each team submits one lab report and does one demo for each lab No more lecture for Lab 6 Once finishing Lab 5, students can start Lab 6 immediately. The lab sheets for Lab 5 and Lab 6 are available in WebCT Both Lab 5 and Lab 6 do not need robot car and wireless camera Use that time to fine-tune your robot car. Don’t wait until the last week! Department of ELECTRONIC AND INFORMATION ENGINEERING 5. Video Object Tracking and Processing by Dr Daniel Lun