1. Motion Noise Separation In Digital Video Motion Noise Separation In Digital Video E. E. A. Yfantis, Y. Terradas, P. Olson F. Image Processing, Computer Graphics and Intelligent Systems Lab. G. School of Computer Science, UNLV 1

2. Video Capture Mechanisms Mos-CCDs (Charge Couple Devices) Composite, S-Video, 3-Channell CMOS (Complementary Metal Oxide Semi) Foveon Analog to Digital Converter Video Decoders Cost-Noise 2

3. Noise Sources Electronic Hardware Power Cables Cable Television-Cable Internet Nearby Antennas Alternating Current Light Fluctuations Due to Atmospheric fluctuations 3

4. Probability Distribution of TheNoise The probability distribution of the noise depends on the channel, light intensity, quality of the camera, shape of the object, Atmospheric conditions, ground temperature, position of the camera with respect to the sun, refresh frequency and speed of the object. 4

5. 16x16 Macroblocks for Motion Motion Detection Is very important to Compression, Pattern Recognition, and many other Application Areas. The 16x16 Macroblock motion Detection, Motion Estimation and motion compensation results into many missclassifications. 5

6. 6 VOP’s EXTRACTION - STEPS MOTION DETECTION For the motion detection process we made some assumptions: The background is stationary. The background is stationary. We have access to the background at the beginning of the sequence. We have access to the background at the beginning of the sequence.

7. 7 MOTION DETECTION (cont’d) Getting information of the background: Getting information of the background: The first step of the motion detection process involves getting the edge-map of the background. To do so we used the Canny Edge Detector. we thought that it would be helpful to, at first, to give a brief description on the implementation of the Canny’s Operator and later come back to the VOP extraction itself. CANNY EDGE DETECTOR Summarizing all the steps within this edge detector algorithm we have: smoothing, gradient calculation, non- maximal suppression, and hysteresis.

8. 8 CANNY EDGE DETECTOR (cont’d) SMOOTHING: This process is done by applying a Gaussian filter to the image. The Gaussian smoothing operator is a 2-D convolution operator that is used to `blur' images and remove detail and noise. The implementation of the Gaussian may be very expensive (many floating- point computations required). Yet, there is a way to create an integer- valued mask that approximates very well this floating-point mask. But we know that the Gaussian distribution is near zero within three standard deviations from the mean. Hence, we can truncate the kernel at this point. The following mask is a suitable integer-valued convolution kernel that approximates a Gaussian with a  of 1.0.

9. 9 CANNY EDGE DETECTOR (cont’d) Using a discrete approximation mask allows the MMX technology to perform up to four operations at the same time considerably reducing the execution time of the process. Throughout this report I will use the CIF sequence named BOWING to show the results at each step. Right now, I will show the resulting smoothed image of the frame number 73 from the video sequence mentioned above. Gray Scale Image Smoothed Image

10. 10 CANNY EDGE DETECTOR (cont’d) By applying the Gaussian filter we are eliminating most of the noise while we keep the most important edges of the image. The greater the , the stronger the blurring effect is. The selection of the value for the standard deviation (  ) depends, mainly, on the contents of the image (textures, shapes, signal/noise ratio, etc) as well as on the results we are seeking. In this case I am working with a  =1.0. GRADIENT CALCULATION: After smoothing the image and eliminating the noise, the next step is to find the edge strength by taking the gradient of the image. The main purpose of doing this is to highlight regions with high spatial derivatives. There are two well known algorithms to find edges: one of them finds the second derivative of the smoothed image and looks for the zero crossing pixels, while the other finds the first derivative of the smoothed image and the pixels that are maximum (locally).

11. 11 CANNY EDGE DETECTOR (cont’d) Let us put the last statement graphically: When finding edges we are looking for the steepest descend as well as the steepest ascend because both represent a high change in the intensity of the image. There are many ways and masks to perform the gradient calculation. In our case we are going to apply an easy approximation of the derivative operation. We just find the difference between the intensity value of the two consecutive pixels in both directions (x and y). By doing that we end up with two values per pixel named dx x,y and dy x,y.

12. 12 CANNY EDGE DETECTOR (cont’d) For instance if we take the pixel at the x,y position, then: Gradient at the pixel x,y Once we have the gradient value for each pixel we could get the magnitude and the orientation of the gradient by doing:

13. 13 CANNY EDGE DETECTOR (cont’d) The output of the gradient calculation step consists of two images each containing the values of the derivatives in the horizontal and vertical direction, respectively. Both images are shown below. Horizontal Derivative Vertical Derivative The next step deals with the gradient and how to use it to find the local maximum pixel in its direction.

14. 14 CANNY EDGE DETECTOR (cont’d) NON-MAXIMAL SUPRESSION: This step works with the magnitude and orientation of the gradient at the pixel under consideration and creates 1 pixel-width edges. From the previous stage we get the values of each component of the gradient so we can get its magnitude and direction. To calculate the direction of the gradient we don’t need to use the arctangent. Instead, we just analyze the value and sign of the components of the gradient. If we are at the pixel p x,y and the values of the derivatives at that pixel are dx and dy, we can approximate the direction of the gradient at p to one of the sectors showed in the figure. For instance, if dx=-10 and dy=12 we can say that the gradient at this pixel goes approximately between 90 and 135 degrees.

15. 15 Once we have figured out the direction of the gradient we need to interpolate the values of the pixels found in the neighborhood of the point under analysis. The pixel that has no local maximum gradient magnitude is eliminated. The comparison is made among the actual pixel and its neighbors along the direction of the gradient. Let us take again the last example where we approximated the gradient orientation as to be between 90 and 135 degrees, then we can build the next figure. CANNY EDGE DETECTOR (cont’d) gradient P x,y PaPa PbPb To compare the values of the gradient we don’t need to apply the square root nor the power functions. We can just compare the summation of the absolute values of dx and dy at each of the pixels under consideration.

16. 16 If the value at p is greater than both p a and p b then the pixel p is a candidate to be an edge pixel, if not p is eliminated.After non-maximal suppression we end up with an image that contains all the pixels that are a local maximum. The corresponding image would be like the one below. CANNY EDGE DETECTOR (cont’d) The problem here is that we may have some pixels that despite being a local maximum, they represent noise. Therefore, we can’t just take this as our final edge map image. At this stage most of the edge detectors apply a threshold process. They define the threshold so that each pixel with a value below the threshold is eliminated. The trade off is that if we pick a high level value we assure that most of the noise is eliminated. However, we could be deleting weak yet meaningful edges. On the other hand, if we utilize a low value for the threshold we get all the important edges as well as many of the noisy ones. Our next goal is to handle this problem.

17. 17 CANNY EDGE DETECTOR (cont’d) Hysteresis: Unlike single level thresholding we will select two levels : high threshold (T H ) and low threshold (T L ). Roughly speaking, edges start from pixels with gradients of T H or more; they then extend over any connected pixels with gradients of T L or more. If you think of following an edge, you need a gradient of T H to start but you don't stop till you hit a gradient below T L. In other words, for a given pixel, if the gradient magnitude is below T L it is unconditionally set to zero. If the gradient is at least T H the pixel is left alone. If the gradient is between these two, then it is set to zero unless there is a path from this pixel to a pixel with a gradient above T H ; the path must entirely be through pixels with gradients of at least T L. The path can include diagonal moves - i.e. we use 8-connectivity. This algorithm does not use the gradient direction estimates to follow edges - it simply treats an edge as a connected set of pixels. The likelihood of streaking is reduced drastically since the line segment points must fluctuate above the upper limit and below the lower limit for streaking to occur. One more time the selection of T H and T L depends on the type, content and predicted signal-to-noise ratios of the image.

18. 18 CANNY EDGE DETECTOR (cont’d) If we now apply hysteresis to the output image from the non-maximal supression step we get the following images for two different threshold levels. T L =50 T H =100 T L =50 T H =200 Once we get the edges map we can just set all the pixels that are edges (greater than zero) to 255 so they are clearly visible. There is one more step called Feature Synthesis but it is not widely used because the results obtained so far are good enough for most purposes.

19. 19 MOTION DETECTION (cont’d) Getting information of the background: Getting information of the background: Going back to the first step of the VOP extraction we need to get the background edge map using the Canny Operator just explained. The goal is to get as much information as possible from the background since that information will be used to identify the pixels that belong to the background. Once again, the selection of the variables for the Canny edge detection depends on the background type. For instance, if our background is cluttered we may need to select a small value of  and low threshold levels for the hysteresis process. This is mainly because the greater the amount of information we get from the background, the easier the detection of the pixels that belong to the background will be. As we can see the selection of the parameters depends on every scene and should be readjusted when the background or the scene changes, if it does. In our example I work with a simple background and normal levels of threshold will be selected. The standard deviation will be set to 1.0 so we can use the discrete Gaussian mask shown before. If we need another standard deviation value we can create the corresponding integer-valued kernel.

20. 20 MOTION DETECTION (cont’d) The following images show the original background frame and the corresponding edge map. This image will be saved so we can use it every time we need it. The next step will take care of two consecutive frames and will look for motion between them.

21. 21 MOTION DETECTION (cont’d) Detecting moving objects: Detecting moving objects: At this point our goal is to identify the objects that are moving in the scene. The following scheme briefly shows all the steps needed to find them. Eliminating background pixels Gray Scale Process Pixel-to-Pixel Subtraction Canny If pixel doesn’t belong to both images  eliminate it Linking pixels Moving Object Model in current frame

22. 22 MOTION DETECTION (cont’d) Gray Scale Process Pixel-to-Pixel Subtraction Canny

23. 23 MOTION DETECTION (cont’d) If pixel doesn’t belong to both images  eliminate it Linking pixels Eliminating background pixels Moving Object Model in current frame

24. 24 MOTION DETECTION (cont’d) At this stage I am able to get the moving parts of the objects in the scene. But VOP needs more than that. Each video object plane represents a whole moving object not just part of it. Now we need to define the initial model of the object and once we got it we should start tracking it all along the sequence. VOP’s EXTRACTION - STEPS INITIAL MODEL Defining the initial model is not an easy task. Some methods take an arbitrary number of frames at the beginning of the sequence until they think they have enough information to define an accurate model for the moving object. It is really difficult to define an initial model when the object shows up gradually on the scene. In those cases the hardest task is to decide whether the object has partially or completely showed up on the screen.

25. 25 INITIAL MODEL (cont’d) To handle more than one object at the same time I decided to create a box around each moving object. This box is created by taking the left-most, right-most, up-most, and bottom-most pixels in each object as its coordinates. This approach entails some difficulties. For instance, let us suppose that there are two moving objects in our scene. How can we identify each moving object if both appear at the same time on the screen? That issue as well as other considerations are open problems. Another very important issue is how to deal with objects that stop moving for an arbitrary period of time or that have moving and static parts. In that case applying the previous motion detection method poses a problem when trying to get the whole object. That is why we need to introduce the Distance Transformation. Distance transformations are widely used for object tracking. When properly used the DT can adapt to rotations and changes in shape. The idea of using a box can improve the tracking process because we already know the actual position of the object and we don’t need to search along the whole image to find the best match.

26. 26 INITIAL MODEL (cont’d) Distance Transformation: The distance transform of a black and white binary image is an operation that, for each pixel, computes the distance to the nearest black pixel (edge pixel in our case). Distance transformations are based on many kinds of distance functions given that the distance functions differ in efficiency or usefulness. The distance I use is the Chamfer Distance. The Chamfer distance transforms are a class of discrete algorithms that offer a good approximation to the desired Euclidean distance transform at a lower computational cost. They can also give integer-valued distances that are more suitable for several digital image processing tasks. Chamfer offers many integer masks, in this case the 3-4 mask is the one I chose but there is another Chamfer mask called 5-7-11. Chamfer 3-4

27. 27 INITIAL MODEL (cont’d) The first step of the DT is to get the edge model of the object to be tracked. With the edge map we need to create a new image based on the weights. That is, we have to initialize the output image with the following weights: If the pixel corresponds to an edge pixel then assign a weight of 0 If the pixel corresponds to an edge pixel then assign a weight of 0 If the pixel doesn’t correspond to an edge pixel assign a high value (theoretically  )If the pixel doesn’t correspond to an edge pixel assign a high value (theoretically  ) Now we have to work on the new weighted image. I used a two passes algorithm to build the DT image of the model. Forward Pass: This first pass goes from (0,0) to (xmax,ymax). The pixels to be weighted in this pass are the ones showed on the figure. p w1w1 w4w4 w2w2 w3w3 If p> 0, set p = min( p, min( g i +w i )) i= 1, 2, 3, 4 Where g i represents the weighted distance according Chamfer and w i is the actual weight of the corresponding pixel.

28. 28 INITIAL MODEL (cont’d) Backward Pass: It goes from (xmax,ymax) to (0,0). The pixels to be weighted in this pass are the ones showed on the figure. p w3w3 w4w4 w2w2 w1w1 If p> 0, set p = min( p, min( g i +w i )) i= 1, 2, 3, 4 Where g i represents the weighted distance according Chamfer and w i is the actual weight of the corresponding pixel. Let us show an easy example of the DT. Object’s Edge MapObject’s DT Map DT

29. 29 The underlying here is the following: First get the edge model of the object to track and apply the DT to find the binary image of the model. After that, use the binary image to track the given object in the next frame looking for the best matching. To find the best match we can calculate the total summation of the weights so we can deal with rotations or changes in the scale. INITIAL MODEL (cont’d) Tracking the model: The idea about creating a box is really useful when tracking the moving object. However, there are many things to be taken into account: When more than one object appear in the same scene we must try to identify each of them separately.When more than one object appear in the same scene we must try to identify each of them separately. Occlusion is one of the most challenging issues here. In that case we have to recognize which object is occluding the other. The color and texture information could help there.Occlusion is one of the most challenging issues here. In that case we have to recognize which object is occluding the other. The color and texture information could help there. We must also define when to stop tracking an object that hasn’t moved for a predetermined period of time.We must also define when to stop tracking an object that hasn’t moved for a predetermined period of time.

30. 30 The detection, creation, and extraction of the video object planes have involved a lot of work and given the variety of backgrounds, types of motion, ranges of motion, and many other factors it is a very interesting field of research. TRACKING THE MODEL RESULTS Right now I’m working on the “box” previously mentioned and I’m dealing, mainly, with the total elimination of the noise from the background. Once I accomplish this, the tracking process should be easily implemented. There are three main steps regarding this implementation:  Motion Detection  Model Initializing VOP Extraction  Model Tracking

31. 31 RESULTS (cont’d) So far I applied the algorithm for motion detection already shown in this report. The results I found are showed with the corresponding videos. Ass you will see when the object stop moving I’m able to get only the portion of the object that has moved. Sequence Name: BOWING Format: CIF – 4:2:0

32. 32 Sequence Name: HALL MONITOR Format: CIF – 4:2:0 RESULTS (cont’d) THE END