1. 1 ECE 495 – Integrated System Design I Introduction to Image Processing ECE 495, Spring 2013

2. 2 ECE 495 – Integrated System Design I Overview What is image processing? What is an image? Introduction to image processing techniques Experimental setup Basic MATLAB commands for image processing

3. 3 ECE 495 – Integrated System Design I What is Image Processing? Image processing is the collective name for techniques used to extract information from digital images or to manipulate them to render variations of the input image. Photo stitchingColor boost Vehicle detection and tracking

4. 4 ECE 495 – Integrated System Design I What is Image Processing? Popular technologies which make use of the camera as a sensor The Wii Remote uses an IR camera to sense its location relative to the Wii Sensor Bar. The Kinect uses image processing techniques on depth images to detect and track locations of multiple persons in the field of view.

5. 5 ECE 495 – Integrated System Design I Pixels Pixel A pixel (abbr. for picture element) is the smallest unit of an image. Therefore, a 640x480 image is a matrix of 640 columns and 480 rows, each element of this matrix is called an image pixel.

6. 6 ECE 495 – Integrated System Design I MATLAB Image Coordinates MATLAB stores images as matrices. In MATLAB, image pixels are referenced using (row, col) values. Origin of the coordinate system (1,1) is the top left corner of the image img Thus, img(4,3) refers to the pixel at the 4 th row and 3 rd column. (1,1)

7. 7 ECE 495 – Integrated System Design I RGB and Grayscale In RGB format, each Pixel has 3 color components: Red, Green, and Blue. Other color representations, e.g. HSV, YUV, CMYK, are also used. Transformations from RGB to these color spaces and back are defined in MATLAB. If only intensity (bright/dark) variations are considered, the resultant image is called a grayscale image. Each pixel has only 1 component: intensity. RGBGray

8. 8 ECE 495 – Integrated System Design I Basic Image Processing techniques Background Subtraction Intensity Thresholding Morphological Operations Color-based Processing Shape-based Processing A combination of the above methods is usually used in basic image processing applications. There is no perfect solution, so keep discussing (and trying) what suits your project best.

9. 9 ECE 495 – Integrated System Design I Background Subtraction Background subtraction is a technique used to isolate useful information in an image (foreground) from the rest of the image (background). A reference image is selected as the background. Each successive image in a video stream is compared against this image. If the difference between the images is significant, the areas which are different are considered to be the foreground for that image.

10. 10 ECE 495 – Integrated System Design I Background Subtraction Reference image or background Incoming image contains a new object. This is our foreground, i.e. useful information for detecting the system state. The new object is identified as the foreground after the background is subtracted.

11. 11 ECE 495 – Integrated System Design I Intensity Thresholding Intensity thresholding is another foreground separation technique. It uses histograms. A histogram is an image statistic which usually operates on an intensity image, i.e. pixels having a single value between 0-255 The range 0-255 is divided into bins, e.g. each intensity value may be given its own bin The Y axis shows the count of number of pixels in an image which lie within limits for a bin. Bin

12. 12 ECE 495 – Integrated System Design I Intensity Thresholding Assumption: Foreground is sufficiently bright/dark compared to the background, i.e. the histogram has a clear separation. Assumption: The threshold value has been estimated using prior observations or can be computed automatically. – This method is very sensitive to lighting conditions Given these assumptions, set all pixels on the foreground side to 255 (bright), and all other pixels to 0 (dark). Foreground Threshold

13. 13 ECE 495 – Integrated System Design I Morphological Operations Morphological operations are used to either remove noise from an image or recover some details lost in thresholding or background subtraction. Dilation adds to the foreground areas in an image, but can also add to the noise. Erosion removes stray foreground pixels from it, but also reduces genuine foreground areas. Usually, a combination of morphological operators is used to improve the output. After dilationAfter erosion Original

14. 14 ECE 495 – Integrated System Design I Color Processing Used in identifying well color for the Minesweeper problem. In MATLAB, a color (RGB) image is stored as a 3D matrix with the third dimension being color. Each pixel has a Red, Green, and Blue value. The values range (for each color) from 0 to 255, e.g. a purely red pixel at (25,30) will have the RGB value (255,0,0). To access the Red value for this pixel in the 3D image matrix, we reference img(25,30,1). Similarly, Green will be img(25,30,2) and Blue, img(25,30,3). img

15. 15 ECE 495 – Integrated System Design I Color Processing Color processing algorithms need to have a tolerance to color, e.g. a RED sticker wont have RGB (255,0,0). This tolerance can be built into the algorithm by – Testing under varied ambient lighting conditions, e.g. morning, evening, and setting color thresholds accordingly. – Experimenting with more variation- tolerant color spaces, e.g. HSV. – Experimenting with logical operators, e.g. a RED sticker = (Red enough) AND (NOT Green enough) AND (NOT Blue enough). – Take average of pixel values in area Is there a red sticker in this well? RGB (173,58,86)

16. 16 ECE 495 – Integrated System Design I Shape-based Processing Once the foreground objects have been detected, it is important to know their properties such as area, shape, location, etc. MATLAB has a function called regionprops which takes a binary image as input and identifies properties of contiguous (connected) pixels in foreground regions. The output of regionprops is an array of structures. Each element of the array contains information about one foreground region. MATLAB help file is very useful Region 1 Region 2 Region 3

17. 17 ECE 495 – Integrated System Design I Shape-based Processing Regionprops can be used to filter out poor foreground detection results. For example: 1 2 3 4 5 6 All detected foreground regions Eliminate those which are too small [5] Eliminate those which are not circular [6] Eliminate those which are not at permitted locations [4] Genuine foreground detections 1 2 3 [1 2 3 4 5 6] [1 2 3 4 6] [1 2 3 4] [1 2 3]

18. 18 ECE 495 – Integrated System Design I Experimental setup When the camera is set up in a known configuration relative to the rest of the system, we can estimate physical dimensions using image data. For example: Camera field of view Fixed height 480 pixels 640 pixels 16 12 This implies 40 pixels/inch along image width 40 pixels/inch along image height A circle 1.25 diameter will be bounded by an area of: 1.25 * 40 pix/inch * 1.25 * 40 pix/inch = 2500 pix Use information to improve game state detection, criteria for algorithms Width Height

19. 19 ECE 495 – Integrated System Design I Some MATLAB functions NameFunctionality imreadRead input image imshowShow the image in a figure window rgb2grayConvert a RGB image to grayscale rgb2hsvConvert a RGB image to HSV imdilateDilation imerodeErosion im2bwConvert image to binary using thresholding regionpropsGet properties of connected components in an image imhistShow image histogram