Ramakrishna Lanka Ramakrishna.lanka@mavs.uta.edu 1001126160 IMAGE ANALYSIS AND OBJECT DETECTION-RECOGNITION USING COMPUTER VISION ALGORITHMS Ramakrishna Lanka Ramakrishna.lanka@mavs.uta.edu 1001126160

Contents Introduction to images and image processing Overview of computer vision Overview of image and video processing workflow Image enhancement Computer vision search algorithms and techniques Tools to be used References

PROPOSAL The objective of this project is to implement computer vision algorithms to analyze an image and detect objects in the image based on object reference images.

Acronyms 3D – Three dimensional CVIU – Computer vision and image understanding FAST – Features from accelerated segment test MSER – Maximally stable extremal regions PCB – Printed circuit board RANSAC – Random sample consensus RGB – Red, green and blue SURF – Speeded up robust features VTP – Virtual terrain Project

What is an Image ? Pictorial reproduction of an object in physical or digital space. A digital Image is made up of Pixels. More the pixels, much detailed is the Image as shown in figure(1). Resolution of an Image is measured by the number of pixels it consists of. Ex: 1366x768 Fig(1). Illustration of image pixels. [19]

Need for Image and video processing. Computer Vision: Object/Activity detection as shown in figure (15) and figure (16). Medical and biological image processing as shown in figure (10) and figure (11). After effects: Adding effects and image correction as shown in figure (2). Efficient use of media in Consumer multimedia products: Storage, display and transfer. Fig(2). Illustration of image after effects.

OVERVIEW OF COMPUTER VISION Vision has given humans the natural ability to picture the world around in a three-dimensional (3D) space. Computer vision gives the man-made device the ability to perceive the world. The goal of computer vision is to extract meaningful information from the images and videos that the image sensors capture.

OVERVIEW OF THE IMAGE AND VIDEO PROCESSING WORKFLOW Fig(2). Image and video processing workflow. [2]

OVERVIEW OF THE IMAGE AND VIDEO PROCESSING WORKFLOW The raw images from the source may contain unwanted image source characteristics or noise. Preprocessing to enhance the images will prepare it for further processing based on the requirement. Image enhancement can be achieved using various filtering techniques and transforms.

Fig(3). Original image of a PCB [21] Negative of an image Fig(3). Original image of a PCB [21] Fig(4). Histogram of the original PCB image N(r, c) = 255 – I(r, c) where 0 <= r <= R and 0 <= c <= C (1.1)   Where, N is the negative of image I, both with dimension R x C.

Fig(7). Negative image of the PCB Negative of an image Fig(7). Negative image of the PCB Fig(8). Histogram of the negative PCB image

Negative of an image Fig(10). Original image of a knee scan [22] Fig(11). Negative image of the knee scan figure (10) Fig(9). Histogram of the knee scan Fig(12). Histogram of the negative knee scan

Thresholding Transformation (1.2) Where s is the value of the resultant pixel intensity and r is the actual pixel intensity. Fig(13). Image of some coins [28] Fig(14). Threshold image of figure(13)

Thresholding Transformation Multilevel thresholding Fig(15). Image of obstacles [24] Fig(16). Multilevel threshold image of figure(15)

Thresholding Transformation Thresholding – color labelling Fig(17). Image of my cluttered desk Fig(18). Threshold segmentation of figure(17)

Powers-Law Transformations / Gamma correction s = crγ (1.3) Where, s is the corrected pixel intensity, r is the actual pixel intensity, c and γ are gamma correction constants. Fig(19). Image of the gamma palette [27] Fig(20). Gamma corrected image of figure(19) with γ=0.7 Fig(21). Gamma corrected image of figure(19) with γ=0.5

Histogram Equalization h(rk) = nk (1.4) Where, h(rk) is the histogram function, rk is the intensity value and nk is the number of pixels with the intensity rk.[4]

Computer Vision search algorithms and techniques Edge feature detection The Sobel operator The Prewitt operators The Robert operator Laplacian/Laplacian of Gaussian operator Canny edge detection Fig(24). Reference object image [25] Fig(25). Object in scene image [25]

Edge feature detection The Sobel operator (1.5) Where Gx is the vertical filter and Gy is the horizontal filter.[14][15] Figure(26) and figure(27) show the edge detections of figure(24) and figure(25). Fig(26). Sobel edge detection of figure(24) Fig(27). Sobel edge detection of figure(25)

Edge feature detection The Prewitt operators (1.6) Where Gx is the vertical filter and Gy is the horizontal filter.[14][15] Figure(28) and figure(29) show the edge detections of figure(24) and figure(25). Fig(28). Prewitt edge detection of figure(24) Fig(29). Prewitt edge detection of figure(25)

Edge feature detection The Robert operator (1.7) Where Gx is the vertical filter and Gy is the horizontal filter.[14][15] Figure(30) and figure(31) show the edge detections of figure(24) and figure (25).

Edge feature detection The Laplacian/Laplacian of Gaussian operator The Laplacian of an image highlights regions of rapid intensity change. The Laplacian is often applied to an image that has first been smoothed with something approximating a Gaussian smoothing filter in order to reduce its sensitivity to noise. [14][15]

Edge feature detection Canny edge detection Low error rate. Edge points should be well localized. Single edge point response.

Computer Vision search algorithms and techniques Local point/region detection Corner point detection – FAST (Features from accelerated segment test ) MSER(Maximally stable extremal regions) feature detection The Robert operator SURF(Speeded up robust features) feature detection. Fig(24). Reference object image [25] Fig(25). Object in scene image [25]

Local point/region feature detection Corner point feature detection - FAST corner detector uses a circle of 16 pixels (a Bresenham circle of radius 3) to classify whether a candidate point p is actually a corner. [31]

Local point/region feature detection Maximally stable extremal regions (MSER) MSER is a method for blob detection in images. The MSER algorithm extracts from an image a number of co-variant regions, called MSERs: an MSER is a stable connected component of some gray-level sets of the image. [29]

Local point/region feature detection Speeded up robust features (SURF) SURF is a fast and robust algorithm for local, similarity invariant representation and comparison. SURF selects interest points of an image from the salient features

Object recognition Recognition [9][10] Recognition is a process that follows detection and/or segmentation. Recognition approaches in this project, Object feature recognition [10] Recognition with point feature matching Recognition with edge feature matching Object Segmentation recognition [1] Recognition with segmentation

Object feature recognition Recognition with point feature matching [1][10] Match the extracted feature points of reference and scene images. The RANSAC (Random sample consensus ) algorithm works by estimating the best set of point matches between the images. This is shown in figure(43). Fig(43). Feature matching between figure(25) and figure(26)

Object feature recognition Recognition with point feature matching [1][10] Fig(44). Object in figure(25) recognized in figure(26)

Project status The aim of this project is to analyze and enhance images; and design a system to identify objects in an image. The identity of the objects will be achieved by using feature detection and recognition based on the features of the reference objects that the system would have learnt before recognition. Image analysis : Done Image Enhancement : Done Feature detection : Done Feature Extraction : Done Segmentation : Done Performance comparison : Ongoing Recognition with point feature matching : Done Recognition with edge feature matching : Done (yet to document) Recognition with segmentation : Done (yet to document) Multiple object recognition : Done (yet to document)

Tools being used Matlab – Image processing tool box [13] Matlab – Computer vision tool box [9] Matlab – R2013a OpenCV v2.4.11 [10] Microsoft Visual Studio v2010 Windows 8.1 64bit Central processor – Intel® Core™ i3-3120 2.50Ghz Graphics processor – NVidia GT 720M 2GB Memory – 8.00 GB

References [1] R. Szeliski, “Computer Vision: Algorithms and Applications”, 2010 Springer, Draft: 3, September 2010. [2] A. Thé, “Image Processing Made Easy”, MathWorks Videos and Webinars, Recorded: 15 Oct 2005. http://www.mathworks.com/videos/image-processing-made-easy-81718.html?form_seq=conf1050 [3] VTP software tutorials webpage (Source for hazy image): http://vterrain.org/Doc/Tut3/ [4] R. Maini and H. Aggarwal, “A Comprehensive Review of Image Enhancement Techniques”, Journal of Computing, Volume 2, Issue 3, pp 8-12 March 2010. : http://arxiv.org/ftp/arxiv/papers/1003/1003.4053.pdf [5] T. Gupta, “Seeing through Chaos: Noise Filtering”, ahumaninmachinesworld.blogspot, January 2014 (Internet Blog on Noise Filtering): http://ahumaninmachinesworld.blogspot.com/2014/01/seeing-through-chaos-noise-filtering.html [6] Computer Vision Algorithms, “COMPUTER VISION TUTORIAL”. Link: http://www.societyofrobots.com/programming_computer_vision_tutorial_pt3.shtml#middle_mass [7] H. Spont´on and J. Cardelino, “A Review of Classic Edge Detectors”, Image Processing On Line, Preprint Issue, pp 1-8, June 2012. Link: http://www.ipol.im/pub/pre/35/preprint.pdf

References [8] ‘Performance criteria’, Document in the Course website of EE5356 Digital Image processing (rate the performance of the filtered/enhanced image). Link: http://www.uta.edu/faculty/krrao/dip/Courses/EE5356/performance.pdf [9] Matlab computer vision toolbox: http://www.mathworks.com/help/vision/ [10] Open source Computer vision software: OpenCV v3.0 – download: http://opencv.org/downloads.html [11] Open source Computer vision software: OpenCV v3.0 – Reference Manual: http://docs.opencv.org/3.0-beta/modules/refman.html [12] R.C. Gonzalez, R.E.Woods and S.L. Eddins, “Digital Image Processing Using MATLAB”, Pearson Education, 2004. [13] Matlab image processing toolbox: http://www.mathworks.com/products/image/index-b.html [14] A. K. Jain, “Fundamentals of Digital Image Processing”. Englewood Cliffs, NJ: Prentice Hall, 1989.

References [15] R.C. Gonzalez and R.E. Woods, “Digital Image Processing”. 3nd edition, Prentice Hall, 2007. [16] Line detection image: http://www.camera-sdk.com/p_253-how-to-accomplish-line-detection-in-c-onvif.html [17] Open source Computer vision software: OpenCV v3.0 – User Guide: http://docs.opencv.org/3.0-beta/doc/user_guide/user_guide.html [18] Image segmentation example link: http://www.bioclinica.com/sites/default/files/u1/atlas-based-brain-segmentation-lg.jpg [19] Test images: http://www.imageprocessingplace.com/root_files_V3/image_databases.htm [20] Computer vision test image: http://www.cs.cmu.edu/afs/cs/project/cil/www/v-images.html [21] PCB image: http://images.fineartamerica.com/images-medium-large-5/printed-circuit-board-russell-shively.jpg

References [22] Medical image – knee: http://www.leutron.com/english/product/vip_d.htm [23] Geometric shapes image: http://www.maxton.com/gallery/con3-all2a.jpg [24] Need for image processing, computer vision path planning image: http://www.juliangough.com/photos/assorted-stolen-images-i-like/1107153 [25] Stop reference image: http://safety.fhwa.dot.gov/intersection/resources/casestudies/fhwasa09010/images/f1.jpg [26] Stop scene image: http://www.driversedguru.com/wp-content/gallery/jamie-gallery/Stop%20Sign.jpg [27] Gamma correction image: http://www.maximumpc.com/files/u16580/HowMonTest1.jpg [28] Coins image: http://upload.wikimedia.org/wikipedia/en/e/ec/Eagle_coins.jpg

References [29] Ppt on MSER : http://www.micc.unifi.it/delbimbo/wp-content/uploads/2011/03/slide_corso/A34%20MSER.pdf [30] H. Bay, A. Ess, T.Tuytelaars and L.V. Gool, "SURF: Speeded Up Robust Features", Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp. 346--359, June 2008. [31] PowerPoint presentation on FAST: http://web.eecs.umich.edu/~silvio/teaching/EECS598_2010/slides/11_16_Hao.pdf