1. Jack Pinches INFO410 & INFO350 S2 2015 INFORMATION SCIENCE Computer Vision I

2. INFO410 S2 2015 COMPUTER VISION I SLIDE 2 INFORMATION SCIENCE Topics  Definition  Related Fields  Application Examples  Application Overview  Typical Tasks  Methods  Feature Detection  Feature Description  Feature Matching  Exam Questions  Resources

3. Defining Computer Vision

4. INFO410 S2 2015 COMPUTER VISION I SLIDE 4 INFORMATION SCIENCE Formal Definition  Making useful decisions about real physical objects and scenes based on images (Shapiro & Stockman, 2001).  Extracting descriptions of the world from pictures of sequences of pictures (Forsyth & Ponce, 2003).  Analysing images and producing descriptions that can be used to interact with the environment (Horn, 1986).  Designing representations and algorithms for relating images to models of the world (Ballard & Brown, 1982).

5. INFO410 S2 2015 COMPUTER VISION I SLIDE 5 INFORMATION SCIENCE Informal Definition  Consists of acquiring, processing, analysing images/video of the real world  To produce information or descriptions  Often used as part of decision making

6. INFO410 S2 2015 COMPUTER VISION I SLIDE 6 INFORMATION SCIENCE Definition  Extract “meaning” from pixels What we seeWhat a computer sees http://www.cs.unc.edu/~lazebnik/spring11/lec01_intro.pdf

7. INFO410 S2 2015 COMPUTER VISION I SLIDE 7 INFORMATION SCIENCE Why is it important?  Images and videos are everywhere, and growing in numbers.  Trying to mimic human vision and understanding with computers.  Humans have good visual understanding – large amount of the brain devoted to visual processing.  Difficult to mimic.  Human level visual perception may be strong AI or “AI complete”.

8. INFO410 S2 2015 COMPUTER VISION I SLIDE 8 INFORMATION SCIENCE Related Fields https://en.wikipedia.org/wiki/Computer_vision#/media/File:CVoverview2.svg

9. Application Examples

10. INFO410 S2 2015 COMPUTER VISION I SLIDE 10 INFORMATION SCIENCE Applications AgricultureAugmented RealityAutonomous VehiclesBiometricsCharacter RecognitionForensicsIndustrial Quality InspectionFace RecognitionMedical Image AnalysisPollution MonitoringProcess ControlRemote SensingRoboticsSecurity and SurveillanceTransport http://www.bmva.org/visionoverview

11. INFO410 S2 2015 COMPUTER VISION I SLIDE 11 INFORMATION SCIENCE Application Examples  Face Detection:  “Feature” included on many digital cameras https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect1.pdf\

12. INFO410 S2 2015 COMPUTER VISION I SLIDE 12 INFORMATION SCIENCE Application Examples  Automatic number plate recognition:  Used by law enforcement  Automatic tolling stations  Road surveillance https://en.wikipedia.org/wiki/File:California_license_plate_ANPR.png

13. INFO410 S2 2015 COMPUTER VISION I SLIDE 13 INFORMATION SCIENCE Application Examples  Facial Recognition:  Airport Security  Check the person matches the passport photo  Used in NZ http://www.secureidnews.com/news-item/uks-stansted-airport-deploys-biometric-e-passport-gates/

14. Application Overview

15. INFO410 S2 2015 COMPUTER VISION I SLIDE 15 INFORMATION SCIENCE Typical Tasks  Detection:  Image scanned for a specific condition. E.g. detection of humans as part of a security system, or vehicles at automatic toll system.  Recognition:  Pre-learned objects can be recognized. E.g. recognition of a specific album cover.  Description:  Extracting data about specific object of interest. E.g. colour information, 2D position or 3D position.

16. INFO410 S2 2015 COMPUTER VISION I SLIDE 16 INFORMATION SCIENCE Application Methods  Image Acquisition:  Image is produced by one or several image sensors. The image may have some associated meta data.  Pre-processing:  Process image to ensure it meets requirements.  Examples:  Re-sample the image to check it’s what’s expected to be outputted.  Noise reduction so sensor noise doesn’t add false information.  Contrast alteration so important features can be detected.

17. INFO410 S2 2015 COMPUTER VISION I SLIDE 17 INFORMATION SCIENCE Application Methods  Feature extraction:  Image features are extracted from the image data.  Examples:  Lines, edges, intersections  Complex examples could include texture, shape, or motion  Detection:  Decide which image regions are required for further processing.

18. INFO410 S2 2015 COMPUTER VISION I SLIDE 18 INFORMATION SCIENCE Application Methods  High level processing:  Usually small set of the original data. Image region which assume contains a specific object.  Examples:  Estimation of object pose or size  Object recognition  Final decision:  Make final decision for the application.  Examples:  Match/no match in recognition application  Flag if human review needed (medical, military applications)

19. Feature Detection

20. INFO410 S2 2015 COMPUTER VISION I SLIDE 20 INFORMATION SCIENCE Feature Detection  Methods that try to extract image information.  Decisions made if there is an image feature or not.  Features will be a subset of the image.  Different approaches available.

21. INFO410 S2 2015 COMPUTER VISION I SLIDE 21 INFORMATION SCIENCE Feature Detection – Image Features  Edges:  Points where there is a boundary between two image regions.  Can be almost any shape. http://stackoverflow.com/questions/11319937/c-sharp-paint-image-within-edges

22. INFO410 S2 2015 COMPUTER VISION I SLIDE 22 INFORMATION SCIENCE Feature Detection – Image Features  Corners (interest points):  Originally called corner as early algorithms performed edge detection. Then looked for edges which have rapid direction change.  Can look for high level of curvature. http://fahmifahim.com/2010/10/22/opencv-corner-detection/

23. INFO410 S2 2015 COMPUTER VISION I SLIDE 23 INFORMATION SCIENCE Feature Detection – Image Features  Blobs:  Aimed at detecting regions of an image.  The region may differ in properties (such as brightness or colour) compared to surrounding images. https://wwwx.cs.unc.edu/~sjguy/CompVis/Features/

24. Feature Description

25. INFO410 S2 2015 COMPUTER VISION I SLIDE 25 INFORMATION SCIENCE Feature Description  Feature Description Methods:  SIFT: Scale Invariant Feature Transform  SURF: Speeded up Robust Features  GLOH: Gradient Location and Orientation Histogram  HOG: Histogram of Orientated Gradients

26. INFO410 S2 2015 COMPUTER VISION I SLIDE 26 INFORMATION SCIENCE Feature Description - SIFT  Detect and describe features in images.  Developed by David Lowe in 1999.  Algorithm described in:  Lowe, David G. (1999). "Object recognition from local scale-invariant features". Proceedings of the International Conference on Computer Vision 2. pp. 1150–1157. doi:10.1109/ICCV.1999.790410  C++ Implementation:  http://www.robots.ox.ac.uk/~vedaldi/code/siftpp.html http://www.robots.ox.ac.uk/~vedaldi/code/siftpp.html https://copilosk.fbk.eu/images/f/f6/Sift.pdf

27. INFO410 S2 2015 COMPUTER VISION I SLIDE 27 INFORMATION SCIENCE  Input an image.  Output is a list of points on the image each associated to a vector of low level descriptors.  These points are called the keypoints.  Provide a local image description.  The points and their descriptions are invariant by rescaling, orientation, and some changes of illumination.  Used to find visual correspondences between images for different applications, such as image alignment or object recognition. Feature Description - SIFT

28. INFO410 S2 2015 COMPUTER VISION I SLIDE 28 INFORMATION SCIENCE Feature Description – SIFT Image Description 813 Keypoints https://copilosk.fbk.eu/images/f/f6/Sift.pdfhttps://copilosk.fbk.eu/images/f/f6/Sift.pdf (slide 4)

29. INFO410 S2 2015 COMPUTER VISION I SLIDE 29 INFORMATION SCIENCE Feature Description – SIFT Application  Align two images to produce a new image.

30. Feature Matching

31. INFO410 S2 2015 COMPUTER VISION I SLIDE 31 INFORMATION SCIENCE Feature Matching – Why do it?  Fusing image from different sensors or times  Detect changes between images  Produce a new large image by overlapping/stitching smaller images E.g. Panorama  Object recognition by comparing an image to a reference image or model

32. INFO410 S2 2015 COMPUTER VISION I SLIDE 32 INFORMATION SCIENCE Feature Matching

33. INFO410 S2 2015 COMPUTER VISION I SLIDE 33 INFORMATION SCIENCE Feature Matching  Different Techniques:  SIFT  RANSAC (Random Sample Consensus)  Cross Correlation  Nearest Neighbour Techniques  Exhaustive Comparison

34. INFO410 S2 2015 COMPUTER VISION I SLIDE 34 INFORMATION SCIENCE Feature Matching – What makes a good feature?  Uniqueness:  No ambiguous matches between images  Look for “interest points”: image regions that unique are (or at least unusual)  Defining unusual?

35. INFO410 S2 2015 COMPUTER VISION I SLIDE 35 INFORMATION SCIENCE Feature Matching – With SIFT  Can detect interest points, create feature descriptors.  Compare feature descriptors between images.

36. INFO410 S2 2015 COMPUTER VISION I SLIDE 36 INFORMATION SCIENCE Feature Matching – With Cross Correlation  Determine the important features:  Unique colour/structure of the neighbourhood  Geometry or topology of neighbourhood Vectors show the movement of the neighbourhood. www.cse.msu.edu/~stockman/RegTutorial/matchingPart1.ppt

37. INFO410 S2 2015 COMPUTER VISION I SLIDE 37 INFORMATION SCIENCE Feature Matching www.cse.msu.edu/~stockman/RegTutorial/matchingPart1.ppt 1)Determine points 2)Match Neighbourhoods

38. Potential Exam Questions

39. INFO410 S2 2015 COMPUTER VISION I SLIDE 39 INFORMATION SCIENCE Exam Questions  Define computer vision, and include the typical tasks that could be expected in a computer vision application.  Explain why it is important to use good feature descriptors for feature matching, use applications examples to support your reasoning.  Explain how you would design and implement a license plate detection and recognition application.

40. INFO410 S2 2015 COMPUTER VISION I SLIDE 40 INFORMATION SCIENCE Resources  Computer Vision:  http://www.cs.unc.edu/~lazebnik/spring11/ http://www.cs.unc.edu/~lazebnik/spring11/  http://users.eecs.northwestern.edu/~yingwu/teaching/EECS432/Notes/intro.pdf http://users.eecs.northwestern.edu/~yingwu/teaching/EECS432/Notes/intro.pdf  SIFT:  https://copilosk.fbk.eu/images/f/f6/Sift.pdf https://copilosk.fbk.eu/images/f/f6/Sift.pdf  http://www.quora.com/What-is-the-best-explanation-of-SIFT-that-you-have-seen-or-heard http://www.quora.com/What-is-the-best-explanation-of-SIFT-that-you-have-seen-or-heard  Features, Matching, and Detection:  http://www.cs.toronto.edu/~kyros/courses/2503/Handouts/features.pdf http://www.cs.toronto.edu/~kyros/courses/2503/Handouts/features.pdf  https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect6.pdf https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect6.pdf  www.cse.msu.edu/~stockman/RegTutorial/matchingPart1.ppt www.cse.msu.edu/~stockman/RegTutorial/matchingPart1.ppt  www.cse.msu.edu/~stockman/RegTutorial/matchingPart2.ppt www.cse.msu.edu/~stockman/RegTutorial/matchingPart2.ppt  http://people.cs.umass.edu/~elm/Teaching/ppt/370/370_10_RANSAC.pptx.pdf http://people.cs.umass.edu/~elm/Teaching/ppt/370/370_10_RANSAC.pptx.pdf