Presentation is loading. Please wait.

Presentation is loading. Please wait.

91.204.201 Computing IV Chapter Two: Core Module. The Core Functionality Xinwen Fu.

Similar presentations


Presentation on theme: "91.204.201 Computing IV Chapter Two: Core Module. The Core Functionality Xinwen Fu."— Presentation transcript:

1 Computing IV Chapter Two: Core Module. The Core Functionality Xinwen Fu

2 References  Application Development in Visual Studio Application Development in Visual Studio  Reading assignment: Chapter 2  An online OpenCV Quick Guide with nice examplesOpenCV Quick Guide By Dr. Xinwen Fu2

3 A few things  Blackboard submission  Report format  Sreenshots By Dr. Xinwen Fu3

4 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu4

5 2.1 Mat - The Basic Image Container  We have multiple ways to acquire digital images from the real world: digital cameras, scanners, computed tomography or magnetic resonance imaging to just name a few. In every case what we (humans) see are images.  When transforming this to our digital devices what we record are numerical values for each of the points of the image. By Dr. Xinwen Fu5

6 Storing Images  A black-white image is nothing more than a matrix containing all the intensity values of the pixel points.  How we get and store the pixels values may vary according to what fits best our need In the end all images inside a computer world may be reduced to numerical matrices and some other information describing the matric itself.  OpenCV is a computer vision library whose main focus is to process and manipulate these information to find out further ones. The first thing to learn and get accommodated with is how OpenCV stores and handles images. By Dr. Xinwen Fu6

7 Mat  Basically a class with two data parts: the matrix header (containing information such as the size of the matrix, the method used for storing, at which address is the matrix stored and so on) a pointer to the matrix containing the pixel values (may take any dimensionality depending on the method chosen for storing)  OpenCV is an image processing library, doing image processing with its functions By Dr. Xinwen Fu7

8 Copy Mat  OpenCV uses a reference counting system. The idea is that each Mat object has its own header.  However the matrix may be shared between two instance of them by having their matrix pointer point to the same address.  Copy operators will only copy the headers, and as also copy the pointer to the large matrix too, however not the matrix itself. By Dr. Xinwen Fu8

9 1. Mat A, C; // creates just the header parts 2. // here we'll know the method used (allocate matrix) 3. A=imread(file, CV_LOAD_IMAGE_COLOR); 4. Mat B(A); // use the copy constructor 5. C=A; // assignment operator  All the above objects, in the end point to the same single data matrix.  Their headers are different, however making any modification using either one of them will affect all the other ones too.  In practice the different objects just provide different access method to the same underlying data. Nevertheless, their header parts are different. By Dr. Xinwen Fu9

10 Refer only to a subsection of the full data  The real interesting part comes that you can create headers that refer only to a subsection of the full data.  For example, to create a region of interest (ROI) in an image you just create a new header with the new boundaries: By Dr. Xinwen Fu10 Coordinates of the top-left corner Rectangle width and height

11 Cleaning Mat  You may ask if the matrix itself may belong to multiple Mat objects who will take responsibility for its cleaning when it’s no longer needed. The short answer is: the last object that used it.  For this a reference counting mechanism is used Whenever somebody copies a header of a Mat object a counter is increased for the matrix. Whenever a header is cleaned this counter is decreased. When the counter reaches zero the matrix too is freed. By Dr. Xinwen Fu11

12 Copy the matrix itself  Because, sometimes you will still want to copy the matrix itself too, there exists the clone() or the copyTo() function.  Now modifying F or G will not affect the matrix pointed by the Mat header By Dr. Xinwen Fu12

13 Tips of using Mat  Output image allocation for OpenCV functions is automatic (unless specified otherwise).  No need to think about memory freeing with OpenCVs C++ interface.  The assignment operator and the copy constructor (ctor)copies only the header.  Use the clone() or the copyTo() function to copy the underlying matrix of an image. By Dr. Xinwen Fu13

14 Storing methods for pixel values.  You can select color space and data type used The color space refers to how we combine color components in order to code a given color.  The simplest one is the gray scale.  For colorful ways we have a lot more of methods to choose from. However, every one of them breaks it down to three or four basic components and the combination of this will give all others. The most popular one is RGB, mainly because this is also how our eye builds up colors in our eyes. Its base colors are red, green and blue. To code the transparency of a color sometimes a fourth element: alpha (A) is added. By Dr. Xinwen Fu14

15 Color Systems  RGB is the most common as our eyes use something similar, used by our display systems.  The HSV and HLS decompose colors into their hue, saturation and value/luminance components, which is a more natural way for us to describe colors. You may dismiss last component, making your algorithm less sensible to light conditions of the input image.  YCrCb is used by the popular JPEG image format.  CIE L*a*b* is a perceptually uniform color space, which comes handy if you need to measure the distance of a given color to another color. By Dr. Xinwen Fu15

16 Data Types for Color  Each of building components has their own valid domains. This leads to the data type used.  The smallest data type possible is char, which means one byte or 8 bits This may be unsigned (so can store values from 0 to 255) or signed (values from -127 to +127).  In case of three components this gives 16 million possible colors to represent (like in case of RGB) Even finer control by using float (4 byte = 32 bit) or double (8 byte = 64 bit) data types for each component.  However, increasing the size of a component also increases the size of the whole picture in the memory. By Dr. Xinwen Fu16

17 Creating explicitly a Mat object  Although Mat is a great class as image container it is also a general matrix class. Therefore, it is possible to create and manipulate multidimensional matrices.  You can create a Mat object in multiple ways  For two dimensional and multichannel images we first define their size: row and column count wise. By Dr. Xinwen Fu17

18  We need to specify the data type to use for storing the elements and the number of channels per matrix point. To do this we have multiple definitions made according to the following convention: CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number] CV_8UC3 means we use unsigned char types that are 8 bit long and each pixel has three items of this to form the three channels. This are predefined for up to four channel numbers.  The Scalar is four element short vector.  Only 2 dimension matrix can use cout By Dr. Xinwen Fu18

19 By Dr. Xinwen Fu19

20 Data Type  A primitive OpenCV data type is one of unsigned char, bool, signed char, unsigned short, signed short, int, float, double, or a tuple of values of one of these types, where all the values in the tuple have the same type.  Any primitive type from the list can be defined by an identifier in the form CV_ {U|S|F}C( ), for example: uchar ~ CV_8UC1, 3-element floating-point tuple ~ CV_32FC3  A universal OpenCV structure that is able to store a single instance of such a primitive data type is Vec. Multiple instances of such a type can be stored in a std::vector, Mat, Mat_, SparseMat, SparseMat_, or any other container that is able to store Vec instances. By Dr. Xinwen Fu20

21 By Dr. Xinwen Fu21

22 Print for other common items  OpenCV offers support for print of other common OpenCV data structures too via the << operator like  2D Point  3D Point  std::vector via cv::Mat By Dr. Xinwen Fu22

23 Print (Cont’d)  std::vector of points By Dr. Xinwen Fu23

24 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu24

25 Goal  How to go through each and every pixel of an image?  How is OpenCV matrix values stored?  How to measure the performance of our algorithm?  What are lookup tables and why use them? By Dr. Xinwen Fu25

26 Color space reduction  Divide the color space current value with a new input value to end up with fewer colors  For instance every value between zero and nine takes the new value zero, every value between ten and nineteen the value ten and so on. By Dr. Xinwen Fu26

27 How the image matrix is stored in the memory? - gray scale image  The size of the matrix depends of the color system used. More accurately, it depends from the number of channels used. By Dr. Xinwen Fu27

28 How the image matrix is stored in the memory? - RGB color system  For multichannel images the columns contain as many sub columns as the number of channels  Note that the order of the channels is inverse: BGR instead of RGB By Dr. Xinwen Fu28

29 Color reduction formula  When you divide an uchar (unsigned char - aka values between zero and 255) value with an int value the result will be also char. These values may only be char values. Therefore, any fraction will be rounded down.  Taking advantage of this fact the upper operation in the uchar domain may be expressed as: By Dr. Xinwen Fu29

30 Measure time code runs  Another issue is how do we measure time?  OpenCV offers two simple functions to achieve this getTickCount() and getTickFrequency(). The first returns the number of ticks of your systems CPU from a certain event (like since you booted your system). The second returns how many times your CPU emits a tick during a second. So to measure in seconds the number of time elapsed between two operations is easy as: By Dr. Xinwen Fu30

31 Lookup table for color reduction  how_to_scan_images imageName.jpg intValueToReduce [G] The final argument is optional. If given the image will be loaded in gray scale format, otherwise the RGB color way is used. The first thing is to calculate the lookup table. By Dr. Xinwen Fu31

32 The Efficient Way By Dr. Xinwen Fu32

33 Iterator By Dr. Xinwen Fu33

34 On-the-fly address calc  In case of color images we have three uchar items per column.  This may be considered a short vector of uchar items, that has been baptized in OpenCV with the Vec3b name. By Dr. Xinwen Fu34

35 On-the-fly address calc By Dr. Xinwen Fu35

36 The Core Function By Dr. Xinwen Fu36

37 Performance Difference  For the best result compile the program and run it on your own speed. For showing off better the differences I’ve used a quite large (2560 X 1600) image.  The performance presented here are for color images. For a more accurate value I’ve averaged the value I got from the call of the function for hundred times. By Dr. Xinwen Fu37

38 Mat_  If you need multiple lookups using this method for an image it may be troublesome and time consuming to enter the type and the at keyword for each of the accesses. To solve this problem OpenCV has a Mat_ data type.  It’s the same as Mat with the extra need that at definition you need to specify the data type through what to look at the data matrix, however in return you can use the operator() for fast access of items.  To make things even better this is easily convertible from and to the usual Mat data type. A sample usage of this you can see in case of the color images of the upper function. Nevertheless, it’s important to note that the same operation (with the same runtime speed) could have been done with the at() function. It’s just a less to write for the lazy programmer trick. By Dr. Xinwen Fu38

39 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu39

40 Mask operation  Recalculate each pixels value in an image according to a mask matrix (also known as kernel). This mask holds values that will adjust how much influence neighboring pixels (and the current pixel) have on the new pixel value. From a mathematical point of view we make a weighted average, with our specified values. By Dr. Xinwen Fu40

41 Basic Method 1. void Sharpen(const Mat& myImage,Mat& Result) 2. { 3. CV_Assert(myImage.depth() == CV_8U); // accept only uchar images 4. const int nChannels = myImage.channels(); 5. Result.create(myImage.size(),myImage.type()); 6. for(int j = 1 ; j < myImage.rows-1; ++j) 7. { 8. const uchar* previous = myImage.ptr (j - 1); 9. const uchar* current = myImage.ptr (j ); 10. const uchar* next = myImage.ptr (j + 1); 11. uchar* output = Result.ptr (j); By Dr. Xinwen Fu41

42 12. for(int i= nChannels;i < nChannels*(myImage.cols-1); ++i) 13. { 14. *output++ = saturate_cast (5*current[i] 15. -current[i-nChannels] - current[i+nChannels] - previous[i] - next[i]); 16. } 17. } 18. Result.row(0).setTo(Scalar(0)); 19. Result.row(Result.rows-1).setTo(Scalar(0)); 20. Result.col(0).setTo(Scalar(0)); 21. Result.col(Result.cols-1).setTo(Scalar(0)); 22. } By Dr. Xinwen Fu42

43 filter2D function  Applying such filters is so common in image processing that in OpenCV there exist a function that will take care of applying the mask (also called a kernel in some places). 1. define a Mat object that holds the mask: 2. Then call the filter2D function specifying the input, the output image and the kernell to use: By Dr. Xinwen Fu43

44 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu44

45 Theory  From our previous tutorial, we know already a bit of Pixel operators.  An interesting dyadic (two-input) operator is the linear blend operator:  By varying from 0 to 1 this operator can be used to perform a temporal cross- dissolve between two images or videos, as seen in slide shows and film productions By Dr. Xinwen Fu45

46 Example 1. #include 2. #include 3. #include 4. using namespace cv; 5. int main( int argc, char** argv ) 6. { 7. double alpha = 0.5; double beta; double input; 8. Mat src1, src2, dst; /// Ask the user enter alpha 11. std::cout<<" Simple Linear Blender "<>input; By Dr. Xinwen Fu46

47 Example 15. /// We use the alpha provided by the user iff it is between 0 and if( alpha >= 0 && alpha <= 1 ) { alpha = input; } 17. /// Read image ( same size, same type ) 18. src1 = imread("../../images/LinuxLogo.jpg"); 19. src2 = imread("../../images/WindowsLogo.jpg"); 20. if( !src1.data ) { printf("Error loading src1 \n"); return -1; } 21. if( !src2.data ) { printf("Error loading src2 \n"); return -1; } 22. /// Create Windows 23. namedWindow("Linear Blend", 1); 24. beta = ( alpha ); 25. addWeighted( src1, alpha, src2, beta, 0.0, dst); 26. imshow( "Linear Blend", dst ); 27. waitKey(0); 28. return 0; 29. } By Dr. Xinwen Fu47

48 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu48

49 Image Processing  A general image processing operator is a function that takes one or more input images and produces an output image.  Image transforms can be seen as: Point operators (pixel transforms) Neighborhood (area-based) operators By Dr. Xinwen Fu49

50 Pixel Transforms  In this kind of image processing transform, each output pixel’s value depends on only the corresponding input pixel value (plus, potentially, some globally collected information or parameters).  Examples of such operators include brightness and contrast adjustments as well as color correction and transformations. By Dr. Xinwen Fu50

51 Brightness and contrast adjustments  Two commonly used point processes are multiplication and addition with a constant: g(x) = α f(x) + β  The parameters α > 0 and β are often called the gain and bias parameters Sometimes these parameters are said to control contrast and brightness respectively.  You can think of f(x) as the source image pixels and g(x) as the output image pixels. Then, more conveniently we can write the expression as: g(i; j) = α f(i; j) + β where i and j indicates that the pixel is located in the i- th row and j-th column. By Dr. Xinwen Fu51

52 Example 1. #include 2. #include 3. #include 4. using namespace cv; 5. double alpha; /**< Simple contrast control */ 6. int beta; /**< Simple brightness control */ 7. int main( int argc, char** argv ) 8. { 9. /// Read image given by user 10. Mat image = imread( argv[1] ); 11. Mat new_image = Mat::zeros( image.size(), image.type() ); 12. /// Initialize values 13. std::cout<<" Basic Linear Transforms "<alpha; 16. std::cout >beta; By Dr. Xinwen Fu52

53 17. /// Do the operation new_image(i,j) = alpha*image(i,j) + beta 18. for( int y = 0; y < image.rows; y++ ) 19. { 20. for( int x = 0; x < image.cols; x++ ) 21. { 22. for( int c = 0; c < 3; c++ ) 23. { 24. new_image.at (y,x)[c] = 25. saturate_cast ( alpha*( image.at (y,x)[c] ) + beta ); 26. } 27. } 28. } 29. /// Create Windows 30. namedWindow("Original Image", 1); 31. namedWindow("New Image", 1); 32. /// Show stuff 33. imshow("Original Image", image); 34. imshow("New Image", new_image); 35. /// Wait until user press some key 36. waitKey(); 37. return 0; 38. } By Dr. Xinwen Fu53

54 Who is Lena?  Lena Söderberg, a Swedish model Lena Söderberg Swedish  cropped from the centerfold of November 1972 issue of Playboy magazine By Dr. Xinwen Fu54

55 Core function  Instead of using the for loops to access each pixel, we could have simply used this command: image.convertTo(new_image, -1, alpha, beta); where convertTo would effectively perform new_image = a*image + beta.  However, we wanted to show you how to access each pixel. In any case, both methods give the same result. By Dr. Xinwen Fu55

56 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu56

57 Example in Visual Studio 2010  Use Point to define 2D points in an image.  Use Scalar and why it is useful  Draw a line by using the OpenCV function line  Draw an ellipse by using the OpenCV function ellipse  Draw a rectangle by using the OpenCV function rectangle  Draw a circle by using the OpenCV function circle  Draw a filled polygon by using the OpenCV function fillPoly By Dr. Xinwen Fu57

58 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu58

59 Example in Visual Studio 2010  Use the Random Number generator class (RNG) and how to get a random number from a uniform distribution. RNG rng( 0xFFFFFFFF ); rng.uniform(a,b); // This generates a randomly uniformed distribution between the values a and b (inclusive in a, exclusive in b).  Display text on an OpenCV window by using the function putText By Dr. Xinwen Fu59

60 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu60

61 Skipped  What is a Fourier transform and why use it?  How to do it in OpenCV?  Usage of functions such as: copyMakeBorder(), merge(), dft(), getOptimalDFTSize(), log() and normalize(). By Dr. Xinwen Fu61

62 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu62

63 Skipped  How to print and read text entries to a file and OpenCV using YAML or XML files?  How to do the same for OpenCV data structures?  How to do this for your data structures?  Usage of OpenCV data structures such as FileStorage, FileNode or FileNodeIterator. By Dr. Xinwen Fu63

64 Outline  2.1 Mat - The Basic Image Container  2.2 How to scan images, lookup tables and time measurement with OpenCV  2.3 Mask operations on matrices  2.4 Adding (blending) two images using OpenCV  2.5 Changing the contrast and brightness of an image  2.6 Basic drawing  2.7 Random generator and text with OpenCV  2.8 Discrete Fourier Transform  2.9 File input and output using XML and YAML  2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu64

65 Skipped  What changed with the version 2 of OpenCV in the way you use the library compared to its first version  How to add some Gaussian noise to an image  What are lookup tables and why use them? By Dr. Xinwen Fu65

66 References  OpenCV documentation OpenCV documentation By Dr. Xinwen Fu66


Download ppt "91.204.201 Computing IV Chapter Two: Core Module. The Core Functionality Xinwen Fu."

Similar presentations


Ads by Google