Chapter Two: Core Module. The Core Functionality Xinwen Fu 91.204.201 Computing IV Chapter Two: Core Module. The Core Functionality Xinwen Fu

References Application Development in Visual Studio Reading assignment: Chapter 2 An online OpenCV Quick Guide with nice examples By Dr. Xinwen Fu

A few things Blackboard submission Report format Sreenshots By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

2.1 Mat - The Basic Image Container We have multiple ways to acquire digital images from the real world: digital cameras, scanners, computed tomography or magnetic resonance imaging to just name a few. In every case what we (humans) see are images. When transforming this to our digital devices what we record are numerical values for each of the points of the image. By Dr. Xinwen Fu

Storing Images A black-white image is nothing more than a matrix containing all the intensity values of the pixel points. How we get and store the pixels values may vary according to what fits best our need In the end all images inside a computer world may be reduced to numerical matrices and some other information describing the matric itself. OpenCV is a computer vision library whose main focus is to process and manipulate these information to find out further ones. The first thing to learn and get accommodated with is how OpenCV stores and handles images. By Dr. Xinwen Fu

Mat Basically a class with two data parts: the matrix header (containing information such as the size of the matrix, the method used for storing, at which address is the matrix stored and so on) a pointer to the matrix containing the pixel values (may take any dimensionality depending on the method chosen for storing) OpenCV is an image processing library, doing image processing with its functions By Dr. Xinwen Fu

Copy Mat OpenCV uses a reference counting system. The idea is that each Mat object has its own header. However the matrix may be shared between two instance of them by having their matrix pointer point to the same address. Copy operators will only copy the headers, and as also copy the pointer to the large matrix too, however not the matrix itself. By Dr. Xinwen Fu

Mat A, C; // creates just the header parts // here we'll know the method used (allocate matrix) A=imread(file, CV_LOAD_IMAGE_COLOR); Mat B(A); // use the copy constructor C=A; // assignment operator All the above objects, in the end point to the same single data matrix. Their headers are different, however making any modification using either one of them will affect all the other ones too. In practice the different objects just provide different access method to the same underlying data. Nevertheless, their header parts are different. By Dr. Xinwen Fu

Refer only to a subsection of the full data The real interesting part comes that you can create headers that refer only to a subsection of the full data. For example, to create a region of interest (ROI) in an image you just create a new header with the new boundaries: Coordinates of the top-left corner Rectangle width and height class Rect_ Template class for 2D rectangles, described by the following parameters: Coordinates of the top-left corner. This is a default interpretation of Rect_::x and Rect_::y in OpenCV. Though, in your algorithms you may count x and y from the bottom-left corner. Rectangle width and height. typedef Rect_<int> Rect; By Dr. Xinwen Fu

Cleaning Mat You may ask if the matrix itself may belong to multiple Mat objects who will take responsibility for its cleaning when it’s no longer needed. The short answer is: the last object that used it. For this a reference counting mechanism is used Whenever somebody copies a header of a Mat object a counter is increased for the matrix. Whenever a header is cleaned this counter is decreased. When the counter reaches zero the matrix too is freed. By Dr. Xinwen Fu

Copy the matrix itself Because, sometimes you will still want to copy the matrix itself too, there exists the clone() or the copyTo() function. Now modifying F or G will not affect the matrix pointed by the Mat header By Dr. Xinwen Fu

Tips of using Mat Output image allocation for OpenCV functions is automatic (unless specified otherwise). No need to think about memory freeing with OpenCVs C++ interface. The assignment operator and the copy constructor (ctor)copies only the header. Use the clone() or the copyTo() function to copy the underlying matrix of an image. By Dr. Xinwen Fu

Storing methods for pixel values. You can select color space and data type used The color space refers to how we combine color components in order to code a given color. The simplest one is the gray scale. For colorful ways we have a lot more of methods to choose from. However, every one of them breaks it down to three or four basic components and the combination of this will give all others. The most popular one is RGB, mainly because this is also how our eye builds up colors in our eyes. Its base colors are red, green and blue. To code the transparency of a color sometimes a fourth element: alpha (A) is added. By Dr. Xinwen Fu

Color Systems RGB is the most common as our eyes use something similar, used by our display systems. The HSV and HLS decompose colors into their hue, saturation and value/luminance components, which is a more natural way for us to describe colors. You may dismiss last component, making your algorithm less sensible to light conditions of the input image. YCrCb is used by the popular JPEG image format. CIE L*a*b* is a perceptually uniform color space, which comes handy if you need to measure the distance of a given color to another color. By Dr. Xinwen Fu

Data Types for Color Each of building components has their own valid domains. This leads to the data type used. The smallest data type possible is char, which means one byte or 8 bits This may be unsigned (so can store values from 0 to 255) or signed (values from -127 to +127). In case of three components this gives 16 million possible colors to represent (like in case of RGB) Even finer control by using float (4 byte = 32 bit) or double (8 byte = 64 bit) data types for each component. However, increasing the size of a component also increases the size of the whole picture in the memory. By Dr. Xinwen Fu

Creating explicitly a Mat object Although Mat is a great class as image container it is also a general matrix class. Therefore, it is possible to create and manipulate multidimensional matrices. You can create a Mat object in multiple ways For two dimensional and multichannel images we first define their size: row and column count wise. By Dr. Xinwen Fu

The Scalar is four element short vector. We need to specify the data type to use for storing the elements and the number of channels per matrix point. To do this we have multiple definitions made according to the following convention: CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number] CV_8UC3 means we use unsigned char types that are 8 bit long and each pixel has three items of this to form the three channels. This are predefined for up to four channel numbers. The Scalar is four element short vector. Only 2 dimension matrix can use cout class Scalar_ Template class for a 4-element vector derived from Vec. template<typename _Tp> class Scalar_ : public Vec<_Tp, 4> { ... }; typedef Scalar_<double> Scalar; Being derived from Vec<_Tp, 4> , Scalar_ and Scalar can be used just as typical 4-element vectors. In addition, they can be converted to/from CvScalar . The type Scalar is widely used in OpenCV to pass pixel values. By Dr. Xinwen Fu

By Dr. Xinwen Fu

Data Type A primitive OpenCV data type is one of unsigned char, bool, signed char, unsigned short, signed short, int, float, double, or a tuple of values of one of these types, where all the values in the tuple have the same type. Any primitive type from the list can be defined by an identifier in the form CV_<bit-depth>{U|S|F}C(<number_of_channels>), for example: uchar ~ CV_8UC1, 3-element floating-point tuple ~ CV_32FC3 A universal OpenCV structure that is able to store a single instance of such a primitive data type is Vec. Multiple instances of such a type can be stored in a std::vector, Mat, Mat_, SparseMat, SparseMat_, or any other container that is able to store Vec instances. By Dr. Xinwen Fu

By Dr. Xinwen Fu

Print for other common items OpenCV offers support for print of other common OpenCV data structures too via the << operator like 2D Point 3D Point std::vector via cv::Mat By Dr. Xinwen Fu

Print (Cont’d) std::vector of points By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Goal How to go through each and every pixel of an image? How is OpenCV matrix values stored? How to measure the performance of our algorithm? What are lookup tables and why use them? By Dr. Xinwen Fu

Color space reduction Divide the color space current value with a new input value to end up with fewer colors For instance every value between zero and nine takes the new value zero, every value between ten and nineteen the value ten and so on. By Dr. Xinwen Fu

How the image matrix is stored in the memory? - gray scale image The size of the matrix depends of the color system used. More accurately, it depends from the number of channels used. By Dr. Xinwen Fu

How the image matrix is stored in the memory? - RGB color system For multichannel images the columns contain as many sub columns as the number of channels Note that the order of the channels is inverse: BGR instead of RGB By Dr. Xinwen Fu

Color reduction formula When you divide an uchar (unsigned char - aka values between zero and 255) value with an int value the result will be also char. These values may only be char values. Therefore, any fraction will be rounded down. Taking advantage of this fact the upper operation in the uchar domain may be expressed as: By Dr. Xinwen Fu

Measure time code runs Another issue is how do we measure time? OpenCV offers two simple functions to achieve this getTickCount() and getTickFrequency(). The first returns the number of ticks of your systems CPU from a certain event (like since you booted your system). The second returns how many times your CPU emits a tick during a second. So to measure in seconds the number of time elapsed between two operations is easy as: By Dr. Xinwen Fu

Lookup table for color reduction how_to_scan_images imageName.jpg intValueToReduce [G] The final argument is optional. If given the image will be loaded in gray scale format, otherwise the RGB color way is used. The first thing is to calculate the lookup table. By Dr. Xinwen Fu

The Efficient Way By Dr. Xinwen Fu

Iterator By Dr. Xinwen Fu

On-the-fly address calc In case of color images we have three uchar items per column. This may be considered a short vector of uchar items, that has been baptized in OpenCV with the Vec3b name. By Dr. Xinwen Fu

On-the-fly address calc By Dr. Xinwen Fu

The Core Function By Dr. Xinwen Fu

Performance Difference For the best result compile the program and run it on your own speed. For showing off better the differences I’ve used a quite large (2560 X 1600) image. The performance presented here are for color images. For a more accurate value I’ve averaged the value I got from the call of the function for hundred times. By Dr. Xinwen Fu

Mat_ If you need multiple lookups using this method for an image it may be troublesome and time consuming to enter the type and the at keyword for each of the accesses. To solve this problem OpenCV has a Mat_ data type. It’s the same as Mat with the extra need that at definition you need to specify the data type through what to look at the data matrix, however in return you can use the operator() for fast access of items. To make things even better this is easily convertible from and to the usual Mat data type. A sample usage of this you can see in case of the color images of the upper function. Nevertheless, it’s important to note that the same operation (with the same runtime speed) could have been done with the at() function. It’s just a less to write for the lazy programmer trick. By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Mask operation Recalculate each pixels value in an image according to a mask matrix (also known as kernel). This mask holds values that will adjust how much influence neighboring pixels (and the current pixel) have on the new pixel value. From a mathematical point of view we make a weighted average, with our specified values. By Dr. Xinwen Fu

Basic Method void Sharpen(const Mat& myImage,Mat& Result) { CV_Assert(myImage.depth() == CV_8U); // accept only uchar images const int nChannels = myImage.channels(); Result.create(myImage.size(),myImage.type()); for(int j = 1 ; j < myImage.rows-1; ++j) const uchar* previous = myImage.ptr<uchar>(j - 1); const uchar* current = myImage.ptr<uchar>(j ); const uchar* next = myImage.ptr<uchar>(j + 1); uchar* output = Result.ptr<uchar>(j); By Dr. Xinwen Fu

for(int i= nChannels;i < nChannels*(myImage.cols-1); ++i) { *output++ = saturate_cast<uchar>(5*current[i] -current[i-nChannels] - current[i+nChannels] - previous[i] - next[i]); } Result.row(0).setTo(Scalar(0)); Result.row(Result.rows-1).setTo(Scalar(0)); Result.col(0).setTo(Scalar(0)); Result.col(Result.cols-1).setTo(Scalar(0)); By Dr. Xinwen Fu

filter2D function Applying such filters is so common in image processing that in OpenCV there exist a function that will take care of applying the mask (also called a kernel in some places). define a Mat object that holds the mask: Then call the filter2D function specifying the input, the output image and the kernell to use: By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Theory From our previous tutorial, we know already a bit of Pixel operators. An interesting dyadic (two-input) operator is the linear blend operator: By varying from 0 to 1 this operator can be used to perform a temporal cross-dissolve between two images or videos, as seen in slide shows and film productions By Dr. Xinwen Fu

Example #include <opencv/cv.h> #include <opencv/highgui.h> #include <iostream> using namespace cv; int main( int argc, char** argv ) { double alpha = 0.5; double beta; double input; Mat src1, src2, dst; /// Ask the user enter alpha std::cout<<" Simple Linear Blender "<<std::endl; std::cout<<"-----------------------"<<std::endl; std::cout<<"* Enter alpha [0-1]: "; std::cin>>input; By Dr. Xinwen Fu

Example /// We use the alpha provided by the user iff it is between 0 and 1 if( alpha >= 0 && alpha <= 1 ) { alpha = input; } /// Read image ( same size, same type ) src1 = imread("../../images/LinuxLogo.jpg"); src2 = imread("../../images/WindowsLogo.jpg"); if( !src1.data ) { printf("Error loading src1 \n"); return -1; } if( !src2.data ) { printf("Error loading src2 \n"); return -1; } /// Create Windows namedWindow("Linear Blend", 1); beta = ( 1.0 - alpha ); addWeighted( src1, alpha, src2, beta, 0.0, dst); imshow( "Linear Blend", dst ); waitKey(0); return 0; } By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Image Processing A general image processing operator is a function that takes one or more input images and produces an output image. Image transforms can be seen as: Point operators (pixel transforms) Neighborhood (area-based) operators By Dr. Xinwen Fu

Pixel Transforms In this kind of image processing transform, each output pixel’s value depends on only the corresponding input pixel value (plus, potentially, some globally collected information or parameters). Examples of such operators include brightness and contrast adjustments as well as color correction and transformations. By Dr. Xinwen Fu

Brightness and contrast adjustments Two commonly used point processes are multiplication and addition with a constant: g(x) = α f(x) + β The parameters α > 0 and β are often called the gain and bias parameters Sometimes these parameters are said to control contrast and brightness respectively. You can think of f(x) as the source image pixels and g(x) as the output image pixels. Then, more conveniently we can write the expression as: g(i; j) = α f(i; j) + β where i and j indicates that the pixel is located in the i-th row and j-th column. By Dr. Xinwen Fu

Example #include <opencv/cv.h> #include <opencv/highgui.h> #include <iostream> using namespace cv; double alpha; /**< Simple contrast control */ int beta; /**< Simple brightness control */ int main( int argc, char** argv ) { /// Read image given by user Mat image = imread( argv[1] ); Mat new_image = Mat::zeros( image.size(), image.type() ); /// Initialize values std::cout<<" Basic Linear Transforms "<<std::endl; std::cout<<"-------------------------"<<std::endl; std::cout<<"* Enter the alpha value [1.0-3.0]: ";std::cin>>alpha; std::cout<<"* Enter the beta value [0-100]: "; std::cin>>beta; By Dr. Xinwen Fu

/// Do the operation new_image(i,j) = alpha*image(i,j) + beta for( int y = 0; y < image.rows; y++ ) { for( int x = 0; x < image.cols; x++ ) for( int c = 0; c < 3; c++ ) new_image.at<Vec3b>(y,x)[c] = saturate_cast<uchar>( alpha*( image.at<Vec3b>(y,x)[c] ) + beta ); } /// Create Windows namedWindow("Original Image", 1); namedWindow("New Image", 1); /// Show stuff imshow("Original Image", image); imshow("New Image", new_image); /// Wait until user press some key waitKey(); return 0; By Dr. Xinwen Fu

Who is Lena? Lena Söderberg, a Swedish model cropped from the centerfold of November 1972 issue of Playboy magazine http://en.wikipedia.org/wiki/Lenna By Dr. Xinwen Fu

Core function Instead of using the for loops to access each pixel, we could have simply used this command: image.convertTo(new_image, -1, alpha, beta); where convertTo would effectively perform new_image = a*image + beta. However, we wanted to show you how to access each pixel. In any case, both methods give the same result. By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Example in Visual Studio 2010 Use Point to define 2D points in an image. Use Scalar and why it is useful Draw a line by using the OpenCV function line Draw an ellipse by using the OpenCV function ellipse Draw a rectangle by using the OpenCV function rectangle Draw a circle by using the OpenCV function circle Draw a filled polygon by using the OpenCV function fillPoly By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Example in Visual Studio 2010 Use the Random Number generator class (RNG) and how to get a random number from a uniform distribution. RNG rng( 0xFFFFFFFF ); rng.uniform(a,b); // This generates a randomly uniformed distribution between the values a and b (inclusive in a, exclusive in b). Display text on an OpenCV window by using the function putText By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Skipped What is a Fourier transform and why use it? How to do it in OpenCV? Usage of functions such as: copyMakeBorder(), merge(), dft(), getOptimalDFTSize(), log() and normalize() . By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Skipped How to print and read text entries to a file and OpenCV using YAML or XML files? How to do the same for OpenCV data structures? How to do this for your data structures? Usage of OpenCV data structures such as FileStorage, FileNode or FileNodeIterator. By Dr. Xinwen Fu

Outline 2.1 Mat - The Basic Image Container 2.2 How to scan images, lookup tables and time measurement with OpenCV 2.3 Mask operations on matrices 2.4 Adding (blending) two images using OpenCV 2.5 Changing the contrast and brightness of an image 2.6 Basic drawing 2.7 Random generator and text with OpenCV 2.8 Discrete Fourier Transform 2.9 File input and output using XML and YAML 2.10 Interoperability with OpenCV 1 By Dr. Xinwen Fu

Skipped What changed with the version 2 of OpenCV in the way you use the library compared to its first version How to add some Gaussian noise to an image What are lookup tables and why use them? By Dr. Xinwen Fu

References OpenCV documentation By Dr. Xinwen Fu