1. Chapter Two: Core Module. The Core Functionality Xinwen Fu
Computing IV
Chapter Two: Core Module. The Core Functionality
Xinwen Fu

2. References Application Development in Visual Studio
Reading assignment: Chapter 2
An online OpenCV Quick Guide with nice examples
By Dr. Xinwen Fu

3. A few things Blackboard submission Report format Sreenshots
By Dr. Xinwen Fu

4. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

5. 2.1 Mat - The Basic Image Container
We have multiple ways to acquire digital images from the real world: digital cameras, scanners, computed tomography or magnetic resonance imaging to just name a few. In every case what we (humans) see are images.
When transforming this to our digital devices what we record are numerical values for each of the points of the image.
By Dr. Xinwen Fu

6. Storing Images
A black-white image is nothing more than a matrix containing all the intensity values of the pixel points.
How we get and store the pixels values may vary according to what fits best our need
In the end all images inside a computer world may be reduced to numerical matrices and some other information describing the matric itself.
OpenCV is a computer vision library whose main focus is to process and manipulate these information to find out further ones.
The first thing to learn and get accommodated with is how OpenCV stores and handles images.
By Dr. Xinwen Fu

7. Mat Basically a class with two data parts:
the matrix header (containing information such as the size of the matrix, the method used for storing, at which address is the matrix stored and so on)
a pointer to the matrix containing the pixel values (may take any dimensionality depending on the method chosen for storing)
OpenCV is an image processing library, doing image processing with its functions
By Dr. Xinwen Fu

8. Copy Mat
OpenCV uses a reference counting system. The idea is that each Mat object has its own header.
However the matrix may be shared between two instance of them by having their matrix pointer point to the same address.
Copy operators will only copy the headers, and as also copy the pointer to the large matrix too, however not the matrix itself.
By Dr. Xinwen Fu

9. Mat A, C; // creates just the header parts
// here we'll know the method used (allocate matrix)
A=imread(file, CV_LOAD_IMAGE_COLOR);
Mat B(A); // use the copy constructor
C=A; // assignment operator
All the above objects, in the end point to the same single data matrix.
Their headers are different, however making any modification using either one of them will affect all the other ones too.
In practice the different objects just provide different access method to the same underlying data. Nevertheless, their header parts are different.
By Dr. Xinwen Fu

10. Refer only to a subsection of the full data
The real interesting part comes that you can create headers that refer only to a subsection of the full data.
For example, to create a region of interest (ROI) in an image you just create a new header with the new boundaries:
Coordinates of the top-left corner
Rectangle width and height
class Rect_ Template class for 2D rectangles, described by the following parameters:
Coordinates of the top-left corner. This is a default interpretation of Rect_::x and Rect_::y in OpenCV. Though, in your algorithms you may count x and y from the bottom-left corner.
Rectangle width and height.
typedef Rect_<int> Rect;
By Dr. Xinwen Fu

11. Cleaning Mat
You may ask if the matrix itself may belong to multiple Mat objects who will take responsibility for its cleaning when it’s no longer needed.
The short answer is: the last object that used it.
For this a reference counting mechanism is used
Whenever somebody copies a header of a Mat object a counter is increased for the matrix.
Whenever a header is cleaned this counter is decreased.
When the counter reaches zero the matrix too is freed.
By Dr. Xinwen Fu

12. Copy the matrix itself
Because, sometimes you will still want to copy the matrix itself too, there exists the clone() or the copyTo() function.
Now modifying F or G will not affect the matrix pointed by the Mat header
By Dr. Xinwen Fu

13. Tips of using Mat
Output image allocation for OpenCV functions is automatic (unless specified otherwise).
No need to think about memory freeing with OpenCVs C++ interface.
The assignment operator and the copy constructor (ctor)copies only the header.
Use the clone() or the copyTo() function to copy the underlying matrix of an image.
By Dr. Xinwen Fu

14. Storing methods for pixel values.
You can select color space and data type used
The color space refers to how we combine color components in order to code a given color.
The simplest one is the gray scale.
For colorful ways we have a lot more of methods to choose from. However, every one of them breaks it down to three or four basic components and the combination of this will give all others.
The most popular one is RGB, mainly because this is also how our eye builds up colors in our eyes. Its base colors are red, green and blue.
To code the transparency of a color sometimes a fourth element: alpha (A) is added.
By Dr. Xinwen Fu

15. Color Systems
RGB is the most common as our eyes use something similar, used by our display systems.
The HSV and HLS decompose colors into their hue, saturation and value/luminance components, which is a more natural way for us to describe colors.
You may dismiss last component, making your algorithm less sensible to light conditions of the input image.
YCrCb is used by the popular JPEG image format.
CIE L*a*b* is a perceptually uniform color space, which comes handy if you need to measure the distance of a given color to another color.
By Dr. Xinwen Fu

16. Data Types for Color
Each of building components has their own valid domains. This leads to the data type used.
The smallest data type possible is char, which means one byte or 8 bits
This may be unsigned (so can store values from 0 to 255) or signed (values from -127 to +127).
In case of three components this gives 16 million possible colors to represent (like in case of RGB)
Even finer control by using float (4 byte = 32 bit) or double (8 byte = 64 bit) data types for each component.
However, increasing the size of a component also increases the size of the whole picture in the memory.
By Dr. Xinwen Fu

17. Creating explicitly a Mat object
Although Mat is a great class as image container it is also a general matrix class.
Therefore, it is possible to create and manipulate multidimensional matrices.
You can create a Mat object in multiple ways
For two dimensional and multichannel images we first define their size: row and column count wise.
By Dr. Xinwen Fu

18. The Scalar is four element short vector.
We need to specify the data type to use for storing the elements and the number of channels per matrix point. To do this we have multiple definitions made according to the following convention:
CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number]
CV_8UC3 means we use unsigned char types that are 8 bit long and each pixel has three items of this to form the three channels. This are predefined for up to four channel numbers.
The Scalar is four element short vector.
Only 2 dimension matrix can use cout
class Scalar_
Template class for a 4-element vector derived from Vec.
template<typename _Tp> class Scalar_ : public Vec<_Tp, 4> { ... };
typedef Scalar_<double> Scalar;
Being derived from Vec<_Tp, 4> , Scalar_ and Scalar can be used just as typical 4-element vectors. In addition, they can be converted to/from CvScalar . The type Scalar is widely used in OpenCV to pass pixel values.
By Dr. Xinwen Fu

19. By Dr. Xinwen Fu

20. Data Type
A primitive OpenCV data type is one of unsigned char, bool, signed char, unsigned short, signed short, int, float, double, or a tuple of values of one of these types, where all the values in the tuple have the same type.
Any primitive type from the list can be defined by an identifier in the form
CV_<bit-depth>{U|S|F}C(<number_of_channels>), for example: uchar ~ CV_8UC1, 3-element floating-point tuple ~ CV_32FC3
A universal OpenCV structure that is able to store a single instance of such a primitive data type is Vec. Multiple instances of such a type can be stored in a std::vector, Mat, Mat_, SparseMat, SparseMat_, or any other container that is able to store Vec instances.
By Dr. Xinwen Fu

21. By Dr. Xinwen Fu

22. Print for other common items
OpenCV offers support for print of other common OpenCV data structures too via the << operator like
2D Point
3D Point
std::vector via cv::Mat
By Dr. Xinwen Fu

23. Print (Cont’d)
std::vector of points
By Dr. Xinwen Fu

24. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

25. Goal How to go through each and every pixel of an image?
How is OpenCV matrix values stored?
How to measure the performance of our algorithm?
What are lookup tables and why use them?
By Dr. Xinwen Fu

26. Color space reduction
Divide the color space current value with a new input value to end up with fewer colors
For instance every value between zero and nine takes the new value zero, every value between ten and nineteen the value ten and so on.
By Dr. Xinwen Fu

27. How the image matrix is stored in the memory? - gray scale image
The size of the matrix depends of the color system used. More accurately, it depends from the number of channels used.
By Dr. Xinwen Fu

28. How the image matrix is stored in the memory? - RGB color system
For multichannel images the columns contain as many sub columns as the number of channels
Note that the order of the channels is inverse: BGR instead of RGB
By Dr. Xinwen Fu

29. Color reduction formula
When you divide an uchar (unsigned char - aka values between zero and 255) value with an int value the result will be also char. These values may only be char values.
Therefore, any fraction will be rounded down.
Taking advantage of this fact the upper operation in the uchar domain may be expressed as:
By Dr. Xinwen Fu

30. Measure time code runs Another issue is how do we measure time?
OpenCV offers two simple functions to achieve this getTickCount() and getTickFrequency().
The first returns the number of ticks of your systems CPU from a certain event (like since you booted your system).
The second returns how many times your CPU emits a tick during a second. So to measure in seconds the number of time elapsed between two operations is easy as:
By Dr. Xinwen Fu

31. Lookup table for color reduction
how_to_scan_images imageName.jpg intValueToReduce [G]
The final argument is optional. If given the image will be loaded in gray scale format, otherwise the RGB color way is used. The first thing is to calculate the lookup table.
By Dr. Xinwen Fu

32. The Efficient Way
By Dr. Xinwen Fu

33. Iterator
By Dr. Xinwen Fu

34. On-the-fly address calc
In case of color images we have three uchar items per column.
This may be considered a short vector of uchar items, that has been baptized in OpenCV with the Vec3b name.
By Dr. Xinwen Fu

35. On-the-fly address calc
By Dr. Xinwen Fu

36. The Core Function
By Dr. Xinwen Fu

37. Performance Difference
For the best result compile the program and run it on your own speed. For showing off better the differences I’ve used a quite large (2560 X 1600) image.
The performance presented here are for color images. For a more accurate value I’ve averaged the value I got from the call of the function for hundred times.
By Dr. Xinwen Fu

38. Mat_
If you need multiple lookups using this method for an image it may be troublesome and time consuming to enter the type and the at keyword for each of the accesses. To solve this problem OpenCV has a Mat_ data type.
It’s the same as Mat with the extra need that at definition you need to specify the data type through what to look at the data matrix, however in return you can use the operator() for fast access of items.
To make things even better this is easily convertible from and to the usual Mat data type. A sample usage of this you can see in case of the color images of the upper function. Nevertheless, it’s important to note that the same operation (with the same runtime speed) could have been done with the at() function. It’s just a less to write for the lazy programmer trick.
By Dr. Xinwen Fu

39. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

40. Mask operation
Recalculate each pixels value in an image according to a mask matrix (also known as kernel).
This mask holds values that will adjust how much influence neighboring pixels (and the current pixel) have on the new pixel value.
From a mathematical point of view we make a weighted average, with our specified values.
By Dr. Xinwen Fu

41. Basic Method void Sharpen(const Mat& myImage,Mat& Result) {
CV_Assert(myImage.depth() == CV_8U); // accept only uchar images
const int nChannels = myImage.channels();
Result.create(myImage.size(),myImage.type());
for(int j = 1 ; j < myImage.rows-1; ++j)
const uchar* previous = myImage.ptr<uchar>(j - 1);
const uchar* current = myImage.ptr<uchar>(j );
const uchar* next = myImage.ptr<uchar>(j + 1);
uchar* output = Result.ptr<uchar>(j);
By Dr. Xinwen Fu

42. for(int i= nChannels;i < nChannels*(myImage.cols-1); ++i) {
*output++ = saturate_cast<uchar>(5*current[i]
-current[i-nChannels] - current[i+nChannels] - previous[i] - next[i]); }
Result.row(0).setTo(Scalar(0));
Result.row(Result.rows-1).setTo(Scalar(0));
Result.col(0).setTo(Scalar(0));
Result.col(Result.cols-1).setTo(Scalar(0));
By Dr. Xinwen Fu

43. filter2D function
Applying such filters is so common in image processing that in OpenCV there exist a function that will take care of applying the mask (also called a kernel in some places).
define a Mat object that holds the mask:
Then call the filter2D function specifying the input, the output image and the kernell to use:
By Dr. Xinwen Fu

44. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

45. Theory
From our previous tutorial, we know already a bit of Pixel operators.
An interesting dyadic (two-input) operator is the linear blend operator:
By varying from 0 to 1 this operator can be used to perform a temporal cross-dissolve between two images or videos, as seen in slide shows and film productions
By Dr. Xinwen Fu

46. Example #include <opencv/cv.h> #include <opencv/highgui.h>
#include <iostream>
using namespace cv;
int main( int argc, char** argv ) {
double alpha = 0.5; double beta; double input;
Mat src1, src2, dst;
/// Ask the user enter alpha
std::cout<<" Simple Linear Blender "<<std::endl;
std::cout<<" "<<std::endl;
std::cout<<"* Enter alpha [0-1]: ";
std::cin>>input;
By Dr. Xinwen Fu

47. Example
/// We use the alpha provided by the user iff it is between 0 and 1
if( alpha >= 0 && alpha <= 1 ) { alpha = input; }
/// Read image ( same size, same type )
src1 = imread("../../images/LinuxLogo.jpg");
src2 = imread("../../images/WindowsLogo.jpg");
if( !src1.data ) { printf("Error loading src1 \n"); return -1; }
if( !src2.data ) { printf("Error loading src2 \n"); return -1; }
/// Create Windows
namedWindow("Linear Blend", 1);
beta = ( alpha );
addWeighted( src1, alpha, src2, beta, 0.0, dst);
imshow( "Linear Blend", dst );
waitKey(0);
return 0; }
By Dr. Xinwen Fu

48. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

49. Image Processing
A general image processing operator is a function that takes one or more input images and produces an output image.
Image transforms can be seen as:
Point operators (pixel transforms)
Neighborhood (area-based) operators
By Dr. Xinwen Fu

50. Pixel Transforms
In this kind of image processing transform, each output pixel’s value depends on only the corresponding input pixel value (plus, potentially, some globally collected information or parameters).
Examples of such operators include brightness and contrast adjustments as well as color correction and transformations.
By Dr. Xinwen Fu

51. Brightness and contrast adjustments
Two commonly used point processes are multiplication and addition with a constant:
g(x) = α f(x) + β
The parameters α > 0 and β are often called the gain and bias parameters
Sometimes these parameters are said to control contrast and brightness respectively.
You can think of f(x) as the source image pixels and g(x) as the output image pixels. Then, more conveniently we can write the expression as:
g(i; j) = α f(i; j) + β where i and j indicates that the pixel is located in the i-th row and j-th column.
By Dr. Xinwen Fu

52. Example #include <opencv/cv.h> #include <opencv/highgui.h>
#include <iostream>
using namespace cv;
double alpha; /**< Simple contrast control */
int beta; /**< Simple brightness control */
int main( int argc, char** argv ) {
/// Read image given by user
Mat image = imread( argv[1] );
Mat new_image = Mat::zeros( image.size(), image.type() );
/// Initialize values
std::cout<<" Basic Linear Transforms "<<std::endl;
std::cout<<" "<<std::endl;
std::cout<<"* Enter the alpha value [ ]: ";std::cin>>alpha;
std::cout<<"* Enter the beta value [0-100]: "; std::cin>>beta;
By Dr. Xinwen Fu

53. /// Do the operation new_image(i,j) = alpha*image(i,j) + beta
for( int y = 0; y < image.rows; y++ ) {
for( int x = 0; x < image.cols; x++ )
for( int c = 0; c < 3; c++ )
new_image.at<Vec3b>(y,x)[c] =
saturate_cast<uchar>( alpha*( image.at<Vec3b>(y,x)[c] ) + beta ); }
/// Create Windows
namedWindow("Original Image", 1);
namedWindow("New Image", 1);
/// Show stuff
imshow("Original Image", image);
imshow("New Image", new_image);
/// Wait until user press some key
waitKey();
return 0;
By Dr. Xinwen Fu

54. Who is Lena? Lena Söderberg, a Swedish model
cropped from the centerfold of November 1972 issue of Playboy magazine
By Dr. Xinwen Fu

55. Core function
Instead of using the for loops to access each pixel, we could have simply used this command: image.convertTo(new_image, -1, alpha, beta); where convertTo would effectively perform new_image = a*image + beta.
However, we wanted to show you how to access each pixel. In any case, both methods give the same result.
By Dr. Xinwen Fu

56. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

57. Example in Visual Studio 2010
Use Point to define 2D points in an image.
Use Scalar and why it is useful
Draw a line by using the OpenCV function line
Draw an ellipse by using the OpenCV function ellipse
Draw a rectangle by using the OpenCV function rectangle
Draw a circle by using the OpenCV function circle
Draw a filled polygon by using the OpenCV function fillPoly
By Dr. Xinwen Fu

58. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

59. Example in Visual Studio 2010
Use the Random Number generator class (RNG) and how to get a random number from a uniform distribution.
RNG rng( 0xFFFFFFFF );
rng.uniform(a,b); // This generates a randomly uniformed distribution between the values a and b (inclusive in a, exclusive in b).
Display text on an OpenCV window by using the function putText
By Dr. Xinwen Fu

60. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

61. Skipped What is a Fourier transform and why use it?
How to do it in OpenCV?
Usage of functions such as: copyMakeBorder(), merge(), dft(), getOptimalDFTSize(), log() and normalize() .
By Dr. Xinwen Fu

62. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

63. Skipped
How to print and read text entries to a file and OpenCV using YAML or XML files?
How to do the same for OpenCV data structures?
How to do this for your data structures?
Usage of OpenCV data structures such as FileStorage, FileNode or FileNodeIterator.
By Dr. Xinwen Fu

64. Outline 2.1 Mat - The Basic Image Container
2.2 How to scan images, lookup tables and time measurement with OpenCV
2.3 Mask operations on matrices
2.4 Adding (blending) two images using OpenCV
2.5 Changing the contrast and brightness of an image
2.6 Basic drawing
2.7 Random generator and text with OpenCV
2.8 Discrete Fourier Transform
2.9 File input and output using XML and YAML
2.10 Interoperability with OpenCV 1
By Dr. Xinwen Fu

65. Skipped
What changed with the version 2 of OpenCV in the way you use the library compared to its first version
How to add some Gaussian noise to an image
What are lookup tables and why use them?
By Dr. Xinwen Fu

66. References
OpenCV documentation
By Dr. Xinwen Fu