1. Introduction to Image processing using processing

2. 2 Image processing The analysis, manipulation, storage, and display of graphical images from sources such as photographs, drawings, and video.

3. 3 Three steps in image processing Image processing spans a sequence of three steps. –The input step (image capture and digitizing) converts the differences in coloring and shading in the picture into binary values that a computer can process. –The processing step can include image enhancement and data compression. –The output step consists of the display or printing of the processed image. Image processing is used in such applications as television and film, medicine, satellite weather mapping, machine vision, and computer based pattern recognition.

4. 4 The PIXEL array and LOCATION calculation We can get at the pixels of the display window, using the array “pixels". Each pixel on the screen has an X,Y coordinate However, the array "pixels" in a one-dimensional array We need a way to access one dimensional array with X, Y coordinate 1. Assume a window or image with a given WIDTH and HEIGHT. 2. We then know the pixel array has a total number of elements equaling WIDTH * HEIGHT. 3. For any given point in the windows (X,Y), the location in our 1 dimensional pixel array is: –LOCATION = X + Y*WIDTH

5. 5 loadPixels() –Loads the pixel data for the display window into the pixels[] array. This function must always be called before reading from or writing to pixels[]. updatePixels() –Updates the display window with the data in the pixels[] array. Use in conjunction with loadPixels().

6. 6 Example(Generate Random Image) size(50,50); loadPixels(); //nested loop to walk through every x and y coordinate //like with our 2D array for (int x = 0; x < width; x++) { for (int y = 0; y < height; y++) { //pick a random number, 0 to 255 int rand = int(random(255)); //create a grayscale color based on random number color c = color(rand,rand,rand); //do our location calculation int loc = x + y*width; //set pixel at that location to random color pixels[loc] = c; } updatePixels(); }

7. 7 Example(Show loaded image) PImage a; void setup() { size(200, 200); a = loadImage("jelly.jpg"); noLoop(); } void draw() { loadPixels(); background(0); for (int x = 0; x < a.width; x++) { for (int y = 0; y < a.height; y++ ) { //calculate the 1D location from a 2D grid int loc = x + y*a.width; //get the R,G,B values from image float r,g,b; r = red (a.pixels[loc]); g = green (a.pixels[loc]); b = blue (a.pixels[loc]); //****DO OUR IMAGE PROCESSING STUFF HERE*******/ //make a new color and set pixel in the window color c = color(r,g,b); pixels[loc] = c; } updatePixels(); }

8. 8 Example(Change brightness) PImage a; void setup() { size(200, 200); a = loadImage("jelly.jpg"); } void draw() { loadPixels(); background(255); for (int x = 0; x < a.width; x++) { for (int y = 0; y < a.height; y++ ) { //calculate the 1D location from a 2D grid int loc = x + y*a.width; //get the R,G,B values from image float r,g,b; r = red (a.pixels[loc]); g = green (a.pixels[loc]); b = blue (a.pixels[loc]); //do some sort of calculation to the RGB values (increase brightness according to the mouse here) r += ((float) mouseX / width) * 255; g += ((float) mouseX / width) * 255; b += ((float) mouseX / width) * 255; //constrain RGB to make sure they are within 0-255 color range r = constrain(r,0,255); g = constrain(g,0,255); b = constrain(b,0,255); //make a new color and set pixel in the window color c = color(r,g,b); pixels[loc] = c; } updatePixels(); }

9. 9 Example(Thresholding) PImage a; void setup() { size(200, 200); a = loadImage("jelly.jpg"); } void draw() { loadPixels(); background(255); for (int x = 0; x < a.width; x++) { for (int y = 0; y < a.height; y++ ) { //calculate the 1D location from a 2D grid int loc = x + y*a.width; //get the R,G,B values from image float r,g,b; r = red (a.pixels[loc]); g = green (a.pixels[loc]); b = blue (a.pixels[loc]); //calculate a threshold from 0-255 based on mouseX int threshold = int(((float) mouseX / width) * 255); color c; //do a threshold test based on brightness if (brightness(a.pixels[loc]) > threshold) { c = color(255,255,255); } else { c = color(0,0,0); } pixels[loc] = c; // color c = color(r,g,b); // pixels[loc] = c; } updatePixels(); }

10. 10 Spatial convolution This process uses a weighted average of an input pixel and its neighbors to calculate an output pixel value. We can use neighboring groups of different sizes, such as 3x3, 5x5, etc. Different combinations of different weights for each pixel can result in different effects Sharpen: -1 -1 -1 -1 9 -1 -1 -1 -1 Blur: 1/9 1/9 1/9 new_pixelvalue(i,j) = old_pixelvalue(i-1, j-1)*(-1) + old_pixelvalue(i-1, j)*(-1) + old_pixelvalue(i-1, j+1)*(-1) + old_pixelvalue(i, j-1)*(-1) + old_pixelvalue(i, j)*(9) + old_pixelvalue(i, j+1)*(-1) + old_pixelvalue(i+1, j-1)*(-1) + old_pixelvalue(i+1, j)*(-1) + old_pixelvalue(i+1, j+1)*(-1)

11. 11 Example(Sharpening) color Convolve(int x, int y, float[] Matrix, PImage a) { float sumR = 0.0; float sumG = 0.0; float sumB = 0.0; //is our matrix 3x3, 5x5, etc.? int MatrixSize = int(sqrt(Matrix.length)); //to keep track of where we are in convolution matrix int countMatrix = 0; for (int i =-MatrixSize/2; i <= MatrixSize/2; i++){ for (int j=-MatrixSize/2; j <= MatrixSize/2; j++){ //what pixel are we testing int xloc = x+i; int yloc = y+j; int loc = xloc + a.width*(yloc+j); //make sure we haven not walked off our image loc = constrain(loc,0,a.pixels.length-1); //calculate the convolution sumB += (blue(a.pixels[loc]) * Matrix[countMatrix]); sumG += (green(a.pixels[loc]) * Matrix[countMatrix]); sumR += (red(a.pixels[loc]) * Matrix[countMatrix]); countMatrix++; } //make sure RGB is withing range sumR = constrain(sumR,0,255); sumG = constrain(sumG,0,255); sumB = constrain(sumB,0,255); //return the resulting color return color(sumR,sumG,sumB); } PImage a; void setup() { size(400, 400); a = loadImage("penguin.jpg"); } void draw() { //we're only going to process a portion of the image so let's set the whole image as the background first image(a,0,0); loadPixels(); //For drawing a black rectangle defining area for processing stroke(0); noFill(); rectMode(CENTER_DIAMETER); rect(mouseX,mouseY,81,81); //what are the boundaries for the pixels we are processing int X1 = constrain(mouseX-40,0,a.width); int Y1 = constrain(mouseY-40,0,a.height); int X2 = constrain(mouseX+40,0,a.width); int Y2 = constrain(mouseY+40,0,a.height); //begin our loop for every pixel for (int x = X1; x < X2; x++) { for (int y = Y1; y < Y2; y++ ) { float[] Matrix= { -1,-1,-1, -1, 9,-1, -1,-1,-1 }; color c = Convolve(x,y,Matrix,a); int loc = x + y*a.width; pixels[loc] = c; } updatePixels(); }

12. 12 Example (blurring) color Convolve(int x, int y, float[] Matrix, PImage a) { float sumR = 0.0; float sumG = 0.0; float sumB = 0.0; //is our matrix 3x3, 5x5, etc.? int MatrixSize = int(sqrt(Matrix.length)); //to keep track of where we are in convolution matrix int countMatrix = 0; for (int i =-MatrixSize/2; i <= MatrixSize/2; i++){ for (int j=-MatrixSize/2; j <= MatrixSize/2; j++){ //what pixel are we testing int xloc = x+i; int yloc = y+j; int loc = xloc + a.width*(yloc+j); //make sure we haven not walked off our image loc = constrain(loc,0,a.pixels.length-1); //calculate the convolution sumB += (blue(a.pixels[loc]) * Matrix[countMatrix]); sumG += (green(a.pixels[loc]) * Matrix[countMatrix]); sumR += (red(a.pixels[loc]) * Matrix[countMatrix]); countMatrix++; } //make sure RGB is withing range sumR = constrain(sumR,0,255); sumG = constrain(sumG,0,255); sumB = constrain(sumB,0,255); //return the resulting color return color(sumR,sumG,sumB); } PImage a; void setup() { size(400, 400); a = loadImage("penguin.jpg"); } void draw() { //we're only going to process a portion of the image so let's set the whole image as the background first image(a,0,0); loadPixels(); //For drawing a black rectangle defining area for processing stroke(0); noFill(); rectMode(CENTER_DIAMETER); rect(mouseX,mouseY,81,81); //what are the boundaries for the pixels we are processing int X1 = constrain(mouseX-40,0,a.width); int Y1 = constrain(mouseY-40,0,a.height); int X2 = constrain(mouseX+40,0,a.width); int Y2 = constrain(mouseY+40,0,a.height); //begin our loop for every pixel for (int x = X1; x < X2; x++) { for (int y = Y1; y < Y2; y++ ) { float[] Matrix = new float[25]; for (int i = 0; i < 25; i++) { Matrix[i] = 1.0f/25.0f;; } color c = Convolve(x,y,Matrix,a); int loc = x + y*a.width; pixels[loc] = c; } updatePixels(); }

13. 13 Example (edge detection) color Convolve(int x, int y, float[] Matrix, PImage a) { float sumR = 0.0; float sumG = 0.0; float sumB = 0.0; //is our matrix 3x3, 5x5, etc.? int MatrixSize = int(sqrt(Matrix.length)); //to keep track of where we are in convolution matrix int countMatrix = 0; for (int i =-MatrixSize/2; i <= MatrixSize/2; i++){ for (int j=-MatrixSize/2; j <= MatrixSize/2; j++){ //what pixel are we testing int xloc = x+i; int yloc = y+j; int loc = xloc + a.width*(yloc+j); //make sure we haven not walked off our image loc = constrain(loc,0,a.pixels.length-1); //calculate the convolution sumB += (blue(a.pixels[loc]) * Matrix[countMatrix]); sumG += (green(a.pixels[loc]) * Matrix[countMatrix]); sumR += (red(a.pixels[loc]) * Matrix[countMatrix]); countMatrix++; } //make sure RGB is withing range sumR = constrain(sumR,0,255); sumG = constrain(sumG,0,255); sumB = constrain(sumB,0,255); //return the resulting color return color(sumR,sumG,sumB); } PImage a; void setup() { size(400, 400); a = loadImage("penguin.jpg"); } void draw() { //we're only going to process a portion of the image so let's set the whole image as the background first image(a,0,0); loadPixels(); //For drawing a black rectangle defining area for processing stroke(0); noFill(); rectMode(CENTER_DIAMETER); rect(mouseX,mouseY,81,81); //what are the boundaries for the pixels we are processing int X1 = constrain(mouseX-40,0,a.width); int Y1 = constrain(mouseY-40,0,a.height); int X2 = constrain(mouseX+40,0,a.width); int Y2 = constrain(mouseY+40,0,a.height); //begin our loop for every pixel for (int x = X1; x < X2; x++) { for (int y = Y1; y < Y2; y++ ) { float[] Matrix = {-1, 0, 1, -2, 0, 2, -1, 0, 1}; float[] Matrix1= {1, 2, 1, 0, 0, 0, -1, -2, -1}; color c = Convolve(x,y,Matrix,a); color c2 = Convolve(x, y, Matrix1, a); int loc = x + y*a.width; // pixels[loc] = constrain(c+c2, 0, 255); pixels[loc]=color(brightness(c2+c)); color c = Convolve(x,y,Matrix,a); int loc = x + y*a.width; pixels[loc] = c; } updatePixels(); }

14. 14 Assignment Implement a program which do sharpening, blurring, edge detection on an image. Switch among sharpening, blurring, edge detection mode on mouse button click

15. 15 Video capture import processing.video.*; Capture cam; void setup() { size(640, 480); // If no device is specified, will just use the default. cam = new Capture(this, 640, 480); } void draw() { if (cam.available() == true) { cam.read(); image(cam, 0, 0); } To run this program we need install Quicktime 7.0 and WinVDIG

16. 16 Play video import processing.video.*; Movie myMovie; void setup() { size(640, 480, P2D); background(0); // Load and play the video in a loop myMovie = new Movie(this, "station.mov"); myMovie.loop(); } void movieEvent(Movie myMovie) { myMovie.read(); } void draw() { tint(255, 20); image(myMovie, mouseX- myMovie.width/2, mouseY- myMovie.height/2); filter(GRAY); }

17. 17 tint() Sets the fill value for displaying images. Images can be tinted to specified colors or made transparent by setting the alpha. To make an image transparent, but not change it's color, use white as the tint color and specify an alpha value. For instance, tint(255, 128) will make an image 50% transparent

18. 18 filter() Filters the display window as defined by one of the following modes: THRESHOLD - converts the image to black and white pixels depending if they are above or below the threshold defined by the level parameter. GRAY - converts any colors in the image to grayscale equivalents INVERT - sets each pixel to its inverse value POSTERIZE - limits each channel of the image to the number of colors specified as the level parameter BLUR - executes a Guassian blur with the level parameter specifying the extent of the blurring. OPAQUE - sets the alpha channel to entirely opaque. ERODE - reduces the light areas with the amount defined by the level parameter. DILATE - increases the light areas with the amount defined by the level parameter

19. 19 Assignment Implement a program which various filtering on an video. Switch among THRESHOLD, GRAY, INVERT, POSTERIZE, BLUR, OPAQUE, ERODE, DILATE filtering mode on mouse button click