1. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 1 Example I: Predicting the Weather Let us study an interesting neural network application. Its purpose is to predict the local weather based on a set of current weather data: temperature (degrees Celsius) temperature (degrees Celsius) atmospheric pressure (inches of mercury) atmospheric pressure (inches of mercury) relative humidity (percentage of saturation) relative humidity (percentage of saturation) wind speed (kilometers per hour) wind speed (kilometers per hour) wind direction (N, NE, E, SE, S, SW, W, or NW) wind direction (N, NE, E, SE, S, SW, W, or NW) cloud cover (0 = clear … 9 = total overcast) cloud cover (0 = clear … 9 = total overcast) weather condition (rain, hail, thunderstorm, …) weather condition (rain, hail, thunderstorm, …)

2. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 2 Example I: Predicting the Weather We assume that we have access to the same data from several surrounding weather stations. There are eight such stations that surround our position in the following way: 100 km

3. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 3 Example I: Predicting the Weather How should we format the input patterns? We need to represent the current weather conditions by an input vector whose elements range in magnitude between zero and one. When we inspect the raw data, we find that there are two types of data that we have to account for: Scaled, continuously variable values Scaled, continuously variable values n-ary representations of category values n-ary representations of category values

4. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 4 Example I: Predicting the Weather The following data can be scaled: temperature (-10… 40 degrees Celsius) temperature (-10… 40 degrees Celsius) atmospheric pressure (26… 34 inches of mercury) atmospheric pressure (26… 34 inches of mercury) relative humidity (0… 100 percent) relative humidity (0… 100 percent) wind speed (0… 250 km/h) wind speed (0… 250 km/h) cloud cover (0… 9) cloud cover (0… 9) We can just scale each of these values so that its lower limit is mapped to some  and its upper value is mapped to (1 -  ). These numbers will be the components of the input vector.

5. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 5 Example I: Predicting the Weather Usually, wind speeds vary between 0 and 40 km/h. By scaling wind speed between 0 and 250 km/h, we can account for all possible wind speeds, but usually only make use of a small fraction of the scale. Therefore, only the most extreme wind speeds will exert a substantial effect on the weather prediction. Consequently, we will use two scaled input values: wind speed ranging from 0 to 40 km/h wind speed ranging from 0 to 40 km/h wind speed ranging from 40 to 250 km/h wind speed ranging from 40 to 250 km/h

6. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 6 Example I: Predicting the Weather How about the non-scalable weather data? Wind direction is represented by an eight- component vector, where only one element (or possibly two adjacent ones) is active, indicating one out of eight wind directions. Wind direction is represented by an eight- component vector, where only one element (or possibly two adjacent ones) is active, indicating one out of eight wind directions. The subjective weather condition is represented by a nine-component vector with at least one, and possibly more, active elements. The subjective weather condition is represented by a nine-component vector with at least one, and possibly more, active elements. With this scheme, we can encode the current conditions at a given weather station with 23 vector components: one for each of the four scaled parameters one for each of the four scaled parameters two for wind speed two for wind speed eight for wind direction eight for wind direction nine for the subjective weather condition nine for the subjective weather condition

7. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 7 Example I: Predicting the Weather Since the input does not only include our station, but also the eight surrounding ones, the input layer of the network looks like this: … our station … north … … northwest The network has 207 input neurons, which accept 207-component input vectors.

8. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 8 Example I: Predicting the Weather What should the output patterns look like? We want the network to produce a set of indicators that we can interpret as a prediction of the weather in 24 hours from now. In analogy to the weather forecast on the evening news, we decide to demand the following four indicators: a temperature prediction a temperature prediction a prediction of the chance of precipitation occurring a prediction of the chance of precipitation occurring an indication of the expected cloud cover an indication of the expected cloud cover a storm indicator (extreme conditions warning) a storm indicator (extreme conditions warning)

9. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 9 Example I: Predicting the Weather Each of these four indicators can be represented by one scaled output value: temperature (-10… 40 degrees Celsius) temperature (-10… 40 degrees Celsius) chance of precipitation (0%… 100%) chance of precipitation (0%… 100%) cloud cover (0… 9) cloud cover (0… 9) storm warning: two possibilities: storm warning: two possibilities: –0: no storm warning; 1: storm warning –probability of serious storm (0%… 100%) Of course, the actual network outputs range from  to (1 -  ), and after their computation, if necessary, they are scaled to match the ranges specified above.

10. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 10 Example I: Predicting the Weather We decide (or experimentally determine) to use a hidden layer with 42 sigmoidal neurons. In summary, our network has 207 input neurons 207 input neurons 42 hidden neurons 42 hidden neurons 4 output neurons 4 output neurons Because of the small output vectors, 42 hidden units may suffice for this application.

11. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 11 Example I: Predicting the Weather The next thing we need to do is collecting the training exemplars. First we have to specify what our network is supposed to do: In production mode, the network is fed with the current weather conditions, and its output will be interpreted as the weather forecast for tomorrow. Therefore, in training mode, we have to present the network with exemplars that associate known past weather conditions at a time t with the conditions at t – 24 hrs. So we have to collect a set of historical exemplars with known correct output for every input.

12. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 12 Example I: Predicting the Weather Obviously, if such data is unavailable, we have to start collecting them. The selection of exemplars that we need depends, among other factors, on the amount of changes in weather at our location. For example, in Honolulu, Hawaii, our exemplars may not have to cover all seasons, because there is little variation in the weather. In Boston, however, we would need to include data from every calendar month because of dramatic changes in weather across seasons. As we know, some winters in Boston are much harder than others, so it might be a good idea to collect data for several years.

13. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 13 Example I: Predicting the Weather And how about the granularity of our exemplar data, i.e., the frequency of measurement? Using one sample per day would be a natural choice, but it would neglect rapid changes in weather. If we use hourly instantaneous samples, however, we increase the likelihood of conflicts. Therefore, we decide to do the following: We will collect input data every hour, but the corresponding output pattern will be the average of the instantaneous patterns over a 12-hour period. This way we reduce the possibility of errors while increasing the amount of training data.

14. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 14 Example I: Predicting the Weather Now we have to train our network. If we use samples in one-hour intervals for one year, we have 8,760 exemplars. Our network has 207  42 + 42  4 = 8862 weights, which means that data from ten years, i.e., 87,600 exemplars would be desirable. Rule of thumb: There should be at least 5 to 10 times as many training exemplars as there are weights in the network.

15. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 15 Example I: Predicting the Weather Since with a large number of samples the hold-one- out training method is very time consuming, we decide to use partial-set training instead. The best way to do this would be to acquire a test set (control set), that is, another set of input-output pairs measured on random days and at random times. After training the network with the 87,600 exemplars, we could then use the test set to evaluate the performance of our network.

16. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 16 Example I: Predicting the Weather Neural network troubleshooting: Plot the global error as a function of the training epoch. The error should decrease after every epoch. If it oscillates, do the following tests.Plot the global error as a function of the training epoch. The error should decrease after every epoch. If it oscillates, do the following tests. Try reducing the size of the training set. If then the network converges, a conflict may exist in the exemplars.Try reducing the size of the training set. If then the network converges, a conflict may exist in the exemplars. If the network still does not converge, continue pruning the training set until it does converge. Then add exemplars back gradually, thereby detecting the ones that cause conflicts.If the network still does not converge, continue pruning the training set until it does converge. Then add exemplars back gradually, thereby detecting the ones that cause conflicts.

17. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 17 Example I: Predicting the Weather If this still does not work, look for saturated neurons (extreme weights) in the hidden layer. If you find those, add more hidden-layer neurons, possibly an extra 20%.If this still does not work, look for saturated neurons (extreme weights) in the hidden layer. If you find those, add more hidden-layer neurons, possibly an extra 20%. If there are no saturated units and the problems still exist, try lowering the learning parameter  and training longer.If there are no saturated units and the problems still exist, try lowering the learning parameter  and training longer. If the network converges but does not accurately learn the desired function, evaluate the coverage of the training set.If the network converges but does not accurately learn the desired function, evaluate the coverage of the training set. If the coverage is adequate and the network still does not learn the function precisely, you could refine the pattern representation. For example, you could include a season indicator to the input, helping the network to discriminate between similar inputs that produce very different outputs.If the coverage is adequate and the network still does not learn the function precisely, you could refine the pattern representation. For example, you could include a season indicator to the input, helping the network to discriminate between similar inputs that produce very different outputs. Then you can start predicting the weather!

18. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 18 Further Examples Online TensorFlow Neural Network Playground: http://bit.ly/1SM2VVh The OCHRE demo applet for optical character recognition: http://www.sund.de/netze/applets/BPN/bpn2/ochre.html … and if you are interested in Deep Learning: ConvNetJS: http://cs.stanford.edu/people/karpathy/convnetjs/

19. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 19 Computer Vision

20. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 20 Computer Vision A simple two-stage model of computer vision: Image processing Scene analysis Bitmap image Scene description feedback (tuning) Prepare image for scene analysis Build an iconic model of the world

21. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 21 Computer Vision The image processing stage prepares the input image for the subsequent scene analysis. Usually, image processing results in one or more new images that contain specific information on relevant features of the input image. The information in the output images is arranged in the same way as in the input image. For example, in the upper left corner in the output images we find information about the upper left corner in the input image.

22. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 22 Computer Vision The scene analysis stage interprets the results from the image processing stage. Its output completely depends on the problem that the computer vision system is supposed to solve. For example, it could be the number of bacteria in a microscopic image, or the identity of a person whose retinal scan was input to the system. In the following lectures we will focus on the lower- level, i.e., image processing techniques. Later we will discuss a variety of scene analysis methods and algorithms.

23. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 23 Computer Vision How can we turn a visual scene into something that can be algorithmically processed? Usually, we map the visual scene onto a two-dimensional array of intensities. In the first step, we have to project the scene onto a plane. This projection can be most easily understood by imagining a transparent plane between the observer (camera) and the visual scene. The intensities from the scene are projected onto the plane by moving them along a straight line from their initial position to the observer. The result will be a two-dimensional projection of the three- dimensional scene as it is seen by the observer.

24. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 24 Camera Geometry

25. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 25 Color Imaging via Bayer Filter

26. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 26 Magnetic Resonance Imaging

27. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 27 Digitizing Visual Scenes In this course, we will mostly restrict our concept of images to grayscale. Grayscale values usually have a resolution of 8 bits (256 different values), in medical applications sometimes 12 bits (4096 values), or in binary images only 1 bit (2 values). We simply choose the available gray level whose intensity is closest to the gray value color we want to convert.

28. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 28 Digitizing Visual Scenes With regard to spatial resolution, we will map the intensity in our image onto a two-dimensional finite array: [0, 0][0, 1][0, 2][0, 3] [1, 0][1, 1][1, 2][1, 3] [2, 0][2, 1][2, 2][2, 3] y’ x’

29. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 29 Digitizing Visual Scenes So the result of our digitization is a two-dimensional array of discrete intensity values. Notice that in such a digitized image F[i, j] the first coordinate i indicates the row of a pixel, starting with 0, the second coordinate j indicates the column of a pixel, starting with 0. In an m×n pixel array, the relationship between image (with origin in ints center) and pixel coordinates is given by the equations

30. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 30 Image Size and Resolution

31. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 31 Intensity Resolution

32. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 32 Intensity Transformation Sometimes we need to transform the intensities of all image pixels to prepare the image for better visibility of information or for algorithmic processing.

33. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 33 Gamma Transformation Gammatransformation:

34. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 34 Gamma Transformation

35. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 35 Linear Histogram Scaling

36. April 14, 2016Introduction to Artificial Intelligence Lecture 20: Image Processing 36 Linear Histogram Scaling For a desired intensity range [a, b] we can use the following linear transformation: Note that outliers (individual pixels of very low or high intensity) should be disregarded when computing I min and I max.