1. Inventory Management Using Computer Vision Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 1

2. Inventory Management Inventory management is a science primarily about specifying the shape and percentage of stocked goods. It is required at different locations within a facility or within many locations of a supply network to precede the regular and planned course of production and stock of materials. Inventory Monitoring involves counting of certain objects and maintaining a database for the stock. Traditionally, this work has been carried out manually and used to be very tedious and time consuming. With the advent of modern age, different successful attempts have been made to automate this process. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 2

3. Work already done in the field of object counting Traditionally, number of objects in a stack of identical goods were counted by dividing the weight of the total stack (as measured on a weighing machine) by that of a single object. Feature extraction algorithms based on Image Processing techniques like Segmentation, Filtering, etc. have been employed. Object detection and counting has also been facilitated using many Supervised Machine Learning techniques. Color Histogram based segmentation using RGB and HYC color spaces has been implemented for this task.

4. Our Approach Image of the inventory taken as Input to the model Fuzzy histogram of the object as well as image is calculated. Histogram Intersection in the whole Inventory scene is computed. The region of interest is extracted by putting a suitable threshold. Sum of Squared Differences are calculated in the region of interest. The points of local minima are searched and a Size filter is applied to reject false alarms. Dip value is calculated at all the interest points and ratio of Dip and SSD is computed The points of interest are filtered out using dip filter. The filtered points of interest are counted and object is marked on image. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 4

5. Stepwise Implementation Regions with Histogram intersection less than 0.6 times maximum intersection are rejected Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 5

6. Result after filtering using histogram intersections Result after SSD calculation Black dots denote the points of minima amongst the Sum of Squared Differences computation Calculation of Sum of Squared Differences Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 6

7. Result after SSD calculation Result after applying Size filter Noisy black dots are reduced and suppressed by Size filter Size filter application Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 7

8. Dip filter application No. of black dots are further reduced by using the dip filter Result after applying the Size filter Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 8 Result after applying the Dip filter

9. Color Histogram What is a color histogram? A color histogram is a representation of the distribution of colors in an image. For digital images, a color histogram represents the number of pixels that have colors in each of the fixed list of color ranges that span the image's color space. The idea is to compute the color histograms of different segments of the input image and find their respective intersections with the color histogram of the prototype (the object to be identified). Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 9

10. Histogram Intersection Two histograms h1 and h2 are compared using a statistical technique called intersection. Histogram intersection is defined as: Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 10

11. Histogram Intersection Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Input Image(Where the object has to be detected) The prototype’s Image is made to move across the input image pixel by pixel, and the color histograms of the two images are made to intersect. 11

12. Histogram Intersection Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Input Image(Where the object has to be detected) The prototype’s Image is made to move across the input image pixel by pixel, and the color histograms of the two images are made to intersect. 12

13. Histogram Intersection Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Input Image(Where the object has to be detected) The prototype’s Image is made to move across the input image pixel by pixel, and the color histograms of the two images are made to intersect. 13

14. A suitable threshold was then chosen and object counting was done only in the region where histogram intersection was greater than the threshold. To choose the threshold, experiments were conducted with different test cases, by taking intersection threshold values ranging from 0.3 to 0.9 times the maximum value of intersection recorded in the image. It is evident that as we increase the threshold, the number of objects remain unchanged till a particular value. Beyond this threshold, the count obtained begins to fall. This particular value is chosen to be the threshold. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram Intersection 14

15. Showing regions with Histogram Intersection greater than 0.4*maximum intersection Showing regions with Histogram Intersection greater than 0.5*maximum intersection Test Image Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Prototype 15

16. Showing regions with Histogram Intersection greater than 0.6*maximum intersection Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram Intersection Showing regions with Histogram Intersection greater than 0.8*maximum intersection Region containing the objects It can be seen that the regions obtained when low threshold is chosen, also enclose the regiolns that are obtained when high intersection is chosen. Our aim is to find the highest value of histogram intersection threshold which can allow all the objects and also not be so high that it starts rejecting true values. 16

17. Subsequently, for a given test case in the plot of the no. of objects detected vs histogram intersection threshold, the first break point is the highest intersection value that can allow all the objects to be selected. Amongst the break points for all the test cases, the minimum break point should be chosen as the threshold. We took many test cases for this purpose. Some of those results are shown below and in next slide. o Plot shows the result for Test case no.1 o The point labeled on the plot is the break points of the curve. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram intersection threshold factor Histogram Intersection No. of objects detected 17

18. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved From all the test cases seen, 0.6 was the least of all the break points. So, the threshold value was chosen as 0.6 times the maximum intersection in that image. Histogram intersection threshold factor No. of objects detected Histogram intersection threshold factor No. of objects detected Histogram intersection threshold factor No. of objects detected Histogram intersection threshold factor No. of objects detected 18

19. The histogram intersections, as described, were calculated. However, this approach was later found to lack robustness towards changes in illumination over the objects to be detected. Image of object under normal conditions Image of object under dim light conditions Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram Intersection 19

20. Using the proposed method of histogram intersection, the algorithm would have rejected the objects in a different illumination environment. So, some suitable modifications were made. HISTOGRAM BIN STRUCTURES USED EARLIER: The RGB color space was divided into different regions called bins by the following fashion. 0 31 63 95 127 159 191 223 255 Red color space shown divided. P1 P2 P3 P4 P5 P6 P7 P8 The three color spaces were divided in a similar fashion. The various combinations of the color space parts gave rise to the bins of the RGB space. P i s represent the parts of the red channel Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram Bin 20

21. To increase the robustness, the sizes of the bins were increased. This is because the colors represented as shown below by dots (though being closely spaced) can be separated during bin formation. For this purpose, instead of 8, only 4 bins are used. P1 P2 P3 P4 P5 P6 P7 P8 P1 P2 P3 P4 0 31 63 95 127 159 191 223 255 0 63 127 191 255 Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Histogram Bin 21

22. Fuzzy Histogram However, some problem still existed which could not be resolved. In some cases (as shown below), though colors were close to each other, they were still being separated in two separate bins. Such a problem would always occur if we use techniques such as bin formation. To resolve this issue, the idea of fuzzy logic was implemented to form fuzzy bin structures as shown below. P1 P2 P3 P4 0 63 127 191 255 0 31 63 95 127 159 191 223 255 P4 P3 P2 P1 Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 22

23. Fuzzy Histogram P4 P3 P2 P1 0 31 63 95 127 159 191 223 255 Consider a color shown by the blue dot in the figure below. Unlike the previous method, the dot contributes both to P1 and P2 and the contribution is given by the value corresponding to the projections over the straight lines above it. The benefit was the increase in robustness and accuracy in object detection. The black dots represent contributions to the two bins. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 23

24. 0 63 127 191 255 10060 18050 The four squares represent four pixels These values are the red channel values of the four pixels. The illustration shows bin allotment to the pixel colors. Formation of a normal histogram P1 P2 P3 P4 Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 24

25. 0 63 127 191 255 Formation of a fuzzy histogram P1 P2 P3 P4 Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 25

26. 0 63 127 191 255 Formation of a fuzzy histogram P1 P2 P3 P4 Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved The mechanism shows the fractional contribution of pixel colors to different bins based on fuzzy logic. 26

27. Now, in the region selected after histogram intersection, sum of squared differences between the pixel values of the prototype and the input image are computed successively, as the prototype is allowed to move over the selected input image pixel by pixel. The two images (the window of the input image over which prototype moves, and the prototype) are a grid of pixels. For finding the SSD, we take corresponding pixels in both of them and then find the squared Euclidean distance between them. Then all these values are added up. The pixels shown in the same color correspond to each other Sum of Squared differences Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 27

28. The points where the calculated SSDs attains local minima have high probability of object presence. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 28 Sum of Squared differences

29. Fig. 3D Plot of the result after applying the mean squared error technique The deep valleys in the plot represent the regions where the objects are present. Apart from these deep valleys, there are also some other regions of local minima which need to be rejected. Minima that needs to be rejected Deep Valley representing the object Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 29 Sum of Squared differences

30. Filters We can see that by now we have got points of minima at places where the object is present. But along with them there are also points of minima that don’t correspond to actual object but are simply false alarms. For rejecting the minima points that correspond to false alarms, we have used two filters. Size filter Dip filter Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 30

31. Size Filter Image showing the minima points. The big dot corresponds to a deep valley. Consider a part of the input image enclosed by the prototype as it moves across it, as shown below. Our method has detected four minima points in the region. The minima point associated with the deepest valley is the correct point. From the knowledge of the dimensions of the prototype, we can predict that in the region shown, only one minima point is possible. So we reject all the points other than the one with the deepest minima that are enclosed within this region Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 31

32. Size Filter Objects identified before size filter Objects identified after size filter Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Accuracy increases after sizefilter 32

33. Dip Filter We defined a variable dip to further filter out the inaccuracies. The idea of a deep valley is clearly visible in the 3D plot obtained after the mean squared error operation. To numerically express this abstract idea of depth of the minima point, we defined a variable dip. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 33

34. Dip Filter a1a1 a3a3 a5a5 ma4a4 a6a6 a7a7 a8a8 a2a2 Definition Consider a grid of 9 pixels chosen from the image after the SSD operation has been applied with a minima point at its center. This variable is a measure of the depth of a minima point. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 34

35. Dip Filter Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 35

36. Dip Filter Dip filter value Test cases It can be observed that the dip filter value for 19 objects is greater than 1, with one exception having a value of 0.9728. From the above analysis, we chose a threshold value of 1 for the dip filter and used it in the algorithm. By testing on many test cases, we find that these two filters are able remove most of the false alarms. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 36

37. Our Approach Image of the inventory taken as Input to the model Fuzzy histogram of the object as well as image is calculated. Histogram Intersection in the whole Inventory scene in calculated. The region of interest is extracted by putting a suitable threshold. Mean Square Error value calculated in the region of interest. The points of local minima are searched and a Suppress filter is applied to reject false alarms. The dip value is calculated at all the interest points and ratio of Square Norm and Dip is computed The points of interest are filtered out using dip filter. The filtered points of interest are counted and object is marked on image. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 37

38. User Interface: Operating the App To get the image from the system, 1. Click on the ‘Browse’ button to choose the prototype from within the system Chooses the prototype’s image already stored in the system There are two ways of selecting the prototype: By getting the prototype’s image from the system Getting the image from a camera Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 38

39. Browse to the folder where image is stored, and select it by clicking ‘open’ button. User Interface: Operating the App Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 39

40. User Interface: Operating the App Alternatively, to choose the image from a camera, click on ‘View Scene’ button. A preview window will open to display the scene in front of the camera. Gives a preview of the scene in front of camera Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 40

41. The preview window showing the scene in front Takes a snapshot of the scene in front of the camera User Interface: Operating the App Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 41

42. Click on the get snapshot button. A snapshot of the scene will be displayed. Takes a snapshot of the scene in front of the camera User Interface: Operating the App Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 42

43. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Crops a suitable object from the scene to be used as prototype 3. Click on the crop button and select the region from the snapshot that needs to be detected in real time processing. User Interface: Operating the App 43

44. User Interface: Operating the App After clicking on the crop button, select the region to be cropped, right click on it and choose ‘crop image’. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 44

45. User Interface: Operating the App For object counting, one can use ‘Compute’ or ‘Quick Compute’ button. The cropped object is displayed in the window above. Calculates the no. of objects using fuzzy histogram Calculates the no. of objects using normal histogram Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 45

46. 3 3 User Interface Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved Objects detected would be shown marked and the count will also be displayed. Object count is displayed 46

47. User Interface 47 Auto run radio button is used to specify the amount of time in minutes for which the object counting process should go on. One can specify the time for which calculations need to be done beforehand by using Auto run radio button. After clicking on the radio button, one can write the amount of time required in minutes in the text box. Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 47

48. 3 User Interface: Operating the App One can also stop the calculations any time by pressing ‘Stop Calculating’ button whenever Used to stop the object counting process Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 48

49. User Interface Video Tutorial Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 49

50. Conclusions The application was tested in various situations and has performed well in most cases. For segmentation, the fuzzy color histogram method took more time for computation, but gave better robustness over varied illumination conditions. If the operation is confirmed to be in good illumination condition, conventional histogram method being faster could be preferred instead. Among the two spatial filters, Dip Filter proves to be a powerful tool and in most cases was found to be alone sufficient in detecting all occurrences of the object Though it must be noted that combination of segmentation and the two filters proves to be a more complete and reliable tool for object detection Copyrights © 2014 IIT Kanpur-The Boeing Company USA. All rights reserved 50