1. 3D Images Search Benchmark By:Michael Gustus Anat Paskin Supervisor:George Leifman

2. Introduction

3. There are numerous algorithms for determining similarity between 3-D objects. Those output a Numeric distance between two objects as an index of similarity.

4. Introduction We have: We have: –Database partitioned into classes of objects (such as plains, cars, trees etc.). We receive: –Algorithms for assessing similarity between any two objects. The system should evaluate how suitable the algorithm is to be used for the task of classifying objects according to those classes. Problem Definition

5. Algorithm Designers’ Difficulties

6. Introduction CHAIRS CLASS example

7. Difficulties The “CHAIRS CLASS”. Does computer knows that these images are chairs?Does computer knows that these images are chairs? Can he tell us that they look alike?Can he tell us that they look alike? How good can computer pick out the others, if we will give him one of them?How good can computer pick out the others, if we will give him one of them? These are the tasks that we will measure for different algorithms and give them an evaluation.

8. Difficulties PLANTS CLASS example

9. Difficulties Finding similar 3D object in a large database of 3D objects: 3D objects are described only by their surfaces.3D objects are described only by their surfaces. No information at all about what the object represents. Is the object an airplane, a person or a ship???No information at all about what the object represents. Is the object an airplane, a person or a ship??? Objects have different spatial orientation: objects might be flipped, rotated, translated and even scaled up or down. There is no information or indication of that in the objects representation.Objects have different spatial orientation: objects might be flipped, rotated, translated and even scaled up or down. There is no information or indication of that in the objects representation.

10. Benchmarking Difficulties

11. Difficulties The system should evaluate how suitable the algorithm is to be used for the task of classifying objects according to those classes.The system should evaluate how suitable the algorithm is to be used for the task of classifying objects according to those classes. This concept is rather hard to define precisely, since there are several quality factors to consider such as accuracy, speed, adjustment to user’s needs etc. It’s hard to systematically determine their relative importance, and give each a quantative measure.This concept is rather hard to define precisely, since there are several quality factors to consider such as accuracy, speed, adjustment to user’s needs etc. It’s hard to systematically determine their relative importance, and give each a quantative measure.

12. Difficulties For example: Some requirements from a "good" similarity algorithm would be invariance under (rigid transformations) scaling, rotation and translation, low sensitivity to small perturbations, sampling resolution, etc. (robustness). Some requirements from a "good" similarity algorithm would be invariance under (rigid transformations) scaling, rotation and translation, low sensitivity to small perturbations, sampling resolution, etc. (robustness).

13. Difficulties Some times objects from the same class look very different (or vise versa, such as pens and rockets). This inserts noise in to the classification process.

14. Goals

15. Our main goal is to find an “objective” method of evaluation that will rank search algorithms by their “search ability”.Our main goal is to find an “objective” method of evaluation that will rank search algorithms by their “search ability”. The secondary goal is to find a test/tests that "successfully" capture the term of "search ability".The secondary goal is to find a test/tests that "successfully" capture the term of "search ability".

16. Goals So, what's the innovation ? In this project we will examine several different benchmarking methods on a fairly large database (about 1600 samples and a few dozens of classes).In this project we will examine several different benchmarking methods on a fairly large database (about 1600 samples and a few dozens of classes). We will run the benchmarks on the "Sphere Algorithm" (only for an example) on a database not specifically designed for any algorithm.We will run the benchmarks on the "Sphere Algorithm" (only for an example) on a database not specifically designed for any algorithm. So the system should be fairly objective in benchmarking 3D comparison algorithms.So the system should be fairly objective in benchmarking 3D comparison algorithms.

17. Goals As to the second goal, we have some benchmarking algorithms such as: –well known : “First/Second tier”, “Nearest Neighbor” etc. –based on innovative ideas, such as: “Decay test“. –and two tests of our own: one of them, “Best First" tries to capture a slightly different concept of "classification ability” rather than "search ability".

18. Implementation

19. Server side: The Benchmarking program runs on a server. It "receives an algorithm", tests it using the method described later, and records the results for later download by clients.The Benchmarking program runs on a server. It "receives an algorithm", tests it using the method described later, and records the results for later download by clients. Client side: The User should implement his algorithm as a Dynamically Linked Library (.dll) in which he provides us interface functions.The User should implement his algorithm as a Dynamically Linked Library (.dll) in which he provides us interface functions.

20. Implementation The uploading is via upload utility on our internet site: http://tochna4.technion.ac.il/misha1/Benchmark.htm

21. Implementation The advantage of using a DLL: 1.Is mostly because it’s the most suitable way to implement an ADT (very generic). 2.It provides full copyrights protection to the algorithm author. 3.It is supported by every MS Windows platform. 4.There is no need to recompile, the linkage is done in runtime.

22. Our Method of testing

23. We've collected a database of ~1600 images, divided into a few dozens of classes. All the tests we've designed rely on the method search(i), which calls the algorithm on image i, for each image j in the database, and orders the results by increasing "estimated distance" the algorithm produced.

24. Our Method of testing All tests we've designed rely only on this ordering, and on the class each j actually belongs to, and don't utilize the distances themselves. 1 0 Distance to image i Other images

25. Our Method of testing The result for each element i is a number in [0..1]. (0 - worst performance, 1 - best performance) It attempts to reflect how well the test did for the object by it's criteria. To evaluate the algorithm's overall performance it's reasonable to average over all the elements in the database (possibly a weighted average, by class sizes). All but one tests use this method.

26. Benchmark Tests

27. Partial success Test Description The Test has a parameter 0 < Frac <= 1. For each element i in a class, let it’s class be of size Si, scan the result, and count how many element are encountered until the Frac*Si elements of i’s class are seen (denote it n). The result for i in this search is The overall Test result is an average over all i’s in the database.

28. Benchmark Tests Motivation The test tries to grasp the concept of “success density”, that is, what is the density of correct results on different scales, as described in the test. Frac indicates the scale we are interested in.

29. Benchmark Tests Gradual decay Test Description For each i in database, scan the test results, for each element j in the result, compute Let Ci be the size of i’s class. The result for i is: (That is, normalize by the best possible result). The overall Result is the average over all ’s.

30. Benchmark Tests Motivation proportion The value of each correct result is estimated according to it’s position in the search, the further a result is located, the less it’s worth, since users are “impatient”, and are less likely to examine results far away. It’s possible to make the base a parameter that changes in proportion to “patience factor”.

31. Benchmark Tests Nearest neighbor Test Description For each i in database, check whether the second result is of i’s class. The final result is an average over all i’s in database. It’s also possible to average by classes (regardless of size, which is actually possible in any other test). Motivation The first result in any reasonable algorithm is the object itself, so it’s not a good quality indication to count it. It checks whether the second result is of the same class, used when It’s important that the element considered best is actually from the same class.

32. Benchmark Tests First/Second Tier First Tier For each element, test the success percentage for the first k elements, where k is the size of the element’s class. Final result is the average of all results over the elements in the database. It’s also possible to average by classes. * also known ( in data retrieval terminology) as recall. Second Tier Same as First Tier, but works with k=2*(size of e’s class for each element we search).

33. Benchmark Tests Best First Test description: For each class i, run First Tier on each element in the class. Find the best result among them, Best_i, then average over the classes. Motivation: Till Now We’ve considered tests that do some kind of evaluation on each element and then averages on all element (possibly weighted by class sizes). Such measures don’t necessarily reflect how well this algorithm can be used as a classifier. That is, given some new element, that belongs to one of the classes in the database, use the database and the algorithm to determine to which class the new element belongs.

34. The user interface

35. The user interface is rather minimalistic.The user interface is rather minimalistic. Benchmarking of an algorithm requires only few basic functions to interface with the system:Benchmarking of an algorithm requires only few basic functions to interface with the system:

36. The user interface this function processes the input off file, and creates a “signature” that holds all the information the algorithm needs about the object, and returns it’s start address as void*. void* CreateSignature(File smp.off);

37. The user interface gets two signatures previously created by it, and returns the computed “distance” between them. int CompareSignature(void* sig1, void* sig2);

38. The user interface The system also supports benchmarking with feedback, if the benchmarked algorithm supports this version. In this case the search runs repeatedly, and for each iteration the algorithm gets info of which signatures of the first few were correct (in the objects’ class), and which weren’t. int CompareSignatureWithFeedBack(…);

39. Results

41. Conclusions

42. Results Nearest Neighbor: As expected for large enough (10+) the results are high. For small classes the results are around 0, unless there is very high similarity in the class. Another element is likely to get a higher grade than all others in class.