Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University.

Similar presentations


Presentation on theme: "Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University."— Presentation transcript:

1 Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University

2 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 2 How Do We Compare Image Retrieval Algorithms? Different research groups use images from different sources. Image sets are of different sizes. Tasks are different. –Each researcher identifies set of queries and targets through subjective criteria. –Cant share keys because image sets are not standard. Answer: Badly!

3 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 3 How Its Usually Done Each researcher tests a proposed algorithm against a few baselines. –e.g., Color Histograms. No data to compare latest techniques. –Test sets are different. –Implementation of baselines may differ also.

4 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 4 Some Difficulties Given a query, which target is most relevant? Context will determine answer. ?

5 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 5 What Should a Good Test Do? Provide comparable results even with different image sets. Offer insight into the behavior of different retrieval algorithms. Run quickly. Allow for easy implementation.

6 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 6 Proposal: Altered-Image Queries f Image from Library Altered Image Query Image Library etc. Look for rank of original: Retrieved ranks:

7 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 7 The Crop Test Crop image to k% of its original area. Simulates close-up shot of same subject. OriginalCrop-50

8 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 8 The Jumble Test Shuffle tiles in image divided on an h k grid. Simulates image with similar elements in a different arrangement. Original Jumble-4 4

9 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 9 The Low-Con Test Decrease contrast to k% of its original range. Simulates altered lighting conditions and/or camera differences. OriginalLow-Con-80

10 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 10 Typical results Most retrievals are at low rank. A few retrievals are at much higher rank. Median: 26 Mean 205 Median and mean summarize the results of multiple repetitions.

11 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 11 Difficulty of Altered-Image Queries Both mean and median increase with difficulty. Note order-of-magnitude changes.

12 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 12 How Stable are Altered-Image Queries? Ran Crop-50 on three entirely different sets of 6000 images. Some consistency even with different test sets. Look for order-of-magnitude change. Set 1Set 2Set 3MeanDev. Median Rank Mean Rank

13 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 13 Does the Number of Images Matter? Found linear dependence on number of images.

14 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 14 How Many Queries Must Be Run? Small % of total image set gives decent figure.

15 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 15 Comparing Algorithms Using Altered-Image Queries Three algorithms compared using altered image queries. Especially good or bad performance can be identified. CropJumbleLow-Con Histograms Correlograms STAIRS (Tuned)

16 June 12, 2000Workshop on Content-Based Access of Image and Video Libraries 16 Final Thoughts Altered-Image Queries are... –Well defined. –Easy to implement. –Consistent over different image sets. A useful addition to our evaluation toolkit. Also offer diagnostic potential.


Download ppt "Using Artificial Queries to Evaluate Image Retrieval Nicholas R. Howe Department of Computer Science Cornell University."

Similar presentations


Ads by Google