1. The Palm-tree Index Indexing with the crowd Ahmed R Mahmood*Walid G. Aref* Eduard Dragut*Saleh Basalamah** *Purdue University**Umm AlQura University

2. Outline Motivation Taxonomy for Crowd-based Indexing Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

3. Motivation 5001000200100 2005001000

4. Outline Motivation Taxonomy for Crowd-based Indexing Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

5. Taxonomy

6. Outline Motivation Taxonomy Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

7. Problem Definition Let S be a set of N keys (e.g., images or videos) and q be a query B + -tree-like index is constructed over S Study how to use human workers to search the index Workers perform subjective comparisons between the query image and tree keys, and make subjective decisions, e.g., – Less than, greater than, almost the same – Better, worse, almost the same – Cheaper, more expensive, almost the same

8. Outline Motivation Taxonomy Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

9. Index Structure Why B + -tree? What is tree order and height? How to construct tree? What are performance metrics?

10. Index Structure Why B + -tree? – To obtain predictive query cost – Cost reduction with more keys per node How is the tree order and height determined? – Set by the ability of workers to process at once a specific number of keys Index height Fixed order Error Index order Fixed height Error Order increase Height decrease Fixed dataset size Error

11. Index Construction: How to grow a palm tree? Key associated with some “Quantitative Value” – Keys have a subjective property and an associated quantitative value – Index constructed based on the quantitative value – Example: Damaged car images with repair cost Key  car image Subjective property  car damage Qualitative value  repair cost 5001000200100 2005001000

12. Index Construction: How to grow a palm tree? (Cont’d) Key associated with some “Qualitative Property” Keys have a subjective property only Index constructed by successive insertions e.g. images of butterflies to be ordered based on beauty

13. Performance Metrics What are performance metrics? – Error: Distance between ground truth and selected result – Cost: Total number of tasks to complete a job Cost Error

14. Outline Motivation Taxonomy Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

15. Traversal Algorithms How to descend the tree? – Leaf-only aggregation – All-level aggregation – All-level aggregation with backtracking

16. Leaf-Only Aggregation 12345678910111213141516 24681012 14 16 371115 513 9 w1 w3w2 Tasks per worker 4 4 4 Even budget distribution – Number of workers = Budget/Tree Height Budget: 12

17. All-Levels Aggregation w1 w2 w3 12345678910111213141516 24681012 14 16 371115 513 9 w1 w2 w3 Even budget distribution – Replication per level = Budget/Tree Height Tasks per level 3 3 3 3 Budget: 12

18. All-Levels Aggregation 12345678910111213141516 24681012 14 16 371115 513 9 Uneven budget distribution based on – Probability of distance d error at level l: P dl – Expected Distance Error per level: EDE tasks per level 6 3 EDE 3 1.5 1.5 Budget: 12 2 1

19. Algorithms: Crowd-Search Backtracking All-Levels Aggregation 12345678910111213141516 24681012 14 16 371115 513 9 Node A Node B Node C Node D

20. Outline Motivation Taxonomy Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

21. Preliminary Experimental Results Experimental Setup Squares dataset – Generated 200 images of squares with different sizes Cars dataset – 1300 image of used cars associated with desired selling prices – Collected using a custom crawler from the Craigslist Website Crowd: – Students in the DB Group at Purdue (and their spouses) – (IRB Approval)

22. Preliminary Experimental Results Sample task

24. Preliminary Experimental Results Higher error on cars dataset Error increases as fanout increases Error decreases as number of replications increase All-levels aggregation has less error than leaf- only aggregation Mean Error while changing the tree fanout and the number of workers (replications)

25. Preliminary Experimental Results Mean Cost while changing the tree fanout and the number of workers (replications) The taller the tree the higher the cost Higher cost on the cars dataset (has more keys) More replications involve higher cost Order increase Height decrease Fixed dataset size Error

26. Outline Motivation Taxonomy Problem Definition The Palm-tree Index Structure Traversal Algorithms Preliminary Experimental Results Conclusions and Future Work

27. Conclusions – The Palm-tree allows employing humans to perform index operations on keys that cannot be indexed by computer Future Work – More extensive experimental evaluation – Mathematical analysis – Multi-dimensional indexing

28. Questions?