Presentation is loading. Please wait.

Presentation is loading. Please wait.

Scalable Multi-Label Annotation Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei.

Similar presentations


Presentation on theme: "Scalable Multi-Label Annotation Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei."— Presentation transcript:

1 Scalable Multi-Label Annotation Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei

2 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird Task: Crowdsource object labels for images. Generalization: musical attributes of songs actions in movies sentiments in documents Application: Benchmarking, training, modeling Multi-label annotation Data Item Labels

3 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Current focus: 200 Category Detection (~100,000 fully labeled images) Large-Scale Visual Recognition Challenge ILSVRC

4 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ?????? Answer Question MachineCrowd Is there a table? Yes

5 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird +????? Answer Question MachineCrowd Is there a table? Yes

6 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++???? Answer Question MachineCrowd Is there a chair? Yes

7 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++-??? Answer Question MachineCrowd Is there a horse? No

8 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++--?? Answer Question MachineCrowd Is there a dog? No

9 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird ++---? Answer Question MachineCrowd Is there a cat? No

10 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Naïve approach: ask for each object cost: estimation: use the crowd-machine diagram show UI. TableChairHorseDogCatBird Answer Question MachineCrowd Is there a bird? No

11 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird Naïve approach: ask for each object Cost: O(NK) for N images and K objects

12 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird Furniture Mammal Animal Hierarchy

13 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird Sparsity Correlation Furniture Mammal Animal Hierarchy

14 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ?????? Answer Question Machine Crowd Furniture Mammal Animal Better approach: exploit label structure

15 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ?????? Answer Question Machine Crowd Is there an animal? No Furniture Mammal Animal Better approach: exploit label structure

16 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question Machine Crowd Is there an animal? No Furniture Mammal Animal Better approach: exploit label structure

17 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question Machine Crowd Is there furniture? Yes Mammal Better approach: exploit label structure Animal Furniture

18 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Answer Question Machine Crowd Is there a table? Yes Mammal Better approach: exploit label structure Animal TableChairHorseDogCatBird ??---- Furniture

19 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird +?---- Answer Question Machine Crowd Is there a chair? Yes Mammal Better approach: exploit label structure Animal Furniture

20 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Answer Question Machine Crowd Is there a chair? Yes Mammal Better approach: exploit label structure Animal TableChairHorseDogCatBird Furniture

21 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Selecting the Right Question Goal: Get as much utility (new labels) as possible, for as little cost (worker time) as possible, given a desired level of accuracy

22 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Accuracy constraint User-specified accuracy threshold, e.g., 95% Majority voting assuming uniform worker quality [GAL: Sheng, Provost, Ipeirotis KDD ‘08] Might require only one worker, might require several based on the task

23 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Cost: worker time (time = money) Question (is there …)Cost (second) a thing used to open cans/bottles14.4 an item that runs on electricity (plugged in or using batteries) 12.6 a stringed instrument3.4 a canine2.0 expected human time to get an answer with 95% accuracy

24 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Utility: expected # of new labels TableChairHorseDogCatBird ?????? Is there a table? Yes No TableChairHorseDogCatBird +????? TableChairHorseDogCatBird -????? utility = 1

25 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Utility: expected # of new labels TableChairHorseDogCatBird ?????? Is there a table? Yes No TableChairHorseDogCatBird +????? TableChairHorseDogCatBird -????? TableChairHorseDogCatBird ?????? Is there an animal? TableChairHorseDogCatBird ?????? TableChairHorseDogCatBird ??---- utility = 1 utility = 0.5 * * 4 = 2 Pr(Y) = 0.5 Pr(N) = 0.5

26 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Pick the question with the most labels per second Query: Is there a... Utility (num labels) Cost (worker time in secs) Utility-Cost Ratio (labels per sec) mammal with claws or fingers living organism mammal creature without legs land or avian creature Selecting the Right Question

27 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Dataset: 20K images from ImageNet Challenge Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy Results

28 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Dataset: 20K images from ImageNet Challenge Labels: 200 basic categories (dog, cat, table…), 64 internal nodes in hierarchy Setup: – training test split – Estimate parameters on training, simulate on test – Future work: online estimation Results

29 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: accuracy Accuracy Threshold per question (parameter) Accuracy (F1 score) Naïve approach Accuracy (F1 score) Our approach (75.67)99.75 (76.97) (60.17)99.62 (60.69) Annotating 10K images with 200 objects

30 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: cost Accuracy Threshold per question (parameter) Cost saving (our approach compared to naïve approach) x x Annotating 10K images with 200 objects

31 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Results: cost Accuracy Threshold per question (parameter) Cost saving (our approach compared to naïve approach) x x Annotating 10K images with 200 objects 6 times more labels per second

32 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Speeds up crowdsourced multi-label annotation by exploiting the structure and distribution of labels. Could be a bargain for you! TableChairHorseDogCatBird Furniture Mamma l Animal Hierarchy Correlation Sparsity Conclusions

33 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei

34 TableChairHorseDogCatBird ?????? Answer Question MachineCrowd Is there an animal? No Furniture Mammal Animal Better approach: exploiting hierarchy, sparsity and correlation

35 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question MachineCrowd Is there an animal? No Furniture Mammal Animal Better Approach

36 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird ??---- Answer Question MachineCrowd Is there furniture? Yes Furniture Mammal Animal Better Approach

37 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird +?---- Answer Question MachineCrowd Is there a table? Yes Furniture Mammal Animal Better Approach

38 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei TableChairHorseDogCatBird Answer Question MachineCrowd Is there a chair? Yes Furniture Mammal Animal Better Approach

39 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Algorithm Initialize all labels to ‘missing’ While there are missing labels do – Select a question Q from all possible questions – Obtain an answer A to question Q from the crowd – Set values of some labels to +1 or -1

40 Deng, Russakovsky, Krause, Bernstein, Berg, Fei-Fei Algorithm Initialize all labels to ‘missing’ While there are missing labels do – Select a question Q from all possible questions – Obtain an answer A to question Q from the crowd – Set values of some labels to +1 or -1


Download ppt "Scalable Multi-Label Annotation Jia Deng Olga Russakovsky Jonathan Krause, Michael Bernstein Alexander Berg Li Fei-Fei."

Similar presentations


Ads by Google