1. Computer Science Department Learning on the Fly: Rapid Adaptation to the Image Erik Learned-Miller with Vidit Jain, Gary Huang, Laura Sevilla Lara, Manju Narayana, Ben Mears

2. 2 Learning on the Fly “Traditional” machine learning  Learning happens from large data sets With labels: supervised learning Without labels: unsupervised learning Mixed labels: semi-supervised learning, transfer learning, learning from one (labeled) example, self-taught learning, domain adaptation

3. 3 Learning on the Fly  Given: A learning machine trained with traditional methods a single test image (no labels)  Learn from the test image!

4. 4 Learning on the Fly  Given: A learning machine trained with traditional methods a single test image (no labels)  Learn from the test image! Domain adaptation where the “domain” is the new image No covariate shift assumption. No new labels

5. 5 Learning on the Fly An Example in Computer Vision  Parsing Images of Architectural Scenes Berg, Grabler, and Malik ICCV 2007. Detect easy or “canonical” stuff. Use easily detected stuff to bootstrap models of harder stuff.

6. 6 Learning on the Fly Claim  This is so easy and routine for humans that it’s hard to realize we’re doing it. Another example…

7. 7 Learning on the Fly Learning on the fly…

8. 8 Learning on the Fly Learning on the fly…

9. 9 Learning on the Fly Learning on the fly…

10. 10 Learning on the Fly What about traditional methods…  Hidden Markov Model for text recognition: Appearance model for characters Language model for labels Use Viterbi to do joint inference

11. 11 Learning on the Fly What about traditional methods…  Hidden Markov Model for text recognition: Appearance model for characters Language model for labels Use Viterbi to do joint inference  DOESN’T WORK! Prob( |Label=A) cannot be well estimated, fouling up the whole process.

12. 12 Learning on the Fly Lessons  We must assess when our models are broken, and use other methods to proceed…. Current methods of inference assume probabilities are correct! “In vision, probabilities are often junk.” Related to similarity becoming meaningless beyond a certain distance.

13. 13 Learning on the Fly 2 Examples  Face detection (CVPR 2011)  OCR (CVPR 2010)

14. 14 Learning on the Fly Preview of results: Finding false negatives Viola-Jones Learning on the Fly

15. 15 Learning on the Fly Eliminating false positives Viola-Jones Learning on the Fly

16. 16 Learning on the Fly Eliminating false positives Viola-Jones Learning on the Fly

17. 17 Learning on the Fly Run a pre-existing detector...

18. 18 Learning on the Fly Run a pre-existing detector... Key Face Non-face Close to boundary

19. 19 Learning on the Fly Gaussian Process Regression negativepositive learn smooth mapping from appearance to score apply mapping to borderline patches

20. 20 Learning on the Fly Major Performance Gains

21. 21 Learning on the Fly Comments  No need to retrain original detector It wouldn’t change anyway!  No need to access original training data  Still runs in real-time  GP regression is done for every new image.

22. 22 Learning on the Fly Noisy Document We fine herefore t linearly rolatcd to the when this is calculated equilibriurn. In short, on the null-hypothesis: Initial Transcription

23. 23 Learning on the Fly Premise  We would like to fine confident words to build a document-specific model, but it is difficult to estimate Prob(error).  However, we can bound Prob(error).  Now, select words with Prob(error)<epsilon.

24. 24 Learning on the Fly “Clean Sets”

25. 25 Learning on the Fly Document specific OCR  Extract clean sets (error bounded sets)  Build document-specific models from clean set characters  Reclassify other characters in document 30% error reduction on 56 documents.

26. 26 Learning on the Fly Summary  Many applications of learning on the fly.  Adaptation and bootstrapping new models is more common in human learning than is generally believed.  Starting to answer the question: “How can we do domain adaptation from a single image?”