16-899A Pixels to Percepts Instructors: Alexei (Alyosha) Efros, 225 Smith Hall, CMU.Lavanya Sharan, Disney Research Pittsburgh Web Page:

Today Instructors Why This Course? Why Perception? Administrative stuff Overview of the course Old Lady or Young Girl?

Lavanya Sharan Postdoctoral Researcher Visual perception, both behavioral and computational aspects. Computer Vision Human Visual Perception Computer Graphics

Light or dark? Real or fake Oreo? Material category? Emotion? Where does he land? GRADUATE WORK RECENT WORK

Alexei “Alyosha” Efros Associate Professor, Robotics & CSD Ph.D 2003, from UC Berkeley (signed by Arnie!) Research Fellow, University of Oxford, ’03-’04 Research Interests Computer Vision, Computer Graphics, Data-driven Approaches

PhD Thesis on Texture and Action Synthesis Smart Erase button in Microsoft Digital Image Pro:

Why This Course? A perfect storm: I wanted to do this for a long time but felt I wasn’t qualified Lavanya happened to be in Pittsburgh and she is more than qualified The first-ever textbook on Perception for CS types (Thompson et al) is coming out We will be giving out printouts soon There are great perception courses in Psych dept (e.g. by Klatzky), but our aims are different: The aim to be much more applied There are real problems in vision/graphics that we can solve Exposure for these who might never take perception otherwise

Why study Perception? It’s fascinating! Ok, but so is astronomy It’s still in the “early days” Relatively easy to get up to speed Still lots of low-hanging fruit OK, OK, but why should I care? I just want to create a seeing robot and/or create the next “Avatars” Two answers: 1) a classic one and 2) a real one

Classic Answer Understanding Human Visual Perception should help us design better vision algorithms After all, human is the only vision system known to work Just like understanding how birds fly should help us design airplanes …WAIT A MINUTE Ok, so there is a fine line between “helping” and “mindless copying” Beware of papers claiming to be “biologically inspired”

The Real Answer Human Perception is an integral part of most vision and graphics endeavors i.e. you can’t get away from it!

“What does it mean, to see? The plain man's answer (and Aristotle's, too). would be, to know what is where by looking.” -- David Marr, Vision (1982)

Vision: a split personality “What does it mean, to see? The plain man's answer (and Aristotle's, too). would be, to know what is where by looking. In other words, vision is the process of discovering from images what is present in the world, and where it is.” Answer #1: pixel of brightness 243 at position (124,54) …and depth.7 meters Answer #2: looks like bottom edge of whiteboard showing at the top of the image Which answer is better? Is the difference just a matter of scale? depth map

Measurement vs. Perception

Proof!

Measurement vs. Perception Pablo Picasso The Guitar Player

Vision as Measurement Real-time stereo on Mars Structure from Motion Physics-based Vision Virtualized Reality

Vision as Understanding Object / Scene Recognition Action / Activity Recognition even scene geometry

Perception in Graphics One of central goals of graphics is to simulate visual experience as realistically as possible So, if we could simulate the physics of light perfectly, why bother with perception? 1) Practical Reasons 2) Artistic Reasons

Complexity Amazingly real But so sterile, lifeless… Why?

Device Limitations Low Dynamic Range Limited modalities (visual + audio) No smell, taste, temperature Limited field of view

Artistic Reasons Reality is boring Visual content creators (film directors, game designers) want hyper-reality

How to study Perception? Ok, you convinced me: Perception is useful. But why not just “open up the brain” and figure out how it works? Two most popular methods in physiology: Cell recordings Recordings from small number of cells (1-100, out of millions) fMRI Overall activity in the brain as function of time Very low spatial resolution

Tale of Martians with an old PC Cell recordings fMRI

Instead, in this course…

Course Goals Read some interesting papers together Learn something new: both you and us! Get up to speed on big chunk of perception research Use perception / human studies in your own research Try your hand at a perception study Learn how to speak Learn how think critically about papers

Course Motto: 50/50 50% Lectures / 50% student presentations 50% textbook / 50% research papers 50% from perception side / 50% from vision/graphics side 50% Alyosha / 50% Lavanya

Course Organization Requirements: 1.Class Participation (25%) Keep annotated bibliography Post on the Class Blog before each class Ask questions / debate / flight / be involved! 2.Three Projects (75%) Two Analysis Projects (25% + 25%) Implement / Evaluate a paper and present it in class 1 CS paper, 1 perception paper Synthesis (Final) Project (25%) Produce a publishable result (study or implementation) Can be continuation of analysis project or something new Can be done solo or in groups of 2

Class Participation Keep annotated bibliography of readings (always a good idea!). The format is up to you. At least, it needs to have: Summary of key points A few Interesting insights, “aha moments”, keen observations, etc. Weaknesses. Unanswered questions. Areas of further investigation, improvement. Before each class: Submit your summary for current readings in hard copy (printout/xerox) Submit a comment on the Class Blog ask a question, answer a question, post your thoughts, praise, criticism, start a discussion, etc.

Analysis Project 1.Pick a paper / set of papers from the list 2.Understand it as if you were the author e.g. Re-implement it; try to replicate the study If there is code, understand the code completely Run it on data the same data (you can contact authors for data and even code sometimes) 3.Understand it better than the author Run it on LOTS of new data (e.g. LabelMe dataset, Flickr dataset, etc, etc), or try to replicate the study with new data Figure out how it succeeds, how it fails, where it fails, and, most importantly WHY it fails Look at which parts of the code do the real work, and which parts are just window-dressing Maybe suggest directions for improvement. 4.Prepare an amazing 45 min presentation Discuss with us twice – once when you start the project, and 3 days before the presentation

Synthesis Project Can grow out of analysis project, or your own research 1-2 people per project. Project proposals mid-semester Project presentations at the end of semester. Results presented as a CVPR-format paper. Hopefully, a few papers may be submitted to CVPR / VSS etc

End of Semester Awards We will vote for: Best Analysis Project Best Synthesis Project Best Blog Comment Prize: dinner in a French restaurant in Paris (transportation not included!) or some other worthy prizes

Course Outline The Big Picture Week 1: Physiology Week 2: Philosophies of Vision Week 3: Psychophysical Methods Illumination and Shadows Week 4: Color, lightness, brightness perception Week 5: Shadows and Shading Shape Perception (week 6) Texture and Materials (week 7) Objects and Scenes Categorization and Recognition (week 8) Geometry and Perspective (week 9) Spatial Cognition (week 10)

Course Outline (cont.) Faces (week 11) Visual Attention (week 12) Motion Perception (week 13) Visual Realism (week 14) Perception and Art (week 15) Final Project Presentations Subject to Change

Perception & Art Livingstone