1. Jiwon Kim Steve Seitz Maneesh Agrawala
The Office of the Past
Jiwon Kim
Steve Seitz
Maneesh Agrawala

2. Where is my W-2?

3. Unify physical and electronic desktops
Video camera
Recognize video of paper on physical desktop
Tracking
Recognition
Linking
Desktop

4. Demo – Remote desktop

5. System overview Video camera Computer User Desk
Here is an overview of our system. In the setup, a video camera is mounted above the desk looking straight down to record the desktop.

6. System overview
Video of desk
Given the video of the physical desktop,

7. System overview Video of desk Images from PDF
..and images of corresponding electronic documents extracted from PDF’s

8. System overview Video of desk Images from PDF Track & recognize
…the system tracks and recognizes the paper documents by matching between the two,
Track & recognize

9. System overview Video of desk Images from PDF Internal representation
…and produces an internal graphical representation that encodes the evolution of the stack structure over time.
Desk
Track & recognize T
T+1

10. System overview Video of desk Images from PDF Internal representation
We call each of these graphs a “scene graph”.
Desk
Track & recognize T
T+1
Scene Graph

11. System overview Where is my W-2? Video of desk Images from PDF
Internal representation
Then, when the user issues a query, such as, where is my W-2 form?,
Desk
Track & recognize T
T+1

12. System overview Where is my W-2? Answer Video of desk Images from PDF
Internal representation
…the system answers the query by consulting the scene graphs.
Track & recognize
Desk
Desk T
T+1

13. Document tracking example
Here’s an example of a move event,
before
after

14. Document tracking example
..moves to the right.
before
after

15. Document tracking example
SIFT [Lowe 04]
To classify the event, we first extract image features in both images.
before
after

16. Document tracking example
We look at the red cluster, and if it contains sufficiently many features, the event is considered a move.
Otherwise it’s a non-move and subjected to further classification.
before
after

17. Document tracking example
Motion: (x,y,θ)
If it’s a move, we obtain the motion from the transformation of red cluster
before
after

18. Document Recognition Match against PDF image database … … File1.pdf
..where we match features in the region identified as the document against a database of PDF images stored on the computer, also using SIFT features.
File1.pdf
File2.pdf
File3.pdf
File4.pdf
File5.pdf
File6.pdf

19. Demo – Paper tracking (-18 min)
Let me show a demo of the query interface to our system, using the same input sequence I demoed at the beginning of the talk. The right window is the visualization panel showing the current state of the desktop. The left window shows a list of thumbnails of the documents found by the system. The user can browse this list and click on the thumbnail of the document of interest to query its location in the stack. The visualization expands the stack that contains the selected document and highlights the document. The user can open the PDF file of the selected document as well.
The interface also supports a couple of alternative ways to specify a document. The user can locate a document by doing a keyword search for the title or the author. Here I’m looking for the document that contains the string “digitaldesk” in its title. The system tells me he paper is in this tack.
The user can also sort the thumbnails in various ways. For example, the documents can be sorted in decreasing order of the last time the user accessed each document. The oldest document at the end of this list lies at the bottom of this stack; the second oldest document no longer exists on the desk; and the next oldest document is at the bottom of this stack, and so forth. On the other hand, the most recent document at the beginning of this list is on top of this stack; the next most recent document is on top of this stack, and so forth.

20. Photo sorting example
Here’s an example of using our system for sorting digital photographs.
Sorting a large number of digital photographs using the computer interface is usually a fairly tedious task.

21. Photo sorting example
In contrast, it is very easy to sort printed photographs into physical stacks. So we printed out digital photographs on sheets of paper, and recorded the user sorting them into physical stacks on the desk.
Here we sort the photographs from two source stacks, one shown on the bottom right of the video, and the other outside the camera view in the user's hand, into three target stacks based on the content of the pictures.

22. Demo – Photo sorting (-20 min)
After processing this video with our system, we can click on each of the three stacks in the query interface, and assign it to an appropriate folder on the computer. Then our system automatically organizes the corresponding digital photographs into the designated folder, and pops up the folder in thumbnail view.
I should point out that one clear drawback is the overhead of first having to print out the photographs on paper. However, we think that this can be useful for people who are not familiar with computer interfaces.

23. Future work Enhance realism More applications
Handle more realistic desktops
Real-time performance
More applications
Support other document tasks
E.g., attach reminder, cluster documents
Beyond documents
Other 3D desktop objects, books/CD’s