Jiwon Kim Steve Seitz Maneesh Agrawala The Office of the Past Jiwon Kim Steve Seitz Maneesh Agrawala

Where is my W-2?

Unify physical and electronic desktops Video camera Recognize video of paper on physical desktop Tracking Recognition Linking Desktop

Demo – Remote desktop

System overview Video camera Computer User Desk Here is an overview of our system. In the setup, a video camera is mounted above the desk looking straight down to record the desktop.

System overview Video of desk Given the video of the physical desktop,

System overview Video of desk Images from PDF ..and images of corresponding electronic documents extracted from PDF’s

System overview Video of desk Images from PDF Track & recognize …the system tracks and recognizes the paper documents by matching between the two, Track & recognize

System overview Video of desk Images from PDF Internal representation …and produces an internal graphical representation that encodes the evolution of the stack structure over time. Desk Track & recognize T T+1

System overview Video of desk Images from PDF Internal representation We call each of these graphs a “scene graph”. Desk Track & recognize T T+1 Scene Graph

System overview Where is my W-2? Video of desk Images from PDF Internal representation Then, when the user issues a query, such as, where is my W-2 form?, Desk Track & recognize T T+1

System overview Where is my W-2? Answer Video of desk Images from PDF Internal representation …the system answers the query by consulting the scene graphs. Track & recognize Desk Desk T T+1

Document tracking example Here’s an example of a move event, before after

Document tracking example ..moves to the right. before after

Document tracking example SIFT [Lowe 04] To classify the event, we first extract image features in both images. before after

Document tracking example We look at the red cluster, and if it contains sufficiently many features, the event is considered a move. Otherwise it’s a non-move and subjected to further classification. before after

Document tracking example Motion: (x,y,θ) If it’s a move, we obtain the motion from the transformation of red cluster before after

Document Recognition Match against PDF image database … … File1.pdf ..where we match features in the region identified as the document against a database of PDF images stored on the computer, also using SIFT features. File1.pdf File2.pdf File3.pdf File4.pdf File5.pdf File6.pdf

Demo – Paper tracking (-18 min) Let me show a demo of the query interface to our system, using the same input sequence I demoed at the beginning of the talk. The right window is the visualization panel showing the current state of the desktop. The left window shows a list of thumbnails of the documents found by the system. The user can browse this list and click on the thumbnail of the document of interest to query its location in the stack. The visualization expands the stack that contains the selected document and highlights the document. The user can open the PDF file of the selected document as well. The interface also supports a couple of alternative ways to specify a document. The user can locate a document by doing a keyword search for the title or the author. Here I’m looking for the document that contains the string “digitaldesk” in its title. The system tells me he paper is in this tack. The user can also sort the thumbnails in various ways. For example, the documents can be sorted in decreasing order of the last time the user accessed each document. The oldest document at the end of this list lies at the bottom of this stack; the second oldest document no longer exists on the desk; and the next oldest document is at the bottom of this stack, and so forth. On the other hand, the most recent document at the beginning of this list is on top of this stack; the next most recent document is on top of this stack, and so forth.

Photo sorting example Here’s an example of using our system for sorting digital photographs. Sorting a large number of digital photographs using the computer interface is usually a fairly tedious task.

Photo sorting example In contrast, it is very easy to sort printed photographs into physical stacks. So we printed out digital photographs on sheets of paper, and recorded the user sorting them into physical stacks on the desk. Here we sort the photographs from two source stacks, one shown on the bottom right of the video, and the other outside the camera view in the user's hand, into three target stacks based on the content of the pictures.

Demo – Photo sorting (-20 min) After processing this video with our system, we can click on each of the three stacks in the query interface, and assign it to an appropriate folder on the computer. Then our system automatically organizes the corresponding digital photographs into the designated folder, and pops up the folder in thumbnail view. I should point out that one clear drawback is the overhead of first having to print out the photographs on paper. However, we think that this can be useful for people who are not familiar with computer interfaces.

Future work Enhance realism More applications Handle more realistic desktops Real-time performance More applications Support other document tasks E.g., attach reminder, cluster documents Beyond documents Other 3D desktop objects, books/CD’s