NOMAD: Camera-Projector System for Medical Applications Group Members: z Budirijanto Purnomo z Paul Alan Roberts z Nicholas Lord

Goals zTo design a system that permits medical display information to be projected onto a planar surface anywhere in the surgical field. zTo provide surgeons with lightweight, portable and sterile displays in the operating room.

System Setup NOMAD screen camera Projector mirror

Advantages zConvenient placement of the presentation of information. zImprove the quality of surgery. zDecrease cost for sterilizing displays. zReduce surgery time. zIncrease space in the operating field.

Usage Instructions [The projector with pan-and-tilt mirror, camera and computer are assumed to be already installed in the room]. 1. The surgeon adjusts the NOMAD screen into a convenient place. 2. Start the calibration program. 3. After the calibration is complete, the surgeon switches to the medical application.

Usage Instructions (continued) 4. The surgeon can interact with the NOMAD screen with virtual buttons on the screen. 5. Go back to step 1 if the surgeon needs to readjust the NOMAD screen’s placement.

Proposed Work zBuild a system prototype. zDevelop an automatic calibration for NOMAD screen. This involves two steps: yDirect pan-and-tilt mirror such that the projected image lies on the NOMAD screen. This can be done by applying an object detection algorithm to the observed camera’s image to obtain an initial guess for directing the pan-and-tilt mirror.

Proposed Work (continued) Then, an iterative process is performed to match the position of the actual projected image with the observed image. yCompute the pose of NOMAD screen from an image captured by the camera. Supply this information to the ED-Projector system developed by IBM to correct oblique projection.

Proposed Work (continued) zAdd a touch screen capability. This involves developing a simple GUI and using IBM’s software to track fingertips movement.

Schedule Feb 15th: IBM’s contact (Rick). Feb 19-21st: Work on proposal. Feb 26th: Set up a simple camera system and use matlab for camera calibration. Feb 28th: Build and design a NOMAD screen. March 5-14th: Develop pose estimation algorithm. March 15th: Receive IBM hardware and software. March 26-28th: Build mounting structure, interface and register projector, camera and mirror.

Schedule (continued) March 29th: Receive full XML interfaces from IBM. April 2-4th: Be able to project image onto the NOMAD screen. April 9-11th: Calibrate vision system for touch screen capabilities. April 16-18th: Develop final report.

Deliverables zMinimum: A system that projects an image accurately onto the NOMAD screen. zExpected: Automatic calibration and dewarping are performed on top of the minimum. The NOMAD screen has a touch screen capability. zMaximum: Allows pan-and-tilt camera and live videos to be projected onto the screen.

Dependencies zHardware: yProjector (ERC lab). yPan-and-tilt mirror, PC and a camera (IBM). yVideo card that supports dual-display (???). yFrame grabber or firewire card (???). yMounting structure (Build). yCables + converters + connectors (IBM, ERC).

Dependencies (continued) zSoftware: yED-software (IBM). yXML interfaces (IBM). yMatlab (ERC). yXVision (CIRL). yC/C++, Java compiler (ERC). zOther: yXML book (???).

Background Readings zEDML, VIML, CAML Reference Manual by IBM. zClaudio Pinhanez, The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. zRahul Sukthankar et al., Smarter Presentations: Exploiting Homography in Camera-Projector Systems.

Background Readings (continued) zYing Wu et al., Visual Panel: From an Ordinary Paper to a Wireless and Mobile Input Device. zTrucco and Verri, Introductory Techniques for 3-D Computer Vision. zXML Reference materials.