TEMPLATE DESIGN © The basic model for a trigonometric setup requires that the HID be seen by at least two cameras at any given time for tracking to work. This experiment consisted of two Nintendo Wii remotes aimed at a common area. The remotes would return the pixel value locations of two infrared LEDs fixed in a horizontal plane on the front of a third Wii remote which would return accelerometer values for pitch [3]. Calculations on these return values find where the Wii remote is pointed on a screen. Practical Infrared Object Tracking with Wii Remotes Sage Browning, Phillip Weber, Jürgen Schulze California Institute for Telecommunications and Information Technology University of California San Diego Overview Final Approach Goal: To develop a low cost, high accuracy method of tracking objects in 3D space utilizing infrared LED's and cameras for use as human interface devices (HIDs) when working with large displays. A Trigonometric Approach Problems with a Trigonometric Approach Limited Working Space: With a maximum vertical viewing angle of 27º, and 45º in the horizontal plane, and with the requirement that a point must be in view of at least two cameras, the Wii remote provides a relatively small space to work in. i = Ø max (p i /p max -.5) y = d/(1/tan( 1 + Ø 1 ) + 1/tan( 2 + Ø 2 )) x = y/tan( 1 + Ø 1 ) -.5d z = y/tan( 3 + Ø 3 ) Ø x = tan -1 ((y 1 - y 2 )/(x 1 - x 2 )) x = ytan(Ø x ) z = ytan(Ø pitch ) X = x + x Z = z + z Loss of Tracking: The point of placing two IR LEDs on the front of a Wiimote is to capture rotation in the horizontal plane (yaw), which cannot be tracked by accelerometers (as they only determine the direction of down). However, in rolling the remote 90º so that the IR LEDs are lined up vertically, yaw tracking is lost. In the picture, only the small horizontal movement is perceived POSIT Method The Wii Remote as a Sensor The Nintendo Wii Remote contains, among other things, eleven buttons three accelerometers bluetooth connectivity an infrared camera a basic image processor These components, coupled with a low price and the availability of Wii remote libraries in a variety of programming languages make it an attractive sensor (or toy). Accelerometers: The remotes three accelerometers show the direction of gravity, and determine pitch and roll. Fig. 1: Demonstrates return values for a Wii remote in a continuous roll [1] Fig 4: View from above. Infrared Camera: The Wii remotes camera reads image data, and an internal processor determines which infrared points are worthy of tracking. Up to four infrared points are tracked, and their radius, and pixel locations in the image plane sized 1024 by 760 pixels are sent via bluetooth. Fig. 2: Wii remote vision [2] Given the pixel values p i of a single point: i = difference in degrees from the center of the camera to the point being tracked Ø i = difference in degrees from the center of the camera to the plane of the screen (x, y, z) = location in real space, given in units of d, where the origin is the point directly between the two Wii remotes d = distance between the two cameras Given (x, y, z) of two points: Ø x = difference from normal in reference to screen x, z = difference from absolute location, and pointed location X, Z = final location pointed to on screen Fig. 3: Geometric representation of camera setup as seen from above. Finding x 1, y 1 (left) and the horizontal direction pointed (right). z 1 and vertical rotation are found similarly. References [1] DarwiinRemote [2] Cwiid Library [3] Wiiuse Library, Michael Laforrest, [4] Dementhon, Daniel & Davis, Larry S., Model-Based Object Pose in 25 Lines of Code, 1995, International Journal of Computer Vision, 15, [5] Oliver Kreylos, Wiimote Hacking In order to achieve complete pose tracking and to increase the area tracked, a new algorithm was used [4]. DeMenthons POSIT code is designed to quickly estimate the location and orientation of a non-coplanar rigid object in 3D space given the objects dimensions, and the camera lens properties. The return values are a rotation matrix and a translation vector. Unfortunately, DeMenthons POSIT algorithm was designed for an object with many more points than is practical for a hand held device, and while his paper indicates a low error, he also uses a cube with eight points. Our handheld device with four points has a much greater error, giving the effect of a jittery object that constantly moves about the screen. Instead of creating our own system, we shifted to extending a pre-existing project [5] to multiple Wii remotes, in place of a single one as the original project was designed. This allows for a much greater tracked area. Further, this system makes use of both infrared and accelerometer data to achieve a more steady tracking algorithm. Under this setup, a Wii remote with infrared LEDs in a non-coplanar arrangement on the front will be tracked by at least one camera at any given time. Though only one camera is needed in order to compute the position and rotation of the remote, overlap between cameras provides a buffer so that all IR points are visible to at least one camera during any given pose calculation. The program was further extended to include a function to calculate initial camera offsets, as it would be useless to point the cameras straightforward and level. Once calculated, the program takes into account these values when making all subsequent pose calculations. The offsets are found by placing a rigid object on a level plane at the desired origin and finding the pose of the object. Since the objects pose is already known, we are in fact finding the pose of the cameras themselves. The rotation and translation offset vectors are then written to file along with the corresponding Wii remotes unique MAC address so that configuration and identification of the Wii remotes need only be run when camera positions are changed.