A Natural Interactive Game By Zak Wilson

Background This project was my second year group project at University and I have chosen it to present as it demonstrates clear and concise means of effectively managing a team of 5-6 on a large project with multiple technical aspects each needing individual isolation during research and development stages, whilst also ensuring eventual smooth integration with all the other sections. Amongst the group we managed to delegate the tasks effectively, ensuring the work was split fairly and catered for the individual desires and expertise. This project included many, if not all aspects of project management that any large project would, requiring the group to meet frequently to discuss progress, revise targets and set new goals.

Description This project will develop a game system using Natural Interaction technologies. It will involve the use of Computer Vision techniques to detect and track human hand movement and Computer Graphics techniques to create game pieces that the human hand will be interacting with. It will also involve the use of Augmented Reality techniques so that the game can be viewed on a Head Mounted Display. Finally, the graphics, vision and Augmented Reality systems developed are integrated to create the game.

Natural Interaction We have become a bit "artificial" when accessing computers. We stop moving, sit, and use peripheral devices such as keyboards, mice or touch screens. Our goal is to demonstrate through a gaming interface, the technology to achieve Natural Interaction within software applications.

Task Delegation Splitting the task into key components and developing each of these separately, then integrating them together at the end allows the workload of the project to be effectively divided amongst the group. 3 main project areas: -Computer Vision (tracking and image processing) -Computer Graphics (C++, OpenGL graphics rendering) -Augmented Reality ToolKit.

Computer Vision - OpenCV This is the part of the project that is all to do with the tracking of the user’s hand, i.e. the part in which natural interaction takes place. 1. Various filters are applied to the inputted video stream. 2. Output a video stream that tracks a selected object based on properties exploited by the Kalman tracking algorithm. 3. Modify the algorithm so it outputs co-ordinates of the centre of the tracked object.

Applying Image Filters to the Input Stream.

The Tracking Algorithm

Computer Graphics - OpenGL The game operations and ball physics are defined using the OpenGL package, allowing specification of how the ball should move within this Mixed Reality Environment. Some Examples: resize_scene-draw_border draw_lives-draw_scores draw_paddle-draw_ball draw_scene-move_ball The only method requiring external input from the OpenCV tracking aspect is the ‘draw_paddle’ definition, rendering the image of a simple paddle where the centre co-ordinates of your hand are calculated.

Augmented Reality - ARToolKit The ARToolKit is software allowing the collaboration and integration of real, and virtual objects into a single environment. Virtual Reality – A virtual environment with real objects or aspects integrated in the framework. Augmented Reality – Virtual objects are added to a real video stream for users to interact with in a real environment. This natural interactive game developed was an Augmented Reality system incorporating virtual objects (the ball and paddle) into a live video stream.

Integration of System Components. We decided to keep it simple, and create only a Virtual Environment without portraying any real world objects, and using the hand tracking algorithm developed we were able to generate a ‘paddle’ based on the positioning of your hand with relation to the camera. -Performance Issues- Reducing Graphical Rendering -More Intuitive- More Immersive