Salevich Alex & Frenkel Eduard Wizard Hunting 2D Shape recognition in 3D environment

Problem Definition and Motivation Main Problem: Nowadays there are many VR applications with complex menus and capabilities, giving hard time for users to adjust and finally give up on using the app. Goal: Our project's goal is to overcome this challenge and present a new way of interacting with an application via recognition of motion and shape instead of menus and buttons. Our solution will allow to link the controller motion in 3d space to a certain behaviour in the application. In our case we will implement the idea as a game in which the user plays a wizard that can draw his spells as shapes and for each one of them a predefined spell will be activated.

Solution-Features Platform : Hardware platform: HTC vive Software platform: Unity 3D,VS 2017,Python,Mnist/Tensorflow Gui: Because of the nature of the app we don’t have a defined in-game gui, all the interactions in the game are via controller movement which are recorded when the user holds the trigger button of the left controller. There is a drawn aim which show where the spell will be shot to, after deciding where to shoot to the user will press the trigger and draw the spell he wants, by releasing the trigger the magic will be invoked.

Solution-Implementation Python: We have created a python application that serves as a server that receives and analyses a shape/picture and send an answer regarding the identity of the shape to the client. We have used Mnist/ TensorFlow algorithm for shape recognition. We have trained the algorithm to recognize 3 different shapes and build an interface for image input and output via TCP sockets. Since the application can receive an unknown shape we altered the recognition accuracy calculation In 2 ways: a) We have added additional shapes to the algorithm, which reduces the change of misrecognition of unknown shapes as legitimate shapes, unknown shapes (as whole) will have more chance to be chosen than a legitimate shape. b) We changed the accuracy calculation so that a shape which is far from being the right one will get significantly lower chance of being chosen then similar shapes.

Solution-Implementation Unity: On unity side we have separated the VR world into 2 different space: The user space – the space which can be seen by the user: The environment The enemies The drawn shapes The magic spells The scene starts with a simple menu that explains which shape will invoke a certain magic spell, the menu is shown or hidden by pressing the menu button at the controllers. The user aim at the target wizard with the help of a circular aim and then holds the trigger to draw the required shape. At the instant the user releases the trigger the drawing will be sent to the image processing space and the right magic should be invoked. If the shape is not recognized nothing will happen.

Solution-Implementation Unity: The image processing space – the space which includes the setup for processing the shapes drawn by the user: A blank canvas for plain background Line drawer component for redrawing the shape from user space A camera to capture the redrawn shape This space is not visible to the user, in this space the shape that was drawn in the user space is redrawn in black with a white background with a fixed orientation aligned with the camera view space using the vertices of the original shape. Afterwards the camera takes the picture of this space which is then transmitted to the shape recognition program. After a quick evaluation the program sends via a TCP socket the identity of the shape to unity. Unity will invoke the correct spell if the recognition is within its known shapes.

Improvements Ideas Improvement Ideas: The following ideas might be implemented as part of future projects: Expand the shape recognition database to support more shapes. Implement an interface to receive shapes images as user input and expand the database according to his unique drawings. Expand the feature to support 3D images/models.

External Links Website: https://sloovi1990.wixsite.com/wizardhunting Git repository: https://github.com/sloovi90/WizardHunting Video link:https://www.youtube.com/watch?v=j_v4hae5gtE