Kinect Part II Anna Loparev

OpenNI API Actually general purpose API Need 3 things (http://cs.rochester.edu/courses/297/fall2011/kinect/openni) SensorKinect Driver OpenNI Library NITE Middleware (gesture support) Requirements: Kinect Visual Studio 2008/2010 C++/C# Requirements (From Tutorial): Python SWIG (Allow Python to interface with OpenNI) OpenCV (Allow Python to work with image data) NumPy

Features Recording for debugging Fully backwards compatible Under the GNU Lesser General Public License (LGPL)

OpenNI vs Microsoft SDK Easy to install Don’t have to do calibration pose but gets more false positives Predictive tracking of joints (Good for when sensor losses player) Leads to false positives Uses lots of processing power Gets full 1024x768 resolution from camera Full access to microphone Support for tilt OpenNI Open source (Allows for commercial use) Works across all platforms Hand-only tracking Gesture recognition Rotation in joints Tilt has to be done through motor drivers

Sample Applications (OpenNI) General NiCRead (Prints out depth at center of image) NiSimpleCreate (C++ version of NiCRead without using SamplesConfig XML) NiSimpleRead (C++ version of NiCRead using SamplesConfig XML) SimpleRead (C# version of NiCRead) NiSimpleViewer (Depth and video viewer) SimpleViewer (C# version of NiSimpleViewer) NiViewer (Shows a bunch of different views and allows recording) NiUserTracker (Pose detection and skeletal tracker) UserTracker (C# version of NiUserTracker) NiUseExtension NiAudioSample Recording NiBackRecorder (Allows user to save certain number of seconds) NiRecordRaw NiRecordSynthetic (Opens recording, does transformation, and re-records) NiConvertXToONI (Copies recording)

Sample Applications (NITE) General Sample-Boxes (Highlight boxes based on hand movement and gestures) Sample-Boxes.net (C# version of Sample-Boxes) Sample-CircleControl (Perform a circle gesture and circle is drawn) Sample-Players (Identifies multiple players and their skeletons) Sample-PointServer Sample-PointViewer (Tracks hand and leaves behind trail) Sample-SceneAnalysis (Identifies people) Sample-SingleControl (Identifies wave gesture) Sample-StickFigure (Places skeleton and people in virtual room) Sample-TrackPad (Clickable grid)

Overview

Overview

NITE Abilities Focus gestures Gesture Status Hand tracking Click Wave Swipe left Swipe right Raise hand candidate Hand candidate moved Gesture Status Gesture started Gesture completed Hand tracking User segmentation (Identify and track people) Skeletal tracking

Capabilities Optional extensions (Providers choose if they want to implement them.) Alternative View (Data appears to come from another sensor) Cropping Frame Sync (between multiple devices) Mirror Pose Detection Skeleton User Position (Optimize output depth map) Error State Lock Aware (Allows node to be locked outside of context)

Generators Map Generator Depth Generator Image Generator IR Generator Scene Analyzer Audio Generator Gesture Generator Hand Point Generator User Generator

Generators (Cont) xn::Generator::StartGenerating() - begin generating data xn::Generator:StopGenerating() - stop generation without destroying node xn::Generator::WaitAndUpdateData() - Update data when it becomes available xn::Context::Wait[…]UpdateAll() - Update all generators ([…] determines when)

Simple Program // Set status to OK XnStatus nRetVal = XN_STATUS_OK; // Create and initialize new context xn::Context context; nRetVal = context.Init(); // Create generators xn::DepthGenerator depth; nRetVal = depth.Create(context) // Start generating data nRetVal = context.StartGeneratingAll();

Simple Program (Cont) // Do main loop while (bShouldRun) { // Wait for new data nRetVal = context.WaitOneUpdateAll(depth); // Make sure update worked if (nRetVal != XN_STATUS_OK) printf(“Failed updating data: %s\n”, xnGetStatusString(nRetVal)); continue; } // DO STUFF

Simple Program (Cont) // Clean-up context.Shutdown(); OR depth.Release(); context.Release();

DEMOS

References/More Info Setup: http://cs.rochester.edu/courses/297/fall2011/kinect/openni http://www.codeproject.com/Articles/148251/How-to-Successfully-Install-Kinect-on-Windows-Open.aspx Extensions: http://code.google.com/p/simple-openni/ Other: http://www.openni.org/ http://www.brekel.com/?page_id=671 http://labs.vectorform.com/2011/06/windows-kinect-sdk-vs-openni-2/