1. Advanced Kinect Tricks

2. Shapes Game
Goal is to demonstrate how to create a simple game that uses Kinect audio and skeletal tracking information
The game displays the tracked skeleton of the players and shapes (circles, triangles, stars, and so on) falling from the sky.
Players can
move their limbs to make shapes change direction or even explode,
and speak commands such as "make bigger"/"make smaller" to increase/decrease the size of the shapes
or "show yellow stars" to change the color and type of falling shapes.

3. Demo

4. How does it work? App.xaml App.xaml.cs FallingShapes.cs
Declaration of application level resources.
App.xaml.cs
Interaction logic behind app.xaml.
FallingShapes.cs
Shape rendering, physics, and hit-testing logic.
MainWindow.xaml
Declaration of layout within main application window.
MainWindow.xaml.cs
NUI initialization, player skeleton tracking, and main game logic.
Recognizer.cs
Speech verb definition and speech event recognition.

5. Step 1 (Register for Events)
Kinect.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor )
Kinect.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(SkeletonsReady);
speechRecognizer = SpeechRecognizer.Create();
speechRecognizer.Start(new KinectAudioSource()); speechRecognizer.SaidSomething += new EventHandler<SpeechRecognizer.SaidSomethingEventArgs>(recognizer_SaidSomething);

6. Process Events Process skeleton data Process speech MainWindow.xaml.cs
Method: void SkeletonsReady(object sender, SkeletonFrameReadyEventArgs e)
Update the joint data positions
Process speech
Method: void recognizer_SaidSomething(object sender, speechRecognizer.SaidSomethingEventArgs e)
Match it with the speech vocabulary and get the command

7. Translate the speech command to a list of operations for the falling objects
For ex. the speech commands “reset” will get translated to:
SetPolies(PolyType.All);
SetDropRate(dropRate);
SetGravity(dropGravity);
SetSize(dropSize);
SetShapesColor(Color.FromRgb(0, 0, 0), true);

8. How do you find out a hit?
Any guess?

9. How do you find out a hit?
Convert joints data into bones/segments data (a bone is a segment that connects two joints)
Now, we can maintain a dictionary of {bone, <bone-data>}
Update the segment's position and compute a smoothed velocity for the circle or the endpoints of the segment based on the time it took to move from the last position to the current one. The velocity is in pixels per second.

10. How do you find out a hit? Let’s see FailingShapes.cs
class FallingThings
public bool Hit(Segment seg, ref Point ptHitCenter, ref double lineHitLocation)
if hit, the center point on the segment being hit is returned, along with the spot on the line from 0 to 1 if a line segment was hit.

11. Bouncing off after a hit
Let’s see
FailingShapes.cs
class FallingThings
public void BounceOff(double x1, double y1, double otherSize, double fXv, double fYv)

12. The useful code that you can reuse
SpeechRecognizer.cs
We have already covered it in lab3
Finding out a hit between the skeleton and an object
The bouncing off effect.