1. 5/19/2015 EEC484/584: Computer Networks 1 EEC-490 Senior Design (CE) Kinect Programming Tutorial 1 Wenbing Zhao wenbing@ieee.org

2. Kinect Application Development Requirement Hardware: a relatively powerful Windows 7 PC/Laptop (Windows XP not supported) Software  Visual Studio 2010 (or Express) for C# (install this first!)  Kinect for Windows SDK  For speech recognition Speech platform runtime (32-bit & 64-bit) Microsoft speech platform sdk MS Kinect English language pack

3. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx Create a new Windows Presentation Foundation app  File->New->Project…  Select “WPF Application”, name the project “KinectBasic”, click OK Add reference to Kinect library  Right click References =>Add Reference…  Select Microsoft.Research.Kinect =>OK

4. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx Define UI  TextBlock Text FontSize  Button Content  Images Leftmost image: change variable name to image Middle image: change variable name to imageCmyk32 Rightmost image: change variable name to depth

5. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx Add events  When “Start” / “Stop” button is pushed In Properties pane => Events tab In “Click” field: can specify event method name Otherwise, a default method name is used (in the form buttonName_Click())

6. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs  Add header include using Microsoft.Research.Kinect.Nui;  Add member variable In public partial class MainWindow:Window {  Runtime kinectNui;  int totalFrames = 0; // for frame rate calculation  int lastFrames = 0; // frame rate  DateTime lastTime = DateTime.MaxValue; // frame rate  const int RED_IDX = 2;  const int GREEN_IDX = 1;  const int BLUE_IDX = 0;  byte[] depthFrame32 = new byte[320 * 240 * 4]; // for frame manipulation

7. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs  Add Kinect init method: private void InitializeNui() {}  Declares kinectNui as a Runtime object, which represents the Kinect sensor instance nui = Runtime.Kinects[0];  Open the video and depth streams, and sets up the event handlers that the runtime calls when a video, depth, or skeleton frame is ready  An application must initialize the Kinect sensor by calling Runtime.Initialize before calling any other methods on the Runtime object kinectNui.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

8. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{}  To stream color images: The options must include UseColor Valid image resolutions are Resolution1280x1024 and Resolution640x480 Valid image types are Color, ColorYUV, and ColorYUVRaw kinectNui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.ColorYuv);

9. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{}  To stream depth and player index data: The options must include UseDepthAndPlayerIndex Valid resolutions for depth and player index data are Resolution320x240 and Resolution80x60 The only valid image type is DepthAndPlayerIndex kinectNui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.DepthAndPlayerIndex);

10. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs, InitializeNui{}  Init timestamp for frame rate calculation: lastTime = DateTime.Now;  Add event handler to handle incoming frames kinectNui.VideoFrameReady += new EventHandler (NuiVideoFrameReady); kinectNui.DepthFrameReady += new EventHandler (NuiDepthFrameReady);

11. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs  Video frame ready event handler void NuiVideoFrameReady(object sender, ImageFrameReadyEventArgs e) { PlanarImage Image = e.ImageFrame.Image; image.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel); imageCmyk32.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Cmyk32, null, Image.Bits, Image.Width * Image.BytesPerPixel); }

12. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: Depth frame ready event handler  Depth is different because the image you are getting back is 16-bit and we need to convert it to 32-bit void NuiDepthFrameReady(object sender, ImageFrameReadyEventArgs e) { var Image = e.ImageFrame.Image; var convertedDepthFrame = convertDepthFrame(Image.Bits); depth.Source = BitmapSource.Create( Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, convertedDepthFrame, Image.Width * 4); CalculateFps(); }

13. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: convert depth frame  Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame that displays different players in different colors byte[] convertDepthFrame(byte[] depthFrame16) { for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4) { int player = depthFrame16[i16] & 0x07; int realDepth = (depthFrame16[i16 + 1] > 3); // transform 13-bit depth information into an 8-bit intensity appropriate // for display (we disregard information in most significant bit) byte intensity = (byte)(255 - (255 * realDepth / 0x0fff)); depthFrame32[i32 + RED_IDX] = intensity; depthFrame32[i32 + BLUE_IDX] = intensity; depthFrame32[i32 + GREEN_IDX] = intensity; } return depthFrame32;}

14. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: calculate frame rate void CalculateFps() { ++totalFrames; var cur = DateTime.Now; if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1)) { int frameDiff = totalFrames - lastFrames; lastFrames = totalFrames; lastTime = cur; frameRate.Text = frameDiff.ToString() + " fps"; }

15. Kinect Getting Started Basic http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx MainWindow.xaml.cs: windows loading/closing private void WindowLoaded(object sender, RoutedEventArgs e) { InitializeNui(); } private void WindowClosed(object sender, EventArgs e) { kinectNui.Uninitialize(); } Time to run the app!