1. EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 4
Wenbing Zhao
4/15/2019
EEC492/693/793 - iPhone Application Development 1 1

2. Outline Working with Kinect color stream Types of color images
Retrieving color images: event & polling models
Steps on handling of color images
Event handler: C# delegate
Build the KinectCam app 2 2

3. Working with Kinect Streams
Kinect SDK supports two image stream formats
Color image stream
Depth image stream
Both are children of ImageStream class
The image frames are stored into a buffer for use by the application. If there is any delay in reading the buffer data and rendering it as images, the buffer will be overwritten => old frame lost
Main steps in handling image streams
Enabling the stream
Capturing the stream frame by frame
Processing the image frames 3

4. Types of Color Images RGB: read-green-blue color space (RGB) YUV:
Each pixel is an array of four: Blue, Green, Red, Alpha
Alpha: transparency degree
32bits per pixel at 640x480 at 30FPS or 1280x960 at 12FPS
YUV:
Y: luminance channel; U: blue channel, V: red channel
16 bits per pixel at 640x480 at 15 FPS
Uses less memory than RGB
Bayer: raw Bayer color image format with a Bayer color filter array (or Bayer filter):
50% green, 25% red, 25% blue
Resolution at 640x480 at 30 FPS or 1280x960 at 12 FPS
Infrared data: 16 bits per pixel at 640x480 at 30 FPS

5. Retrieving Color Image Stream
Two ways: event model and polling model
Event model
Kinect sends the frame to the app whenever a new frame is captured by the sensor
Need to register a event handler for the event

6. Polling Model
The application sends a request to the sensor whenever there is a need to get an image frame
Pass the time interval after which the sensor will return the image frame

7. Capturing Color Images
Main steps
Starting Kinect sensor
Enable color stream channel with desirable image format
Subscribe (register) an event handler (a method) with the stream channel
Process the image frame
Render the image frame on the user interface

8. Enable Color Stream Channel
In class ColorImageStream, two methods
public void Enable();
By default, use RgbResolution640x480Fps30
public void Enable(ColorImageFormat format);
RgbResolution640x480Fps30
RgbResolution1280x960Fps12
YuvResolution640x480Fps15
RawYuvResolution640x480Fps15
InfraredResolution640x480Fps30
RawBayerResolution640x480Fps30
RawBayerResolution1280x960Fps12
Undefined
You can enable only one type of color stream at a time

9. Registering an Event Handler
this.sensor.ColorFrameReady += colorFrameReady;
What is colorFrameReady?!
It is the event handler that you will implement
You can use any method name as you wish
The method must have the following signature: void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
Sender: the object that fires the event
e: the object that holds data to be retrieved (i.e., image frame)

10. Event Handler Event handler is similar to a function pointer in C/C++
In C# (and objective C), the event handler is a delegate with pedefined signature
General format: public delegate void MyEventHandler(object sender, MyEventArgs e);
Sender: files the event
e: holds the data to be handled
A delegate allows you to pass methods of one class to objects of other classes that can call those methods

11. Delegate in C# (http://www. developerfusion
using System; 
// Step 1. Declare a delegate with the signature of the encapsulated method 
public delegate void MyDelegate(string input); 
//Step 2. Define methods that match with signature of delegate declaration 
class MyClass1{ 
public void delegateMethod1(string input){         Console.WriteLine("delegateMethod1 and input to the method is {0}",input);     } 
   public void delegateMethod2(string input){         Console.WriteLine("delegateMethod2 and input to the method is {0}",input);     }  }  }

12. Delegate in C# (http://www. developerfusion
//Step 3. Create delegate object and plug in the methods  class MyClass2 {     public MyDelegate createDelegate(){         MyClass1 c2=new MyClass1();         MyDelegate d1 = new MyDelegate(c2.delegateMethod1);         MyDelegate d2 = new MyDelegate(c2.delegateMethod2);         MyDelegate d3 = d1 + d2;         return d3;     }  } 

13. Delegate in C# (http://www. developerfusion
//Step 4. Call the encapsulated methods through the delegate  class MyClass3{     public void callDelegate(MyDelegate d, string input){         d(input);     }  }  class Driver{     static void Main(string[] args){         MyClass2 c2 = new MyClass2();         MyDelegate d = c2.createDelegate();         MyClass3 c3 = new MyClass3();         c3.callDelegate(d,"Calling the delegate");     }  }

14. Event Handler Delegate for ColorImage
namespace System {
public delegate void EventHandler<TEventArgs> (object sender, TEventArgs e); }
In Kinect.Sensor class
That is why we can this:
public event EventHandler<ColorImageFrameReadyEventArgs> ColorFrameReady;
this.sensor.ColorFrameReady += colorFrameReady;

15. Retrieve the Color Image Frame
void colorFrameReady(object sender,
ColorImageFrameReadyEventArgs e) {
using (ColorImageFrame imageFrame = e.OpenColorImageFrame())
……. }
What is using(a new object) { }?
It defines the scope of the new object
Once the last statement in the {} block is executed, the object is garbage deleted

16. Handling/Displaying Color Image
// copy image to a byte array
imageFrame.CopyPixelDataTo(colorPixels);
// width in bytes of a single row of pixel data including padding.
int stride = imageFrame.Width * imageFrame.BytesPerPixel;
// Write the pixel data into our bitmap
this.colorBitmap.WritePixels(
new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
this.colorPixels,
stride,
0);
// Don’t do this for stride calculation
// If format is changed, it may not work!
this.colorBitmap.PixelWidth * sizeof(int);

17. Color Image Handling
public void WritePixels(Int32Rect srcRect, Array pixels, int stride, int offset);
Updates the pixels in the specified region of the bitmap.
srcRect: The rectangle of the System.Windows.Media.Imaging.WriteableBitmap to update
pixels: The pixel array used to update the bitmap.
stride: The stride of the update region in pixels.
offset: The input buffer offset

18. WriteableBitmap Class
Inherits from BitmapSource
Constructor
pixelWidth:width of the bitmap
pixelHeight: height of the bitmap
dpiX: horizontal dots per inch (dpi)
dpiY: vertical dots per inch
pixelFormat: pixel format
palette: finite set of colors
public WriteableBitmap(int pixelWidth, int pixelHeight, double dpiX, double dpiY, PixelFormat pixelFormat, BitmapPalette palette);

19. System.Windows.Media.PixelFormats
Bgr32 property: blue, green, red channels (3 bytes), last byte is set to 0
Bgra32
Bgr24
Gray2
Gray16
Etc.

20. Build KinectCam App Create a new WFP project named KinectCam
Draw the GUI for the app
Modify xaml file to add WindowLoaded and WindowClosing as before
Add member variables for displaying color images
Start Kinect sensor, enable color stream, register color stream handler, connect image control to the bitmap for display
Process each color image

21. Design GUI
Add label title
Add image control

22. Add Member Variables KinectSensor sensor;
// Bitmap that will hold color information
private WriteableBitmap colorBitmap;
// Intermediate storage for the color data received from the camera
private byte[] colorPixels;

23. Initialization (in WindowLoaded())
if (KinectSensor.KinectSensors.Count > 0) {
this.sensor = KinectSensor.KinectSensors[0];
if (this.sensor != null && !this.sensor.IsRunning) {
this.sensor.ColorStream.Enable();
this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength];
this.colorBitmap = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);
this.image1.Source = this.colorBitmap;
this.sensor.ColorFrameReady += this.colorFrameReady;
this.sensor.Start();
} else {
MessageBox.Show("No device is connected!");
this.Close(); }

24. Handling/Displaying Color Images
void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) {
using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) {
if (null == imageFrame)
return;
imageFrame.CopyPixelDataTo(colorPixels);
int stride = imageFrame.Width * imageFrame.BytesPerPixel;
// Write the pixel data into our bitmap
this.colorBitmap.WritePixels(
new Int32Rect(0, 0, this.colorBitmap.PixelWidth, this.colorBitmap.PixelHeight),
this.colorPixels, stride, 0); }