1. EEC-693/793 Applied Computer Vision with Depth Cameras
Lecture 5
Wenbing Zhao
11/18/2018
EEC492/693/793 - iPhone Application Development

2. Outline Important Classes on Color Image
Retrieving/Setting Color Image Format
Capturing Frames on Demand
Frame Number
Frame rate
Saving frames
Extend the KinectCam app with more controls 2

3. Important Classes on Color Image
We have dealt with two classes related to Kinect color images
ColorImageStream: handles a stream of ColorImageFrame
ColorImageFrame
ColorImageStream
this.sensor.ColorStream.Enable();
ColorStream is of ColorImageStream type
ColorImageFrame imageFrame = e.OpenColorImageFrame()

4. ColorImageStream
Infrared data: 16 bits per pixel at 640x480 at 30 FPS
Sealed class: no other class can inhert a sealed class

5. ColorImageFrame

6. More on ColorImageFrame

7. Retrieving/Setting Color Image Format
Can retrieve the color image format for the current image frame using the ImageFrame.Format property
public ColorImageFormat ImageFormat {get;set;}
Retrieve format
this.ImageFormat = imageFrame.Format;
Setting format
this.sensor.ColorStream.Enable(RgbResolution640x480Fps30);

8. Capturing Frames on Demand
Polling: request an image frame on demand
ColorImageStream::openNextFrame()
OpenNextFrame() accepts one parameter specifying how long the sensor should wait before it sends the image frame
If there is no frame ready within the provided time, the method will return null
In the polling model, the application opens a channel for the stream, and whenever the application needs a frame, it sends a request to get the frame
int millisecondsWait = 10;
if (this.sensor.ColorStream.IsEnabled) {
ColorImageFrame colorImageFrame = this.sensor.ColorStream.
OpenNextFrame(millisecondsWait); }

9. Frame Number
Frame number: Every frame has a number to identify the frame
This number is incremented with each new frame generated
It is read only in image frame
int frameNumber=imageFrame.FrameNumber;

10. Frame Rate Calculation
private int TotalFrames;
private DateTime lastTime = DateTime.MaxValue;
private int LastFrames;
int currentFrameRate = 0;
private int GetCurrentFrameRate() {
++this.TotalFrames;
DateTime currentTime = DateTime.Now;
var timeSpan = currentTime.Subtract(this.lastTime);
if (this.lastTime == DateTime.MaxValue || // first time to run timeSpan >= TimeSpan.FromSeconds(1)) // average 1 second {
currentFrameRate = (int)Math.Round((this.TotalFrames - this.
LastFrames) / timeSpan.TotalSeconds);
this.LastFrames = this.TotalFrames;
this.lastTime = currentTime; }
return currentFrameRate;

11. Saving Color Image private void SaveImage() {
using (FileStream fs = new FileStream(string.Format("{0}.jpg", Guid.NewGuid().ToString()), System.IO.FileMode.Create)) {
BitmapSource imageSource = (BitmapSource)image1.Source;
JpegBitmapEncoder jpegEncoder = new JpegBitmapEncoder();
jpegEncoder.Frames.Add(BitmapFrame.Create(imageSource));
jpegEncoder.Save(fs);
fs.Close(); }
using System.IO;

12. Saving Color Images Periodically
Call the SaveImage () method on a Tick event of DispatcherTimer
Automatic periodic image saving. 4 steps.
Step1: Define the DispatcherTimer object under the System.Windows.Threading namespace:
private DispatcherTimer timer = new DispatcherTimer();

13. Saving Color Images Periodically
Step 2: Write the start method with an interval of 10 seconds and attach the Tick event handler:
public void StartTimer() {
this.timer.Interval = new TimeSpan(0, 0, 10);
this.timer.Start();
this.timer.Tick += handleTickEvent; }
// handleTickEvent: event handler delegate to be implemented

14. Saving Color Images Periodically
Step 3: On the Tick event handler, call the SaveImage() method:
public void handleTickEvent(object sender, object e) {
if (this.sensor.IsRunning && this.sensor.ColorStream.IsEnabled)
this.SaveImage(); }
Step 4: To launch the timer thread, call timer.StartTimer();

15. Saving Color Images Periodically
To stop timer
public void StopTimer() {
this.timer.Stop();
this.timer.Tick -= this.handleTickEvent; }

16. Adjusting Elevation Angle
Kinect elevation angle can be changed between
MaxElevationAngle (27 degrees)
MinElevationAngle (-27 degrees)
Adjusting angle
private void SetSensorAngle(int angleValue) {
if(angleValue > sensor.MinElevationAngle || angleValue < sensor.MaxElevationAngle)
this.sensor.ElevationAngle = angleValue; }

17. Build KinectCam2.0 Extend the KinectCam we built last time
Revise GUI design
Adding image controls
Add code to handle image controls

18. Revised GUI

19. Settings Changing frame format on the fly
RgbResolution640x480Fps30, enable this by default
RgbResolution1280x960Fps12
Customize the checkbox names and the event handlers this time
private void ChangeToHighResolution(object sender, RoutedEventArgs e) {
if (this.sensor != null) {
this.normalresolution.IsChecked = false;
this.sensor.ColorStream.Enable(
ColorImageFormat.RgbResolution1280x960Fps12);
this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; }

20. Settings Changing frame format on the fly
Enable one, and disable the other
Need to reset the colorPixels array length for different formats
private void ChangeToNormalResolution (object sender, RoutedEventArgs e) {
if (this.sensor != null) {
this.highresolution.IsChecked = false;
this.sensor.ColorStream.Enable(
ColorImageFormat.RgbResolution640x480Fps30);
this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; }

21. Settings Displaying frame rate Changing elevation angle
Calculated and displayed for each frame received
At the end of void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
textBox1.Text = ""+GetCurrentFrameRate();
Changing elevation angle
private void IncrementAngle(object sender, RoutedEventArgs e) {
int angleValue = this.sensor.ElevationAngle + 1;
if (angleValue < sensor.MaxElevationAngle) {
this.sensor.ElevationAngle = angleValue;
this.AngleBox.Text = "" + angleValue; }
private void DecrementAngle(object sender, RoutedEventArgs e) {
int angleValue = this.sensor.ElevationAngle - 1;
if (angleValue > sensor.MinElevationAngle) ……

22. Settings Save image on demand (Save Image button)
private void SaveImage(object sender, RoutedEventArgs e) {
SaveImage(); }
private void SaveImage()
using (FileStream fileStream = new FileStream(string.Format("{0}.jpg", Guid.NewGuid().ToString()), System.IO.FileMode.Create))
BitmapSource imageSource = (BitmapSource)image1.Source;
JpegBitmapEncoder jpegEncoder = new JpegBitmapEncoder();
jpegEncoder.Frames.Add(BitmapFrame.Create(imageSource));
jpegEncoder.Save(fileStream);
fileStream.Close();

23. Settings Save image periodically (via checkbox)
Make sure you designate event handler for both checked and unchecked event!

24. Settings Save image periodically (via checkbox)
private void PeriodicSavingChanged (object sender, RoutedEventArgs e) {
if (checkBox1.IsChecked == true) {
StartTimer();
} else { this.timer.Stop(); } }
private DispatcherTimer timer = new DispatcherTimer();
public void StartTimer() {
this.timer.Interval = new TimeSpan(0, 0, 10);
this.timer.Start(); this.timer.Tick += this.handleTickEvent;
public void StopTimer() {
this.timer.Stop(); this.timer.Tick -= this.handleTickEvent;
public void handleTickEvent(object sender, object e) {
if (this.sensor.IsRunning && this.sensor.ColorStream.IsEnabled) {
this.SaveImage();