1. Microsoft 2016
5/27/2018 9:34 PM
THR3077
Develop for depth camera sensors on Windows 10 Anniversary update – IoT with processing
Andreas Erben
CEO Americas – daenet
CTO – Industrial Holographics
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Developer targeted session
5/27/2018 9:34 PM
Developer targeted session
This session is for you if you like to code
We talk about how (depth) cameras can be accessed in Windows 10, Anniversary Edition
This session is for your if you like technology
We deal with “real” stuff, cameras, but wait, not just cameras, depth cameras. Depth cameras were extremely expensive until only a few years ago.
This session is for you if you like to think different about IoT
While we will not talk about details of IoT implementations, the concept of a “thing” in “Internet of Things” will be used. This includes human beings. 
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Depth Cameras Provide an additional dimension – distance
The following vendors as examples either have support for UWP or are working on support for UWP. This does not attempt to be a complete list.
Provide an additional dimension – distance
Common technologies for depth cameras
Structured Light
Time of Flight
Only affordable since Kinect for Windows V1 and V2

4. Before Windows 10 Anniversary Edition
5/27/2018 9:34 PM
Before Windows 10 Anniversary Edition
Different depth camera different driver
Each camera type device required its own driver.
Different depth camera, different features
Each camera has different technical features
Different depth camera, different API
For each camera a developer would have to use a different set of APIs to develop for that camera
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. With Windows 10 Anniversary Edition
5/27/2018 9:34 PM
With Windows 10 Anniversary Edition
Different Cameras – same API
Different drivers still necessary
Special features of a camera might require special handling
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Depth Cameras Vendors Microsoft: Kinect for Windows V2
The following vendors as examples either have support for UWP or are working on support for UWP. This does not attempt to be a complete list.
Microsoft: Kinect for Windows V2
Intel: RealSense
Orbbec
orbbec3d.com

7. Preview driver for Kinect
Registry
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DriverFlighting\Partner]
"TargetRing"="Drivers"

8. The coding concepts 5/27/2018 9:34 PM
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. It’s all Media MediaFrameSourceGroup
5/27/2018 9:34 PM
It’s all Media
MediaFrameSourceGroup
Get all sources.
Select Sources you care about with MediaSourceKind
MediaSourceKind.Custom, .Color, .Infrared, Depth
Set up MediaCaptureInitializationSettings
To allow multiple apps: use MediaCaptureSharingMode.SharedReadyOnly.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. From SourceInfo to FrameArrivedEvents
SourceInfo.ID
Contains the unique ID for the stream.
Get FrameSource
myMediaCapture.FrameSources.TryGetValue(mySourceInfo.Id, out myFrameSource);
CreateFrameReaderAsync
FrameReader myReader = await myMediaCapture(CreateFrameReaderAsync(myFrameSource);
Wire up Eventhandler to FrameReader
myReader.FrameArrived += FrameReader_FrameArrivedHandler;

11. Let’s start it receiving frames
FrameReader.StartAsync
Look at status to make sure - MediaFrameReaderStartStatus.Success

12. Handle the event MediaFrameReader is the sender, get its frame
myMediaFrameReader.TryAcquireLatestFrame()
VideoMediaFrame or BufferFrame
BufferFrame is for arbitrary (custom) data, such as SkeletalData, choose VideoMediaFrame for rectangular data
For Video – get SoftwareBitmap
For BufferFrame – usually custom decoding necessary

13. Depth to Color
// Create the coordinate mapper used to map depth pixels from depth space to color space.
DepthCorrelatedCoordinateMapper^ coordinateMapper = depthFrame->VideoMediaFrame->DepthMediaFrame->TryCreateCoordinateMapper
(colorFrame->VideoMediaFrame->CameraIntrinsics, colorFrame->CoordinateSystem);
// Unproject depth points to color image.
coordinateMapper->UnprojectPoints(m_colorSpacePoints, colorCoordinateSystem, m_depthSpacePoints);
… after that can use same index for both inputs
UINT index = y * colorWidth + x;
float fadeValue = 1 - max(0, min(((m_depthSpacePoints[index].z - depthFadeStart) / (depthFadeEnd - depthFadeStart)), 1));
outputPixels[index].R = static_cast<byte>(static_cast<float>(outputPixels[index].R) * fadeValue); …

14. Transform coordinate systems
CameraIntrinsics^ colorIntrinsics = colorFrame->VideoMediaFrame->CameraIntrinsics;
SpatialCoordinateSystem^ colorCoordinateSystem = colorFrame->CoordinateSystem;
SpatialCoordinateSystem^ depthCoordinateSystem = depthFrame->CoordinateSystem;
IBox<float4x4>^ boxedDepthToColorTransform
= depthCoordinateSystem->TryGetTransformTo(colorCoordinateSystem)
Point transformedPoint …transform method is from WindowsNumerics.h/.inl
= colorIntrinsics->ProjectOntoFrame(transform(point, depthToColorTransform));
transformedPoint.X *= widthScale;
transformedPoint.Y *= heightScale;

15. Kinect for Windows – BufferFrame
C++ Implementation: BufferView()
To access custom data
C++ Implementation: GetCustomData()

16. Cast to friendly data structures
// {84520B1F-AB61-46DA-AB1D-E01340EF884E}
DEFINE_GUID(PoseSet_BodyTracking, 0x84520b1f, 0xab61, 0x46da, 0xab, 0x1d, 0xe0, 0x13, 0x40, 0xef, 0x88, 0x4e);
// {F142C82C-3A57-4E7D BDBD6CCFE2}
DEFINE_GUID(PoseSet_HandTracking, 0xf142c82c, 0x3a57, 0x4e7d, 0x81, 0x59, 0x98, 0xbd, 0xbd, 0x6c, 0xcf, 0xe2);

17. 5/27/2018
Demo:
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. API demos across the platform + added value
UWP Samples:
CameraFaceTracking
Wires up camera input with Face Detection APIs
HoloGraphicFaceTracking
Even same pattern on HoloLens - Wires up FaceTracking on HoloLens, same pattern to implement
Think beyond: e.g. combine with Cognitive Services

19. Please evaluate this session
5/27/2018 9:34 PM
Please evaluate this session
Your feedback is important to us!
From your PC or Tablet visit MyIgnite at
From your phone download and use the Ignite Mobile App by scanning the QR code above or visiting
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.