1. Win8 on Intel Programming Course Desktop : Perceptual Computing Cédric Andreolli www.Intel-Software-Academic-Program.com paul.guermonprez@intel.com Intel Software 2013-03-20

2. Agenda

3. Introduction to Perceptual Computing What is the Perceptual Computing? The Intel Perceptual Computing SDK What can the SDK do? What is it for? Set up the programming environment Setting up the Visual Studio C++ environment will be explained here Using the SDK Some of the SDK functionalities will be explained here

4. Introduction to Perceptual Computing

5. Introduction What is Perceptual Computing? Perceptual computing offers a new way to interact with your computer What do I need? It works with: A camera created by Creative* A SDK created by Intel*

6. Introduction The Creative camera 3D depth sensor 720p camera Dual-array microphone

7. Introduction The Intel* Perceptual SDK Provides a lot of services that use the camera as input Gesture recognition Skeleton tracking Face recognition Voice recognition Voice synthesis Image capture The camera associated with the SDK provides a new way to interact with your Ultrabook

8. The Intel* Perceptual SDK

9. Architecture The SDK provides libraries and header files that you must add to your projects It can be used in C++ but also in C#

10. The Intel* Perceptual SDK High and low levels programming The SDK is flexible You can use Utility classes that provide an easy way to access gesture recognition and high level process You can also work directly on the raw images and implement your own algorithms More information You can download the SDK for free http://software.intel.com/en- us/vcsource/tools/perceptual-computing-sdk

11. The Intel* Perceptual SDK Starting with the SDK Download the SDK and install it Be sure to remember the installation directory

12. The Intel* Perceptual SDK SDK directories The bin folder contains a lot of examples that give you an idea of what you can do with the SDK The doc folder contains the documentation in PDF format The include folder contains the main headers The lib folder contains the SDK library libpcx.lib The sample folder contains the source code of the programs in the bin folder

13. The Intel* Perceptual SDK The Sample folder in details This folder also contains important tools in the common folder

14. The Intel* Perceptual SDK The Sample/common folder in details This folder contains Important headers in the include directory Important library in the lib directory

15. The Intel* Perceptual SDK The Sample/common folder in details In the lib folder, you can find 2 libraries libpxcutils.lib for the release libpxcutils_d.lib for the debug Headers can be found in the $(SDK)/include directory but also in the $(SDK)/sample/common/include. Libraries can be found in the $(SDK)/lib directory But also in the $(SDK)/sample/common/lib.

16. Set up the environment

17. Using the SDK in C++ The SDK has been programmed in C++ This is an efficient language Using the SDK in C++ Open Visual Studio and create a C++ win32 console application project

18. Set up the environment What do we have to do? Visual Studio’s compiler must know where to find Header files Library directories Libraries to link Open your project properties Right click on your project’s name and select Properties In C/C++ -> Code Generation -> Runtime Library Select /MTd (we are working in Debug mode)

19. Set up the environment Change the Runtime Library

20. Set up the environment Add include directories The SDK installation added some macro to your Visual Studio environment In your project properties, on the left, select VC++ Directories In Include Directories, select and add $(PCSDK_DIR)/include $(PCSDK_DIR)/sample/common/include

21. Set up the environment Add include directories Setting up the include Directories allows Visual Studio to find the headers

22. Set up the environment Add library directories We also need to indicate Visual Studio where to find the libraries In your project properties, on the left, select VC++ Directories In Library Directories, select and add $(PCSDK_DIR)/lib/$(Platform) $(PCSDK_DIR)/sample/common/lib/$(PlatformName)/$(PlatformToolset)

23. Set up the environment Add library directories Setting up the library Directories allows Visual Studio to look for the libraries in the good directories

24. Set up the environment Link the libraries Setting up the library directories is not enough, we must also tell the compiler which are the libraries that our project will use In the project properties, on the left, select Linker, then Input In the field Additional Dependencies, click on

25. Set up the environment Link the libraries Add the following libraries: libpxc_d.lib libpxcutils_d.lib We are working in Debug mode, if you work in release mode, remove the “_d”

26. Set up the environment Link the libraries Linking the libraries tells the compiler which are The libraries that contain the implementation of the Functions that you are using

27. Using the SDK - HelloWorld

28. Programming a Hello World Programming an application that displays the images acquired by the camera is really simple with the SDK You only need 1 C++ file and few lines of code

29. Using the SDK - HelloWorld 1- Add the includes 2- Request access to the depth image

30. Using the SDK - HelloWorld 3- Create the main loop

31. Using the SDK - HelloWorld Full code

32. Using the SDK - HelloWorld See the result You can run the program, you will see windows displaying the depth map Using the SDK to display an image that comes from the Camera is really simple. You can also try to retrieve the color image.

33. Using the SDK - Gestures

34. Create your own class A good practice when using the SDK is to create your own class that inherits from the UtilPipeline Create and set a new project named SDKTest (win32 application) Create also a new class (PipelineTest) that inherits from UtilPipeline Right click on the project’s name and select Add and Class

35. Using the SDK - Gestures Create your own class In the popup, click on Add Set the name for your class and click on Finish

36. Using the SDK - Gestures Inherit from UtilPipeline Open the header (PipelineTest.h) and change the content to

37. Using the SDK - Gestures Step 1 - The OnGesture method When a gesture is recognized by the SDK, the method OnGesture is called

38. Using the SDK - Gestures Step 2 - The OnAlert method The OnAlert method is called when an unusual event append (your hands go out of the camera range, etc.)

39. Using the SDK - Gestures Step 3 - The OnNewFrame method The OnNewFrame method is called when the SDK is able to provide a new frame

40. Using the SDK - Gestures Bonus – Add a function to debug later This example will also be used to work on skeleton tracking so add a function to display logs

41. Using the SDK - Gestures It’s time to code! Go in the.cpp file and enable the gesture pipeline in the constructor This step is mandatory when you want to use the gesture recognition but also when you want to track the skeleton nodes

42. Using the SDK - Gestures Implement the OnGesture The SDK informs your application when a gesture is identified by calling the function OnGesture The variable data provides information that let you know which gesture has been recognized

43. Using the SDK - Gestures The full OnGesture function

44. Using the SDK - Gestures The full OnAlert function

45. Using the SDK - Gestures Work with nodes Using the skeleton nodes can be intereseting to implement new controls in your application To print out the nodes positions, fill the printNodes function Here is the Full code

46. Using the SDK - Gestures Full code explained

47. Using the SDK - Gestures The label definitions More information You can get more information in the documentation

48. Using the SDK - Gestures Gestures and alerts As you can see, manipulating the gestures or the alerts is done in a similar way. You can let the SDK inform you when such an event has been detected.

49. Using the SDK - Gestures Lot of features The UtilPipeline class provides 4 main features in realtion with gestures recognition, voice recognition, face recognition and access to raw images

50. Using the SDK - Gestures Access raw images Manipulating raw images allows you to implement your own algorithms The UtilPipeline allows you to access the raw images through the UtilPipelineRaw class Create a new project Configure a new C++ win32 console application that works with the SDK Name it « RawImages »

51. Using the SDK - Gestures New project creation

52. Using the SDK - Gestures Inherit from UtilPipeline The UtilPipeline inherits from UtilPipelineRaw so everything needed to manipulate raw images is also included in the UtilPipeline class

53. Using the SDK - Gestures OnNewFrame implementation

54. Using the SDK - Gestures How to get image pixels? From the previous code, you can directly manipulate the images

55. Using the SDK - Gestures Working with planes The planes hold different kind of informations For the depth map plane[0] holds the depth map plane[1] holds the confidence map plane[2] holds the UV map

56. Using the SDK - Gestures Write your filters Manipulating raw images allows to write your own filters The lab will show you how to retrieve the first 20 centimeters starting from the closest object in front of the camera Using the raw images provides a full control on the events you want to work on.

57. Using the SDK - Gestures Try the lab A lab is available with this course! You will learn to use the Perceptual SDK in C++ C# Unity You will also learn how to create a simple filter and how to control a character in a 3D game

59. License Creative Commons – By 3.0 You are free: to Share — to copy, distribute and transmit the work to Remix — to adapt the work to make commercial use of the work Under the following conditions: Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). With the understanding that: Waiver — Any of the above conditions can be waived if you get permission from the copyright holder. Public Domain — Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license. Other Rights — In no way are any of the following rights affected by the license: – Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations; – The author's moral rights; – Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights. Notice — For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page. http://creativecommons.org/licenses/by/3.0/