Presentation is loading. Please wait.

Presentation is loading. Please wait.

Senior Design Group 16 Client / Adviser: Dr. Tom Daniels Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer.

Similar presentations


Presentation on theme: "Senior Design Group 16 Client / Adviser: Dr. Tom Daniels Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer."— Presentation transcript:

1 Senior Design Group 16 Client / Adviser: Dr. Tom Daniels Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer

2  Freshmen computer engineering college students and high school students need an interesting way to be introduced to programming and problem solving.  In the past, Iowa State students have been taught to create simple programs in the C programming language that use Nintendo Wiimotes as input devices.  Software needs to be developed similar to Wii Wrap, but uses the Microsoft Kinect™ sensor as input to student-written programs.

3  RGB Webcam Camera 30 Hz (center)  IR Projector Projects Grid for Depth Mapping  IR Camera at 30 Hz collects depth data (right)  Microphones give room audio  Small Panning motor turns sensor bar from side to side  Onboard image preprocessing  Projector power draw requires extra AC power

4  Wii Remote / Wiimote  11 buttons + Power  Roll/Pitch/Yaw Gyro  Infrared Sensor  Speaker (unsupported)  Optional Nunchuck

5

6  Three Modules: 1.Hardware interface to read in raw data (provided) 2.Interface simplification / data processing 3.Graphical interface to display output

7

8  Issue: Decrease delay in printf output  Solution: Added a fflush(stdout) statement immediately after printf  Issue: IR data becomes erratic if sensor loses source for one frame of data  Solution: Save last good frame of IR data (x,y) and loop the last good data until new data arrives

9  OpenKinect ◦ Tricky to get started with in windows ◦ Natively supports Unix ◦ Body tracking not built-in ◦ Needs a calibration pose for body tracking ◦ No event when new Video or Depth frame is available  Microsoft Kinect SDK ◦ Sample Projects ◦ Documentation ◦ Simple installer ◦ Natively supports Windows ◦ Full body tracking built-in

10  Data Available ◦ RGB Video Frames (640x480 max resolution) ◦ Depth (mm) Data Frames (320x240 max resolution) ◦ Skeletal tracking points (20 points per player)  How we get the data ◦ Pass an event object to alert us when new data is available ◦ Pass a video/depth/skeleton frame pointer to read in next frame ◦ Run event loop and process each frame as RGB/Depth/Skeleton data

11

12  Windows File Mapping is an inter-process communication tool that maps a block of memory by name  Shared Memory in Kinect Wrap ◦ File Mapping is set up by Kinect Wrap backend program with a specific name for each block ◦ Backend gets data from Kinect SDK and dumps it into shared memory space when each new frame becomes available ◦ Our API Library copies frame from shared memory into local memory space for student program to use.

13

14  Provides an easy-to-use interface for students while maximizing functionality  Functions: ◦ getVideoFrame(rgb** pFrame) ◦ getDepthFrame(short** pFrame) ◦ getSkeleton(Point3D* pSkel) ◦ setOutputDisplay(rgb** pFrame) ◦ setOutputText(char* sText) ◦ setPixel(rgb** pFrame, int x, int y, KCOLOR color)

15

16  Kinect Wrap Library is a static Windows library that students must include their C projects  Library handles backend processing of API  Abstracts all of the technical workings of the Kinect SDK, File Mapping API, and GUI design

17

18  Abstracts away the details of drawing Kinect data via a graphical API  Kinect GUI receives data via shared memory  Data is drawn as a grid of vertices, each with a custom RGB value via DirectX  Intend to port GUI code to OpenGL and work on constructing a more intuitive GUI in future

19

20  Field Testing ◦ Compared time delay of data transmission between the current and previous iterations of the Wii Wrap code. ◦ Found the apparent delay seen in the Wii Wrap GUI to be significantly decreased. ◦ Test results corroborated by CPRE 185 lab TAs during classroom use. ◦ No previous functionality was affected by these changes.

21  Assumed that the Kinect SDK functions deliver correct data from Kinect sensor  Shared Memory ◦ Wrote known data into shared memory space with one program and had second program read it out and verify that the data has not been altered  Student API Interface ◦ Verified that buffers of known data sent to the student program are not altered before being read ◦ Verified that a known function run by student code produces the expected output frame data given a known input frame

22  GUI Output ◦ Tested whether frames were being updated consistently by writing specific colors to known pixels in a given output frame

23  setPixel() ◦ Tested for boundary values. ◦ Attempted to set pixels outside the frame (640x480) ◦ Attempted to set pixel RGB values outside of  getVideoFrame() & setOutputDisplay() ◦ Check that output frame RGB data matches input data ◦ Found that with simple student processing code, any reduction in output fps from device frame rate is minimal  getDepthFrame() ◦ Used a plane at a known distance from the Kinect to find how accurate the depth values in mm are vs. reality.

24 int main() { KinectLib_Init(); // Initialize the library rgb* frame[VIDEO_HEIGHT];// Create 2D frame of rgb data short* dframe[DEPTH_HEIGHT];// Create 2D frame of short depth data Point3D* skeleton;// Create array of skeleton data while(1){ getVideoFrame(frame);// Map video frame to pointers getDepthFrame(dframe);// Map depth frame to pointers if(skeletonTracked()) { skeleton = getSkeleton();// Map Skeleton data } setPixel(frame, 23, 18, BLUE);// Arbitrarily set pixel (23, 18) to Blue displayVideoFrame(frame);// Display the modified video frame in the GUI } KinectLib_Destroy();// Destroy Kinect library instance return 1; }

25  Multi-Kinect Support  More comprehensive GUI program  Additional Library Functionality  Create code to share Kinect data over Ethernet

26 seniord.ece.iastate.edu/dec1116/


Download ppt "Senior Design Group 16 Client / Adviser: Dr. Tom Daniels Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer."

Similar presentations


Ads by Google