Verification of specifications and aptitude for short-range applications of the Kinect v2 depth sensor Cecilia Chen, Cornell University Lewis’ Educational and Research Collaborative Internship Project (LERCIP) NASA Glenn Research Center, Graphics & Visualization/Dr. Herb Schilling 7/22/2014

Purpose of this study Validate the published specifications for the Microsoft Kinect v2 depth sensor Resolution and x-y position accuracy Depth-sensing accuracy Near-range sensing limit Determine the sensor’s potential for use in short-range applications Feasibility of repurposing the Kinect for functions requiring an operating distance of approximately 0.5 m from the sensor

Overview of topics discussed Introduction Microsoft Kinect v2 Infrared and Depth streams Time-of-flight Preliminary Calculations Is the depth camera’s error range acceptable? Possibility of errors in position due to low resolution Possibility of errors in angle due to noise

Overview of topics discussed Calibration and Experimental Verification Software preparation Calibration Experimental verification Conclusions Findings Sponsors

Introduction

Microsoft Kinect v2 Primarily used for gaming and natural user interface Color, infrared, and depth streams 512 × 424 depth resolution 0.5–4.5 m depth sensing range 30 frames per second

Infrared and Depth streams

Time-of-flight TOF is a form of LIDAR Software calculates the distance between a point and the sensor based on round-trip time and speed of light (c=3× 10 8 𝑚/𝑠) Emitter sends pulses of infrared light Detector senses returning light

Preliminary Calculations

Is the depth camera’s error range acceptable? Microsoft claims depth measurements are accurate to within 1 mm Guidelines for short-range use of the depth sensor Defined by an operating distance (between the Kinect and object) of approximately 0.5 m Near-range of the Kinect v2 supposedly starts at 0.5 m Given a surface oriented perpendicular to the depth axis: Position – be able to locate a point on the surface to within 2 mm Angle – be able to measure inclinations on the surface to within 10° Both to be confirmed through calculations using an upper error bound

Possibility of errors in position due to low resolution 512 × 424 depth resolution Field of view: 70° horizontal 60° vertical y z x

Possibility of errors in position due to low resolution Assuming a distance of 0.7 m between the surface and Kinect: tan 35° = 𝑥/2 0.7 𝑚 𝑥≈0.980 𝑚 35° x/2 0.7 m Horizontal slice of FOV

Possibility of errors in position due to low resolution Assuming a distance of 0.7 m between the surface and Kinect: tan 30° = 𝑦/2 0.7 𝑚 𝑦≈0.808 𝑚 y/2 0.7 m 30° Vertical slice of FOV

Possibility of errors in position due to low resolution Assuming a distance of 0.7 m between the surface and Kinect: 𝐴𝑟𝑒𝑎 𝑠𝑒𝑒𝑛 𝑏𝑦 𝐾𝑖𝑛𝑒𝑐𝑡=𝑥𝑦 𝐴𝑟𝑒𝑎= 0.980 𝑚 0.808 𝑚 =0.792 𝑚 2 Pixel density: 512×424=217,088 𝑝𝑖𝑥𝑒𝑙𝑠 217,088 𝑝𝑖𝑥𝑒𝑙𝑠 0.792 𝑚 2 ≈274,101 𝑝𝑖𝑥𝑒𝑙𝑠/ 𝑚 2

Possibility of errors in position due to low resolution Given a circular section of the surface 50 mm in diameter: 𝐴=𝜋 𝑟 2 𝐴=𝜋∙( 0.025 2 ) 𝑚 2 𝐴=𝜋∙0.000625 𝑚 2 =0.00196 𝑚 2 𝑃𝑖𝑥𝑒𝑙𝑠 𝑎𝑐𝑟𝑜𝑠𝑠 𝑠𝑢𝑟𝑓𝑎𝑐𝑒=𝑝𝑖𝑥𝑒𝑙 𝑑𝑒𝑛𝑠𝑖𝑡𝑦×𝑎𝑟𝑒𝑎 274,101 𝑝𝑖𝑥𝑒𝑙𝑠 𝑚 2 ×0.00196 𝑚 2 ≈538 𝑝𝑖𝑥𝑒𝑙𝑠 This works out to a ratio of 3.65 square millimeters (3.65≈ 1.9 2 ) per pixel. 50 mm diameter

Possibility of errors in angle due to noise For simplicity, take a plane defined by two points 50 mm apart representing the same circular surface: 50 mm diameter x y z Axis orientation

Possibility of errors in angle due to noise z x y 50 mm Kinect θ Δz

Possibility of errors in angle due to noise How large can Δz get before θ falls outside the allowed angle range? tan 𝜃 = ∆𝑧 50 𝑚𝑚 Let θ = 10°: ∆𝑧≤8.8 𝑚𝑚 Δz 50 mm θ

Calibration and Experimental Verification

Software preparation Use C# to interact with Microsoft Visual Studio and the Kinect for Windows SDK Modify and expand the Depth Basics code included in the SDK (Preview 1404) Decreasing the grayscale gradient range for easier visual distinguishability between depths Making the window recognize a hovering cursor Writing the (x, y, z) coordinates of the cursor to the image window in real-time Averaging depth readings at a given point to smooth out jumpy data Capturing depth arrays of multiple frames Outputting collected data to CSV files for further analysis

Before After

Calibration Step 0: Design and assemble a rig to hold the Kinect steady Step 1: Level the sensor Step 2: Set up a base surface for measurements at a height similar to the intended distance between the Kinect and interaction area

Calibration Step 3: Check for irregularities

Experimental verification Step 4: Measure a sloped calibration block of known height 50 mm

Base, z1 Top of block, z2 Height, Δz Actual (mm) -- 50 Measured (mm) 571 0 (538 minimum) Error

Experimental verification Step 5: Lower the base height Step 6: Measure a sloped calibration block of known height 50 mm

Base, z1 Top of block, z2 Height, Δz Actual (mm) -- 50 Measured (mm) 712 662

Experimental verification Step 7: Repeat with calibration block(s) of different heights 64 mm

Base, z1 Top of block, z2 Height, Δz Actual (mm) -- 64 Measured (mm) 712 647 65

Conclusions

Findings Successful verification of the published specs Kinect v2 appears to be a promising depth sensor for short-range applications Expectation or Guideline Experimental confirmation Acceptable? Position (resolution) ±2 mm (1.9 mm)2 per pixel resolution Yes Angle (noise) ≤10° Requires Δz ≤ 8.8 mm at z = 0.7 m Very Depth accuracy ±1 mm Near-range 0.5 m 0.538 m No