Java and the Kinect FRC 2012 Kickoff Team 1279 ColdFusion January 7, 2012

Presentation Overview Kinect with Java focused –C/C++ looks similar FRC focused –Not the full Kinect SDK Quick and Dirty –Practical Example, no Bells or Whistles Simple End-to-End Application –Driver, WPILibJ, Application

Java Application Sample Application –Xylophone1279 Demo Code walkthrough –Resources: Source & Presentation – Contact Info

Java Application Beta KinectStick Class –Y-axis Only Standard range (-128 to 127) –Wrist – Shoulder Angle Solves Some Issues Restrictive Modified KinectStick Class –X, Y, and Z axes Standard range (-128 to 127) –Skeleton Joint Values

Java Application import edu.wpi.first.wpilibj.KinectStick; public class Player implements XyloConstants { public KinectStick rStick, lStick; public Player(){ rStick = new KinectStick(2); lStick = new KinectStick(1); }

Java Application void playKinectStick(){ double lAngle, rAngle; while (ds.isEnabled() && ds.isOperatorControl()){ lAngle = 0.0; rAngle = 0.0; lAngle += ( lStick.getY() * 40.0); rAngle += ( rStick.getY() * 40.0); … }

Java Application Conceptualization –Analog & Real-time Map Movements to Actions –Natural –Full Analog Spectrum –Digital & Parsing Encode Commands –Un-natural –Precise

Java Application void playKinectRaw(){ double rX = 0, rY = 0, lX = 0, lY = 0; while (ds.isEnabled() && ds.isOperatorControl()){ rX = rStick.getX(); if (rX <= -0.2) rElbow.setAngle(RNOTES[0]); else if (rX > && rX <= -0.05) rElbow.setAngle(RNOTES[5]); … else if (rX > 0.70 && rX <= 0.85) rElbow.setAngle(RNOTES[11]); else if (rX > 0.85) rElbow.setAngle(RNOTES[12]); if (rStick.getY() < 0.25) rWrist.setAngle(RWRIST_DOWN); Timer.delay(STRIKE); rWrist.setAngle(RWRIST_UP); Timer.delay(0.25); }

Java Application void playKinectNL(){ while (ds.isEnabled() && ds.isOperatorControl()){ rX = rStick.getX(); rY = rStick.getY(); lX = lStick.getX(); lY = lStick.getY(); if (rX <= 0) rxIndex = 0; else rxIndex = 1; index = (rxIndex << 3) | (ryIndex << 2) | (lxIndex << 1) | lyIndex;//merge into 0-15 switch(index){ case 0://0000 r quadrant 3, l quadrant 3 rElbow.setAngle(RNOTES[RSAFE]); lElbow.setAngle(LNOTES[C0]); play = true; break; … default://unknown symbol, don't play anything lElbow.setAngle(LNOTES[LSAFE]); rElbow.setAngle(RNOTES[RSAFE]); play = false; } if (play && (index < 6)) lWrist.setAngle(LWRIST_DOWN); Timer.delay(STRIKE); lWrist.setAngle(LWRIST_UP); }

Java Application Implementation –Bounds Don’t Map to the Screen –Those Pesky Humans Size Distance Sneezing –Noise Filtering Averaging

WPILibJ Kinect class KinectStick class –Axes X, Y, Z, Throttle, Twist, Custom –Buttons Standard 15 Button 9 reserved –Data Representation Built for sbyte to double Granularity vs. Noise

WPILibJ Modifying Source Code –NetBeans Considerations WPILibJ as a project –Compiling Don’t Clean, Build ALL takes time! –Save Copies! If you break something… Plug-in Updates

WPILibJ byte joystick1[] = new byte[6]; short button1; byte joystick2[] = new byte[6]; short button2; public void read() { backingNativeMemory.getBytes(0, joystick1, 0, 6); button1 = backingNativeMemory.getShort(6); backingNativeMemory.getBytes(8, joystick2, 0, 6); button2 = backingNativeMemory.getShort(14); }

WPILibJ public double getY(Hand hand) { byte tmp; getData(); if (m_id == 1) { tmp = tempOutputData.data.joystick1[1]; return normalize(tmp); } else { tmp = tempOutputData.data.joystick2[1]; return normalize(tmp); } private double normalize(byte rawValue) { if(rawValue >= 0) return rawValue / 127.0; else return rawValue / 128.0; }

FRC Kinect Server Kinect Overview –Depth Capture Skeleton with X, Y, and Z (distance) –Video Capture Built-in Camera, no WPI Kinect Image API (yet?) –Audio Capture Microphone Array, FRC Friendly?

FRC Kinect Server Modifying Source Code –Gesture file Program Files (x86)\FRC Kinect Server\Source\Kinect Server –Writeable Joystick Class Axes Buttons –Compiling and replacing Kill the Service! –Save Copies!

FRC Kinect Server Limitations –Noise Apps ‘jumpy’ –Bandwidth 6 Bytes, and a 16-bit Integer (bit 9 reserved) Full Skeleton X, Y, Z coming –Data Types sbyte

FRC Kinect Server Programming Environment –Visual Studio/Express (free) IDE similar to NetBeans or Eclipse –C# (free) Indistinguishable from C++ (need C#!) Reasonable Learning Curve from Java editions/visual-csharp-expresshttp:// editions/visual-csharp-express –Kinect SDK beta v.2 (free)

FRC Kinect Server x16 = (Int16)(skeleton.Joints[JointID.HandRight].Position.X * 128); if (x16 > 127) x16 = 127; if (x16 < -128) x16 = -128; rightAxis[(uint)WritableJoystick.Axis.X] = (sbyte)x16; rightAxis[(uint)WritableJoystick.Axis.Y] = (sbyte)y16; rightAxis[(uint)WritableJoystick.Axis.Z] = (sbyte)z16; rightAxis[(uint)WritableJoystick.Axis.Throttle] = 0; rightAxis[(uint)WritableJoystick.Axis.Twist] = 0; rightAxis[(uint)WritableJoystick.Axis.Custom] = 0; //Use "Button 9" as Kinect control enabled signal buttons |= (ushort)WritableJoystick.Buttons.Btn8; joy[0].Set(leftAxis, buttons); joy[1].Set(rightAxis, buttons);

Questions? ? Thanks! Best of Luck to All!