V ISION -B ASED T RACKING OF A M OVING O BJECT BY A 2 DOF H ELICOPTER M ODEL : T HE S IMULATION Chayatat Ratanasawanya October 30, 2009

O VERVIEW Classification of visual-servo systems Components of the simulation The non-linear model of the helicopter The LQR controller Perspective projection model & camera calibration Coordinate systems Determination of the position of the ball The simulation: logic and result Summary Questions/comments

V ISUAL - SERVO SYSTEMS TAXONOMY In 1980, Sanderson and Weiss introduced a taxonomy of visual servo systems. Two questions: 1. Is the control structure hierarchical, with the vision system providing set-points as input to the robot’s joint-level controller, or does the visual controller directly compute the joint-level inputs? 2. Is the error signal defined in 3D (task space) coordinates or directly in terms of image features?

4 SYSTEM STRUCTURES Dynamic position-based look-and-move structure Dynamic image-based look-and-move structure

4 SYSTEM STRUCTURES Position-based visual servo structure Image-based visual servo structure

T HE IMAGE JACOBIAN Used in image-based control The matrix describes how image feature parameters change with respect to changing manipulator pose.

T HE SIMULATION - AN INTRODUCTION The system being simulated can be categorized as a dynamic position-based look-and-move system. The non-linear model of the helicopter and the joint- level LQR controller (implemented by Quanser). Perspective projection model & camera calibration. Coordinate systems Determination of the ball’s position in world frame & in camera frame.

T HE NON - LINEAR MODEL OF THE HELICOPTER A block in Simulink provided by Quanser, which captures the dynamic equations of the helicopter plant.

T HE LQR CONTROLLER A controller design technique that works with the state-space representation of a plant/system. with weighting matrices Q and R, calculate Has the same action as a PD or a PID controller. In the simulation, it is a joint-level controller.

P ERSPECTIVE PROJECTION MODEL & CAMERA CALIBRATION The projection model is used to relate the position of an object in the camera frame to the pixel coordinate of the image of that object on the image plane.

C OORDINATE SYSTEMS The world frame: stationary frame attached to the pivot point. The helicopter frame: attached to the helicopter at the pivot point. It moves with the helicopter. The camera frame: attached to the camera at the center of projection. x y z x y z x y z

D ETERMINATION OF THE BALL ’ S POSITION : THE SCENARIO Initially a ping-pong ball is right in front of the camera. r1r1 r2r2 r3r3

D ETERMINATION OF THE BALL ’ S POSITION : THE SCENARIO The ball is moved to a new position. The helicopter hasn’t moved yet. r1r1 r 2n r 3n

D ETERMINATION OF THE BALL ’ S POSITION : THE SCENARIO The helicopter moves to the new position to align the ball to the camera. r 3n r1r1 r 2ss

D ETERMINATION OF THE BALL ’ S POSITION : RECAP The ball is initially right in front of the camera. We know the pose of the helicopter (  0 and  0 ). The ball is moved. Get new ball position in the camera frame from inverse projection model. Use the current pose to calculate new ball position in the world frame. Use ball position in the world frame from the previous step to calculate the desired pose (  d and  d ). Pass these values to the LQR controller.

P UTTING IT ALL TOGETHER : THE SIMULATION

I MPLEMENTATION The first step towards implementation has been taken; i.e. locating the ball’s centre of gravity in real time.

SUMMARY Visual-servo systems taxonomy Components of the simulation The non-linear dynamic model The controller Projection model Coordinate systems Locating the ball The simulation First step towards implementation

Q UESTIONS / COMMENTS 7 th annual UVS Canada conference 2009 Victoria, BC. November 2-5, 2009