National Research Council Canada Conseil national de recherches Canada National Research Council Canada Conseil national de recherches Canada Institute for Information Technology Visual Information Technology Group Institute for Information Technology Visual Information Technology Group Canada Dr. Dmitry O. Gorodnichy Computational Video Group Institute for Information Technology National Research Council Canada Adding safety to autonomous robot manipulation using video-cameras ( Monitoring of robot motions ) ROSA final workshop. April 17, 2002

2. ROSA: Adding safety using video. (Dmitry Gorodnichy) Setup and Driving Desire -There are many video-cameras on the station -14 cameras on ISS -4 cameras on SSRMS -Can they be used to make operations on the station -easier? -safer? -in particular, the manipulation of SSRMS? -autonomous -manual

3. ROSA: Adding safety using video. (Dmitry Gorodnichy) Cameras and what they can see

4. ROSA: Adding safety using video. (Dmitry Gorodnichy) Application #1 1.Automatically detect the malfunctioning of arm joint encoders, by observing the arm with video- cameras. “Compare what is seen with what should be seen” -Requires -Knowledge and agreement of knowledge of both SSRMS kinematics and camera configuration -Somebody to operate the cameras

5. ROSA: Adding safety using video. (Dmitry Gorodnichy) Application #2 2. Help operator to manipulate the arm, by automatically detecting it in the view of video- cameras “Is there SSRMS in the image? If yes, where it is?” -Requires -Images only -SSRMS description This can be a part of any SSRMS monitoring system

6. ROSA: Adding safety using video. (Dmitry Gorodnichy) Two problems to be resolved Kinematics problem - to find out where the arm should be seen -> FastKin, Cosmos Vision problem - to find out where the arm is actually seen -> R.A.C.E. Major challenge – Recognition Problem Extract the image of SSRMS = Recognize SSRMS in the image

7. ROSA: Adding safety using video. (Dmitry Gorodnichy) Don’t forget: Vision is very ill-posed problem i.e. everything (approaches,complexity,results,success…) depends on the images used

8. ROSA: Adding safety using video. (Dmitry Gorodnichy) R.A.C.E. software -For testing the applicability of different approaches -Uses custom made kinematics & image processing -Integrates: FastKin (MDR) Cosmos (NRC) -Features: joints setup selection camera selection

9. ROSA: Adding safety using video. (Dmitry Gorodnichy) R.A.C.E. allows … Allows one to detect parts of the arm which are possible to see by a camera and -to remove background -to show it by colouring -to obtain the usefulness of the image -to calculate the discrepancy level between what is seen and what should be seen (according to the kinematics) -With different image processing techniques

10. ROSA: Adding safety using video. (Dmitry Gorodnichy) Approaches tested -Intensity discontinuity based (+ gradient filters + Hough transform): assumes edges can be seen and many of them are parallel …………………………………...… good -Texture based segmentation (+ region-growing + shape-from- shading): assumes colour histogram of SSRMS is known ….good -Geometry (Cylinder) based: assumes SSRMS is a collection of cylinders of a known width ……………… very good - Morphology: skeletonization, AND/NOR/NOT of the above approaches …………………………….……. best - Contour-based (snakes): assumes there’s a complete contour ….bad

11. ROSA: Adding safety using video. (Dmitry Gorodnichy)

12. ROSA: Adding safety using video. (Dmitry Gorodnichy) Results and Conclusions Tests conducted on images generated by Cosmos showed that: while it is not always possible to detect completely the arm position, R.A.C.E can significantly facilitate understanding of image context (such as “where SSRMS is”), which can be used 1) for automatic detection of the best views 2) for operator-based manipulation of the arm. Most efficient recognition techniques are determined Possibility of automatic detection of joint malfunctioning is shown (provided that there is the agreement between the kinematics equations and the camera parameters) Ready for experiments with real images.