1. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Dept. of Mechanical and Automation Engineering Yan Lu, Josh Lam, Yeung Yam Preliminary Study on Vision-based Pen-and-Ink Drawing by a Robotic Manipulator

2. Intelligent Control Systems Laboratory The Chinese University of Hong Kong  Motivation  Background  Methodology  Demonstration  Summary Outline

3. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Pen-and-Ink is a traditional form of art, which is still popular today. Motivation

4. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Skill-dependent Motivation Time-consuming

5. Intelligent Control Systems Laboratory The Chinese University of Hong Kong A. Robot Platform  5-DOF manipulator  0.1mm movement resolution  Visual feedback by CCD camera Background

6. Intelligent Control Systems Laboratory The Chinese University of Hong Kong B. Pen-and-Ink Properties 1.Stroke is monochromatic; 2.Stroke contributes to both tone and texture; 3. Strokes work collectively, that is, no single one is of critical importance. Background

7. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Main challenge Only monochromatic strokes can be used to convey three key aspects of an image: structural contents tone– brightness and darkness and textures Methodology

8. Intelligent Control Systems Laboratory The Chinese University of Hong Kong A. Structural Contents Step 1. Outline extraction– Canny edge detection Step 2. Structural significance of edges– measured by the “lifetime” in the Gaussian scale space of the original image ( Orzan et al. 2007 ) http://artis.imag.fr/Publications/2007/OBBT07/ Step 3. Outline delineation by strokes of different widths according to structural significance Methodology

9. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Original imageStructural significance

10. Intelligent Control Systems Laboratory The Chinese University of Hong Kong B. Tone Expression The key is to place a stroke at a right location, based on the rule that strokes should be placed evenly: close together in dark areas, and widely spaced in bright areas. To accomplish this hatching process, we need the vision system to provide visual feedback. Methodology

11. Intelligent Control Systems Laboratory The Chinese University of Hong Kong B. Tone Expression (Cont’d) Step 1. Capture the current drawing by camera, blur it by an average filter, transfer it to top view by homography, denoted as I op ; Step 2. Compute the difference between current image and object image: I df =I oj -I op Methodology

12. Intelligent Control Systems Laboratory The Chinese University of Hong Kong B. Tone Expression (Cont’d) Step 3. Determine the importance image as: Step 4. Place the next stroke at the point where the value of the importance image is the largest Step 5. Repeat 1~4 steps Methodology Where i denotes the i -th stroke, and a is a parameter, empirically set as 0.1~0.5.

13. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Object image I oj Current image I op Importance image I ip Methodology B. Tone Expression (Cont’d) Example of determining stroke position

14. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Methodology C. Texture Representation Strokes should be appropriately orientated to represent desirable textures. A natural way is to orientate a stroke according to the gradient direction of the original image. However, in practice, image gradient directions are often too noisy to be used directly.

15. Intelligent Control Systems Laboratory The Chinese University of Hong Kong C. Texture Representation (Cont’d) Problem illustration Methodology Original imageLocal texture By gradient

16. Intelligent Control Systems Laboratory The Chinese University of Hong Kong C. Texture Representation (Cont’d) Since strokes work collectively, a local texture should be expressed by a group strokes with consistent orientations. We propose to select a set of “strong points” whose gradients are large enough to be believed, and calculate stroke direction by interpolation using a few nearest “strong points”. Methodology

17. Intelligent Control Systems Laboratory The Chinese University of Hong Kong C. Texture Representation (Cont’d) Stroke Orientation at p Methodology Where p i is one of the n strong points nearest to p, θ(p i ) is the gradient direction at p i, and d(p,p i ) denotes the distance between p and p i.

18. Intelligent Control Systems Laboratory The Chinese University of Hong Kong C. Texture Representation (Cont’d) Methodology Strong pointsBy gradient By gradient interpolation

19. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Methodology D. Stop Criterion A criterion function is proposed to terminate the hatching process automatically as follows: The first component is a descending function wrt. stroke number i, while the second is an ascending one. They work together to determine the stop point.

20. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Methodology D. Stop Criterion (Cont’d) The hatching process stops at the minimum point of the criterion function, which can be tuned by b. b=0.9, i=1400b=1.0, i=1000

21. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Methodology Overview

22. Intelligent Control Systems Laboratory The Chinese University of Hong Kong Demonstration i=0 i=300 i=700 i=1026 i=1283 i=1502

23. Intelligent Control Systems Laboratory The Chinese University of Hong Kong

24. Intelligent Control Systems Laboratory The Chinese University of Hong Kong We have presented a robotic system for automated Pen-and-Ink rendering  Structural contents delineated according to structural significance of edges  Tone expression based on visual feedback  Texture representation by interpolating local gradient directions  Automatic stop using a criterion function Summary

25. Intelligent Control Systems Laboratory The Chinese University of Hong Kong