Processing Images and Video for An Impressionist Effect Automatic production of “painterly” animations from video clips. Extending existing algorithms for producing “painterly” pictures to create temporally coherent animations. Reduce user interaction to a bare minimum.

Methodology Overview Brush strokes are clipped to edges detected in the original image sequence and are oriented normal to the gradient direction of the original image. Scattered data interpolation is used when gradient is near zero. Brush stroke list is maintained and manipulated through the use of optical flow fields to enhance temporal coherence.

Rendering Strokes Rendered with an anti-aliased line centered at a point, with a given length, radius, and orientation. Color is determined through bilinear interpolation of the color at the point in the original image. Strokes are spaced apart by a user specified distance. Strokes are drawn using a random ordering to help achieve a hand-drawn look.

Randomizing Strokes Random amounts are assigned to the length and radius of the stroke, within a user specified range. Colors are perturbed by a random amount usually between [-15,15] for r,g,b. Colors are then scaled by an intensity usually within [.85,1.15] and then clamped to [0,255]. The orientation angle is randomized by adding a random angle amount between [-15º,15º]. All these randomizations are then stored in a data structure for each stroke and are not regenerated on a frame to frame basis.

Clipping and Rendering 1.An intensity image is created from the original color image using the equation: (30*red + 59*green + 11*b)/100 2.The intensity image is blurred using a Gaussian kernel, or a B-spline approximation to the Gaussian. 3.The resulting blurred image is Sobel filtered where the value of the Sobel filter is: Sobel(x,y) = Magnitude (Gx, Gy)

Sobel Filtering The Sobel filter consists of two kernels which detect horizontal and vertical changes in an image. The magnitude of an edge: M Sobel [x][y] = |Gh[x][y]| + |Gv[x][y]| The direction of an edge: φ Sobel [x][y] = tan -1 (Gv[x][y] / Gh[x][y])

Clipping and Rendering Con’t 4.Start the line at the center point and “grow” the line in its orientation direction until either an edge is reached or maximum length obtained. An edge is reached if the Sobel value decreases in direction of orientation that the line is growing. 5.Once we have determined the endpoints, the stroke is drawn using the perturbations stored in our stroke data structure. A linear falloff is used in a 1.0 pixel radius region ( alpha transition from 1 to 0 ).

Brush Stroke Orientation Strokes are drawn with a constant color in the gradient- normal direction. Brush strokes in a region of constant color smoothly interpolate the directions defined at the region’s boundaries. A thin-plate spline is used for the interpolation because it cannot be assumed that the data lies on a uniformly spaced grid. The gradient at the center point is bilinearly interpolated and the direction angle is computed using: Arctan(Gy/Gx) + 90º ( normal to the gradient ). The perturbations are added and the result is stored as this stroke’s direction angle.

Animating the Images Since there is no a priori information about pixel movement, vision techniques are applied to guide the brush strokes. Optical flow vector fields are utilized to determine pixel movement. Brush stroke centers are mapped to subpixel locations in the new image.

Handling Sparse Regions After using the vector flow field there may be regions which are too sparsely populated. Sparse regions don’t provide the coverage we wish to have. Use Delaunay triangulation to find triangles with area greater than maximum specified. Add pixels to reduce the area of these triangles. The Delaunay triangulation has the property that the circumcircle (circumsphere) of every triangle (tetrahedron) does not contain any points of the triangulation.

Delaunay Triangulation

Handling Dense Regions Dense regions will greatly slow down the rendering process. The edge list of the triangulation is traversed. If the distance between points is less than a user-specified distance, the stroke that is drawn closer to the back is removed. Display list of strokes determines stroke order.

Potential Problems The method of optical flow estimation employed by Litwinowicz assumes that lighting is constant and that occlusion may be ignored, resulting in objects appearing and disappearing. Stroke movement is only as good as the motion estimate technique. Noise in the images will cause noise in the animations.

Possible Enhancements and Future Work Scale the offset of the stroke based on the length of the stroke. Implement additional rendering styles to produce other artistic styles. This would involve modifying the stroke drawing algorithm to emulate the different styles. Apply this technique to 3D models where the motion of the objects in the scene would be known a priori. This could be accomplished by removing most of the vision algorithms from the system and replacing them with data structures that would hold all the necessary information for all the models of the items in the animation.