Geometric Algorithms in 3D Real-Time Rendering and facial Expression Modeling Prof. Marina L. Gavrilova University of Calgary

Quote: "Perseverance is not a long race; it is many short races one after another." -Walter Elliott

Brief Overview of the Area Geometric Approaches for mesh reconstruction and image morphing Real-Time Optimally Adapting Mesh Algorithms Non Photo-realistic Facial Expression Modeling and Morphing Algorithms Swarm Real-time Visualization System Presentation Outline

Objective: To showcase the power of the geometric and topology-based approaches in 3D visualization and modeling

Relevant Areas Computer-Aided Design Geographical Information Systems Biometrics 3D Animation Computer Games Strategic Planning Defense Other areas

State of the Art TERRAIN MODELING Multi-resolution dynamic mesh Viewer-dependent refinement Culling techniques View frustum algorithms Triangle-count preservation Subdivision refinement Controlled LOD Visual continuity and smooth transition are crucial Lindstrom, Hoppe, Duchaineauy, Hesse, Silva, Skala, Kolingerova EXPRESSION MODELING Multi-resolution adaptable mesh Light source and view point dependent rendering Control points and areas Tracking algorithms for transforming Fitting algorithms for modeling Wavelets to extract specific characteristics Visual continuity and feature preservation are crucial Jain, Salesin, Gooch, Gordon, Thalmann, Plemenos, Sourin

Our Approach TERRAIN MODELING Challenge: massive amount of data, smooth rendering Methodology: Multi-resolution dynamic triangle mesh with new refinement strategy New split and merge queue operations Frame-to-frame coherence algorithm for local updates New geo-morph smoothing algorithm Outcome: Visual continuity, rendering efficiency, frame-rate constancy for massive terrain EXPRESSION MODELING Challenge: massive amount of detail, tracking changes Methodology: NPR approach to feature extraction and modeling New Distance-Transform algorithm for morphing Delaunay triangulation for mesh representation Image segmentation and feature-line extraction algorithm Outcome: Expressive detail preservation, efficient storage, fast processing, 3D controllable deformations

Our Focus Topology preservation and geometry plays an important role in many algorithms, allowing for rendering efficiency, visual continuity and feature preservation.

Part I: 3D Real-Time Massive Terrain Visualization

Sourcing data from real world

Understanding Height Fields DEM: Digital Elevation Model Contains only relative Height Regular interval Pixel color determine height Discrete resolution

Converting Height field data into 3D topological mesh Pixel value (z) is used as Height Map Vertices are generated as points in 3D A Mesh is triangulated

Terminology DEM: Digital Elevation Model VDR: Viewer Dependent Refinement ROAM: Realtime Optimally Adapting Mesh CLOD: Continuous Level of Detail

What is mesh? It is often required to convert geometric objects in any other form to polygonal meshes of geometric simplexes (i.e. triangles) before rendering Meshes are used for computer modeling, biometric, GIS, cartographic, and animation applications Mesh optimization techniques are used to reduce the mesh to an acceptable size (user defined), minimizing the loss of quality (subject to some form of error analysis).

Mesh definition A mesh is defined as a pair (V,K) where V is the set of vertices and K is a simplicial complex specifying the connectivity of the mesh simplices (the adjacency of the vertices, edges and faces). The 0-simplices are called vertices, the 1- simpices are called edges, and the 2-simplices are called faces. Triangulation is an example of mesh.

Example: 3D Terrain Rendering Uses DEM as input of the application Generates dynamic mesh to achieve frame coherent animated view in real-time

Contribution In our work we used the ROAM definition introduced in [Duchaineauy], and developed a number of algorithms that rely heavily on topological properties of the modeled terrain in order to handle massive data sets. The observation of hardware performance reacting to strip length allowed us to design a fast and effective greedy approach to dynamically maintain triangle stripping. The method resulted in achieving rendering efficiency and frame rate constancy. We further improved the rendering efficiency while preserving important terrain features utilizing topological information. We eliminated the priority queue for merge operation, amortized update cost by allowing split queue to degenerate, took advantage of frame-to-frame coherence for local update and devised an efficient method for mesh-smoothing using geo- morph while preserving important terrain features.

Methodology applications Massive terrain rendering Realistic deformation modeling using mass spring system Adaptive spatial memory subdivision in robot motion planning simulation

Basic Approach The entire optimization process can be viewed as an optimal spatial subdivision problem. There exist only two legal moves to achieve optimality: subdivide and merge. Models that are based on this concept do not require computationally extensive optimization process. Instead, the mesh progressively adapts to an optimal solution. These models usually define a multi-criteria based error metric to assess the energy of the mesh.

Conventional Approach Regular Grid and TIN: Triangulated Irregular Network Problems: –Significant preprocessing –Generates too many triangles –the visible area might constitute a tiny fraction of that detail –Viewpoint dependent refinement is not possible with static meshes. –Have a bad triangle to screen pixel ration. –Unable to obtain a steady performance. Unable to Render Massive Terrain

Problems with static mesh

Problems with static mesh … Detail 98.5% 1.5% Screen 10% 90%

A Static Mesh (40,000 tri)

Dynamic mesh Dynamic mesh generation is based on real-time adaptive models that allow local transformation of a given mesh to different levels of details and adapt progressively to the changes in the rendering environment. Dynamic models use some form of hierarchical data structure to store a mesh in multiple LOD. There are many different versions of dynamic mesh currently under research and development. Several patented technologies have been released in the industry (i.e. Intels MRM). The two most prominent approaches are Progressive Meshes (PM) and Real-time Optimally Adapting Mesh (ROAM).

ROAM : Can Subdivide when more details necessary Merge & Split Queue Tree Structured

ROAM (25,000 Triangles)

The ROAM Advantage

ROAM advantages An efficient top-down dynamic mesh model. Display of complex surfaces can be provided by dynamically computing a multi-resolution triangular mesh for each view. One can minimize geometric distortions on the screen while maintaining a fixed triangle count. Visual discontinuities could be minimized, and efficient mesh corrections can ensure selected lines of site, or object proximity are correctly represented. An incremental priority-queue based approach in ROAM allows exploiting frame-to-frame coherence for fast efficient computation of the adapting mesh.

Geometric subdivision Problems with Geometric Subdivisions

ROAM principle The basic operating principle of ROAM

Quadtree and Bintree for ROAM Comparison between Quadtree (top) and Bintree (bottom)

ROAM Model DEM Interpolation Engine Triangle Bin Tree Data structure Render Refine Error Metric Speed: 15.0 FPS Size: Tri

Our ROAM Model Speed: 45.0 FPS Size: Tri DEM Interpolation Engine Triangle Quad Tree Data structure Error Metric Refine Render G Q Wavelet Error Analysis Cleanup Operation

ROAM Implementation and Improvements Pre-process DEM Reduce overhead for maintaining Split queue Use degenerated split queue Replace priority merge queue (HEAP) with a simple FIFO queue. Merge arbitrary obscured regions (outside visible frustum) only Improve smoothness, reduce popping effect

Proposed Unique Geometry- based methodology DEM Analysis Adaptive Loop Subdivision Geo-Morphing Visible frustum computation CCM - Cascading Coupling March method for massive terrain

Preprocessing of DEM input Convolute the surface (2 nd Difference) Efficient LOD distribution Used in Advanced Shading (feature preservation)

Differentiating the Surface: Convolution Original DEM of Kluane Park First differentiation. Blue area Shows steep mountains. Final LOD distribution image Shows rocky areas require more details Second Differentiation. Blue areas indicate steep mountain peaks.

Differentiating the Surface (cont’d)… Take Weighted Average of neighboring pixel Gradients 2 passes gives us the rate of gradient changes

Statistical Analysis of ROAM Bottleneck: Split and Merge Operation and visibility test Only 2.7% triangles are merged within the visible frustum 75% of split operations occurs on closest 5% region of the visible frustum Taking above into account, special priority function for split queue was developed based on U=Uniformity F=Frustum priority D=Curvature priority P=Distance priority DET=Curvature (preprocessed) Lev(f)=Subdivision level of face f

Methodology: Speeding up the process - Amortizing refinement Updating the Split queue over multiple frames Allow split queue (heap) to degenerate Refine integrity using error threshold test Re-generate queue after constant number of frames Dramatic gain in speed

Methodology: Maintaining Degenerated S-Queue

Methodology: Geo-Morph Smooth interpolation of vertex over subdivision Requires maintenance of one additional queue (G-queue)

Methodology: Smoothing Compute face normal Compute vertex normal Local computation of normal during refinement (Geo- Morph) Apply vertex normal when rendering Advanced shading: Linear combination of face and vertex normal (Hybrid)

Example of smoothing

Result with Texturing

Massive Terrain Model: ALS Adaptive Loop Subdivision and Cascading Coupling March

Quote: “Map of a region was a size of a city, map of an Empire was a size of a Province." Jorge Luis Borges

Loop Scheme

Patching alignment A B C A BC Faces: Regular Tface Regular Tface

Adaptive Loop Subdivision

Recursive Split A A A A BB B B CC C C AB CA BC Split Prefer: Tface Retract Tface

Patch Stitching: CCM “CASCADING COUPLING MARCH”

Method Description We develop CCM to stitch two quadrants of the adaptive mesh patch, and to ensure seamless connectivity by progressively refining and connecting the subdivision tree (quadtree) data structure. We ensure integrity of the data structure among multiple patches so that subdivision and simplification are not limited to the boundaries of the patches. CCM allows to consider the entire data structure as a number of strongly connected sets of quad-trees. As a result we can refine the mesh as one seamless continuous structure (not patches stitched together).

CCM idea We perform a coupled recursive march of the two quad- trees from the top to the bottom of the tree, subdividing or connecting border triangles. To increase geometric regularity we used non-orthogonal axis and used space warps to convert the scheme back to orthogonal axis for rendering. This is crucial when we are performing viewer dependent refinements. Patch locating and coordinate transform algorithms are designed to maintain a list of visible patches. Additional patches are added and connected using CCM as the simulation advances. Obscured patches are deleted using recursive march. This new method allows a greater flexibility, efficiency and control over smoothness

Two Adjacent Patches

Snapshot 3 … Level 2

Results on Mesh Rendering - Strategies evolved, Runtime and other aspects

Results (Output obtained)

Results (Runtime) (Cont’d)

Summary of Results High Speed (45 FPS average) for faces!! Static mesh upper limit: 55 FPS Continuity is achieved (Geo-Morph) – 300 vertices on average for 45 FPS –very small overhead Visual appeal (smoothing & Texture) Less variant Frame rate Efficient Understructure (amortized update)

More Results on Mesh Rendering See Animation Demos