1. www.crs4.it/vic/ Web3D 2012 - Los Angeles, August 2012 Adaptive Quad Patches: an Adaptive Regular Structure for Web Distribution and Adaptive Rendering of 3D Models Enrico Gobbetti 1, Fabio Marton 1, Marcos Balsa Rodríguez 1, Fabio Ganovelli 2 and Marco Di Benedetto 2 1 CRS4 Visual Computing Group, Sardinia Italy 2 ISTI-CNR Visual Computing Laboratory, Pisa Italy

2. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Context 3D scanning systems has become commodity hardware –Easy to use –Fast –Portable Increasing virtual world creation –Museums –Virtual shops and galleries –Games WebGL –Mobile devices –Web integration –Internet –More users

3. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Motivation 3D models –Heavy –Non-trivial to render –Non-linear inspection Constrained environments –Lightweight –Interpreted –Scripted Reuse components already optimized! Javascript

4. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Pipeline overview Unattended software pipeline Compact multiresolution representation 4

5. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive Quad Patches – Approach Remote rendering –Compact multiresolution representations Storage Distribution Real-time rendering Mesh reparametrization –Point cloud -> quad mesh –Multiresolution –Tightly packed texture atlas Low CPU overhead –GPU accelerated adaptive tessellation algorithm WebGL based –Portable platforms

6. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Related work General solutions available for Desktop environments [Yoon et al. 2008] Mesh compression – MPEG-4 [Jovanova et al. 2008] View dependent LOD and progressive streaming [Hoppe 1997] –Compute view dependent triangulation each frame -> CPU bound Surface patches [Cignoni et al. 2004] –Effective in terms of speed -> require non-trivial data structures and techniques for decompression Geometry images [Gu et al. 2002] –Exploit current GPU capabilities / optimized libraries for compression and streaming of images 6

7. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Related work Quad remeshing –Single-disk parametrization [Floater and Hormann 2005] Simplest -> only genus-0 almost zero Gaussian curvature –Base mesh to parametrize the model [Petroni et al. 2010] Quad-based domain -> tight packing and simpler handling of chart boundaries Detail rendering –GPU raycasting [Oliveira et al. 2000] –Displacement mapping in GPU [Shiue et al. 2005] –Semi-uniform adaptive patch tessellation [Dyken et al. 2009] 7

8. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Pipeline overview Unattended software pipeline Compact multiresolution representation 8

9. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Generate clean manifold triangle mesh –Poisson reconstruction [Kazhdan et al. 2006] –Remove topological noise Discard connected components with too few triangles Parameterize the mesh on a quad-based domain –Isometric triangle mesh parameterization Abstract domains [Pietroni et al. 2010] –Remap into a collection of 2D square regions Resample each quad from original geometry –Associates to each quad a regular grid of samples (position, color and normal) Pre-processing (Reparameterization) 9

10. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Pre-processing (geometry representation) Coarse geometry –Quads (4 corners) Detailed geometry –Displacement Texture With respect to corner interpolation at (u,v) coordinates 10

11. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Collection of variable resolution quad patches –Coarse representation of the original model Multiresolution pyramids –Detail geometry –Color –Normals Shared border information –Ensure connectivity Pre-processing (Multiresolution)

12. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Ensure continuity (detail geometry, color, normals) –Special filtering maintains shared information on borders Per patch quantization (quad ranges) –Max/min at patch corners -> Avoid discontinuities PNG or S3TC compression –Geometry -> lossless PNG –Color and normals could be lossy Tiled storage –N pyramids –Separated mipmap levels (per pyramid) –Mipmap levels splitted into tiles (e.g. 512x512) Pre-processing (Multiresolution - filtering)

13. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches 4 3D corners with 16 bit / component 4 Corners quantization ranges –Bilinear interpolation of quantization ranges ensure continuity among quads. Patches of quantized 3D displacements –With respect to base quad Adaptive rendering (Quad geometry information)

14. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive rendering (LOD selection) For each quad –For each edge Evaluate edge LOD –Quad LOD = max edge LOD (available) –Edge LOD = Min(edge LOD, neighbor patches available LOD) –If LOD not available Post request Use finest available LOD Priority = Required LOD - Available LOD

15. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive rendering (GPU Storage) 1 VBO per resolution level reused 1 stream of (u,v,e) information per vertex (e) determines the vertex (0,1,2,3, or 4 for inner vertices) K preallocated incomplete (filled on demand) texture mipmaps of –Displacements –Normals –Colors Texture with quad ranges (dequantization) Texture with edge LODs

16. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive rendering (Pyramid Rendering) Bind pyramid textures For each quad –Draw VBO with proper level triangulation –Shaders Vertex: Dequantize using (u,v) coordinates Snap edge vertices (u,v,e) Displacement mapping Fragment Color Normal Shading.

17. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive rendering (Shader Geometry Dequantization) Base offset = quad corners Interpolation Interpolate dequantization ranges Dequantize displacement Sum displacement to base offset

18. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Rendering example Patches Levels Shading

19. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Adaptive streaming Server –Tile repositories –No processing –Berkeley DB + Apache2 Client (WebGL) –Priority = desired LOD – available LOD –Separate thread for receiving data PNG decompression –Insertion into pyramid when ready 19

20. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Results (preprocessing) 20 Adaptive rendering –1 pixel accuracy – 37 fps (min 13 fps) Network streaming –Required 312Kbps (max 2.8Mbps) for no delay –ADSL 8Mbps – 2 s fully refined model from scratch

21. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Results 21 Tests configuration –1.6Ghz laptop –4GB RAM –Nvidia Geforce GTX 260M with 1GB VM –Linux Gentoo 2.6.39 –Chromium browser 19.0.1084.24b –ADSL 8MBps

22. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Conclusions Effective creation and distribution system –Fully automatic –Compact, streamable and renderable 3D model representations –Low CPU overhead –WebGL Desktop Mobile Limitations and future work –Closed objects with large components –Visual approximation (lossy) –Explore more aggressive compression techniques –Occlussion culling –More sophisticated shading/shadowing techniques 22

23. Web3D 2012 - Los Angeles, August 2012 Marcos Balsa, Adaptive Quad Patches Questions and contacts CRS4 - Visual Computing Group http://vic.crs4.it http://vic.crs4.it ISTI-CNR - Visual Computing Lab http://vcg.isti.cnr.it http Speaker: Marcos Balsa mbalsa@crs4.it mbalsa@crs4.it The DIVA project http://diva-itn.ifi.uzh.ch/ This research is partially supported by the EU FP7 Program under the DIVA project (290277). 23