1. CASA 2006 CASA 2006 A Skinning Approach for Dynamic Mesh Compression Khaled Mamou Titus Zaharia Françoise Prêteux

2. 3D animation industry Context & Objectives  Applications  Virtual and augmented reality  Cartoons  Video games and CGI films

3. Dynamic 3D content Context & Objectives How to exchange, transmit and visualize such 3D content in a platform-independent manner ?!  Content creation  Motion capture  Skinning models  Physical-based simulation  …

4. Context & Objectives  Principle  Represent the animation sequence as a set of key-meshes Key-frames Animation Interpolation 3D animation industry: key-frame representations Dynamic 3D content  Apply interpolation procedures to generate the in-between frames at the desired framerate

5. Context & Objectives Constant topology Time-varying geometry  Sequence of meshes with:  Constant topology  Time-varying geometry Dynamic 3D content Key-frame representations: dynamic 3D meshes  Advantages  Generality  Interoperability  Content protection  Drawbacks  Huge amount of data  Need of compact representations

6. Objectives Context & Objectives  Compression efficiency Compactness of the coded representation  Progressive transmission Bitstream adaptation to different, fixed or mobile communication networks and terminal devices  Scalable rendering Bitstream adaptation for real-time rendering

7. Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

8. Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

9. Wavelets Vertex prediction State of the art MPEG-4/AFX-IC Dynapack AWC GV PCA-based LPCA CPCA PCA Clustering Dynamic 3D mesh compression RT D3DMC  Emerging field of research  Four families of approaches

10. Wavelets Vertex prediction State of the art MPEG-4/AFX-IC Dynapack  A more elaborated motion model: skinning AWC GV PCA-based LPCA CPCA PCA Clustering Dynamic 3D mesh compression Skinning-based compression RT D3DMC  New motion-based segmentation procedure  Temporal DCT-based compression of the residual errors  Principle: extension of the RT technique

11. Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

12. General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

13. Motion-based segmentation Skinning-basedcompression  Objective Partition the mesh vertices into clusters whose motion can be accurately described by a single affine motion

14. Motion-based segmentation Skinning-basedcompression  Principle  For each vertex v, select a neighborhood v*

15. Vector of homogeneous coordinates of vertex p at frame i Motion-based segmentation Skinning-basedcompression  For each frame i, compute an affine transform A i v … Frame 0Frame 1Frame (F-1)  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v

16. Motion-based segmentation Skinning-basedcompression  For each frame i, compute an affine transform A i v  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v Cow  Determine the partition π = ( π k ) k by applying the k-means clustering algorithm to the set ( α v ) v

17. Motion-based segmentation Skinning-basedcompression Dancer  For each frame i, compute an affine transform A i v  Principle  For each vertex v, select a neighborhood v*  Store the ( A i v ) i of each vertex as a single vector α v  Determine the partition π = ( π k ) k by applying the k-means clustering algorithm to the set ( α v ) v

18. General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

19. Affine motion estimation Skinning-basedcompression  Principle  Model the motion of each cluster k at each frame i by an affine transform H i k  Predict the geometry of frame i from frame 0 by using the affine transforms ( H i k ) k

22. Affine motion estimation Skinning-basedcompression  Performances  Captures well the object motion Frame 0Frame 36Predicted frame 36 Error distribution 0% 4%  Induces discontinuities at the level of clusters boundaries We need a more elaborated motion model

23. Skinning model Skinning-basedcompression  Objective Derive a continuous motion field  Principle  Linearly combine the affine motion of adjacent clusters with appropriate weighting coefficients  Compute the animation weights by solving a least squares minimization problem

26. General view Skinning-basedcompression Temporal DCT Affine motion and weights estimation Motion-based segmentation Static encoder Quantization and arithmetic encoding Prediction residuals Affine transforms Animation weights Partition (Mi)(Mi) M0M0 Compressed DCT coefficients Compressed M 0

27. DCT-based compression of the residual errors Skinning-basedcompression  Objective  Compress the residual errors by exploiting the temporal correlations Prediction error at frame i and vertex v  Principle  For each vertex v, compute the spectra of its x, y and z errors  Concatenate the spectral coefficients of all vertices into a single vector S  Quantize and arithmetically encode S Well-adapted to progressive transmission

28. Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

29. Evaluation corpus: Snake Experimentalresults 9179 vertices 134 frames

30. Evaluation corpus: Dancer Experimentalresults 7061 vertices 201 frames

31. Evaluation corpus: Humanoid Experimentalresults 7646 vertices 154 frames

32. Evaluation corpus: Chicken Experimentalresults 3030 vertices 400 frames

33. Objective evaluation: criteria  Compression rates: bits per frame per vertex (bpfv)  Distortion measures: RMSE [MESH tool, Aspert et al, 2002] D : length of the diagonal of the object’s bounding boxExperimentalresults

34. Compression results: Chicken Experimentalresults  Performances  D3DMC & skinning: best performances  Skinning: up to 47% gain over D3DMC in term of bitrates RMSE bpfv

35. Compression results: Snake Experimentalresults  Performances  PCA: worst performances (F>>V not verified)  Skinning: up to 45% gain over RT in term of bitrates RMSE bpfv

36. Compression results: Humanoid Experimentalresults  Performances  AFX-IC: poor performances: elementary predictor  Skinning: up to 67% gain over D3DMC in term of bitrates RMSE bpfv

37. Compression results: Dancer Experimentalresults RMSE bpfv  Performances  GV: re-meshing related problems  Skinning: up to 65% gain over GV in term of bitrates

38. Outline Experimental evaluation Conclusion et perspectives Previous work Skinning-based compression

39. Conclusion & perspectives Summary  A new skinning-based compression techniques for dynamic meshes  Specifically efficient for articulated dynamic meshes  Gains range from 47% to 67% in terms of bitrates over state-of-the-art encoders

40. Future work Conclusion & perspectives  Optimize the motion-based segmentation stage: How to determine automatically the number of clusters?  Multiple and dynamic skinning models: Temporal segmentation of the sequence  Compression of other attributes: normals, texture coordinates…

41. Thank you