Adaptive Point Cloud Rendering A Feasibility Study Group May13-11 Christopher Jeffers - Communicator Eric Jensen - Webmaster Joel Rausch - Coordinator Client: Siemens PLM Software Advisor: Professor Simanta Mitra

Outline What Is Siemens PLM Software? What Is a Point Cloud? What Are We Doing? What Are the Requirements? What Solutions are There? How Are We Going to Do It? What Have We Done? What Will We Do? Questions? © Kean Walmsley http://through-the-interface.typepad.com/.a/6a00d83452464869e2013485dc88bd970c-pi

Siemens PLM Software “A business unit of the Siemens Industry Automation Division” “A leading global provider of product lifecycle management (PLM) software and services.” “Nearly 6.7 million licensed seats and 69,500 customers worldwide” “Headquartered in Plano, Texas.” “Works collaboratively with companies to deliver open solutions that help them turn more ideas into successful products.” “For more information on Siemens PLM Software products and services, visit www.siemens.com/plm.” Images and text from Siemens PLM’s Goelus Brochure (http://www.plm.automation.siemens.com/en_us/products/open/geolus/index.shtml#lightview%26uri=tcm:1023-7061%26title=Geolus%20-%20PLM%20Components%20Brochure%20-%208123%26docType=.pdf)

Star Wars: Rogue Squadron © LucasArts What is a Point Cloud? A collection of vertices In computer graphics, a way to represent an object or scene as set of points Polygons Star Wars: Rogue Squadron © LucasArts Lines/Vectors Star Wars © Atari Points © Nghia Ho Click Animation L: http://prohozhdenie-igr.com/2269-star-wars-rogue-squadron-bonus-mission-18-the.html M: http://www.cosc.brocku.ca/Offerings/3P98/course/lectures/intro/star_wars_large.png R: http://nghiaho.com/?attachment_id=1720

© Stanford Computer Graphics Laboratory Why Have Point Clouds? 3D scanners output points clouds 19-year old toy © 1993 Bandai Stanford Armadillo © Stanford Computer Graphics Laboratory

Primary Business Use Case Siemens PLM provides software that can simulation a layout of machinery on a factory floor. Currently, the model of the factory is based on the building plans The problem: these floor plans are not always maintained, as new machinery is added. Siemens PLM’s idea: Scanned the build, so it is always correct. Leica terrestrial LIDAR (Light Detection And Ranging) © 2007 David Monniaux

How to Deal With a Point Cloud? Convert it to a polygon mesh? Readily available, but not designed for their needs and good results require creation-time knowledge Convert it to multiple meshes? Not an easy solution = = /

So What Are We Doing? We are testing the feasibility rendering a point cloud as a collection of points. Our Goals Create a medium-scale (100s of millions to billions of points) point cloud rendering kernel using C++ and OpenGL. Implement a data structure to organize the point cloud data. Implement a traversal or selection algorithm to choose the correct points to render from the current viewpoint. Create a viewer that gives the user basic input controls to manipulate the view of the point cloud.

Requirements Functional Non-Functional Allow the user to load a point cloud in a TBD format . Allow the user to rotate, zoom, and pan the view of the point cloud. Non-Functional Rendering kernel must be able to display millions of points concurrently. The application must maintain an interactive frame rate (> 1 fps) Selection algorithm should take no more than 10 sec to run.

Requirements, cont. Inverse Requirements Constraints Mesh reconstruction from point clouds Segmentation (RANSAC) Point cloud compression Collision detection Brute Force Approaches Constraints Must be written in C++ programming language. Must use OpenGL library for rendering graphics. All data must fit within a few gigabytes of main memory.

How to Render A Billion Points? We looked through 10+ papers. We selected three methods to further investigate and test. Splatting using Octrees (Kobbelt 02) QSplats (Rusinkiewicz 00) Ray Cast Filtering using K-d Trees (Botsch 03) High-Quality Point-Based Rendering on Modern GPUs (Kobbelt 02) Efficient High Quality Rendering of Point Samples Geometry (Linsen 01) Point Cloud Representation (Moenning 03) A New Point Cloud Simplification Algorithm (Rusinkiewicz 00) QSplat: A Multiresolution Point Rendering System for Large Meshes (Rusu 11) Point Cloud Library (Wimmer 06) Instant Points Fast Rendering of Unprocessed Point Clouds (Rusinkiewicz 01) Streaming QSplat: A Viewer for Networked Visualization of Large, Dense Models (Havran 00) Heuristic Ray Shooting Algorithms (Römisch 09) Sparse Voxel Octree Ray Tracing on the GPU (Kashyap 10) Fast Raytracing of Point Based Models using GPUs (Marques) GPU Ray Casting Group: May13-11 Client: Siemens PLM

Splatting (Point Rendering) Developed by Lee Westover - Vis 1989; SIGGRAPH 1990 Every point is treated as a glob of color and is splat on a the frame, like, as Westover said, a snow ball. Requires front-to-back or back-to-front ordering when using alpha bending to smooth rendering. We probably will be using a FastSplat or geometry shader method. Auto animation

Splatting using Octrees – Octree A tree structure is built by sub-dividing Euclidian space. A node defines a virtual point at its center Pros Easily partitionable Fixed depth Cons Requires knowledge of the bounding box at creation times All information of the point cloud is lost. Space: O(d*n) © WhiteTimberwolf Group: May13-11 Client: Siemens PLM

Splatting using Octrees – Algorithm Based on Efficient High Quality Rendering of Point Samples Geometry by Mario Botsch, Andreas Wiratanaya, Leif Kobbelt Method: Modified DFS of the octree. Traverse a node when is in the viewing frustum. Terminate branch when no children exist or are traversable or at the resolution boundary of the frustum. Order children in traversal back-to-front of the frustum when blending. Pros Back-to-front order for alpha blending. Easy to off-load parts of the traversal to the GPU Cons Hidden points may be rendered There is a resolution limit and fixed depth. Space: O(d*n), where d is the depth Time: Average: O(A*8k), where k is the resolution boundary and A is the completeness of the octree Worst: O(d*n) bound by O(8d) Manual animation

Ray Cast Filtering – K-D Tree BST in k dimensions. A node defines a point and splitting plane. Pros All information of the point cloud is preserved. Four paths of traversal (L, R, Both, None). Cons Creating a balanced tree requires median finding Cannot be rebalance. Space: O(n) © KiwiSunset (upper) © Myguel (lower) Group: May13-11 Client: Siemens PLM

Ray Cast Filtering – Algorithm Derived from ray tracing Use view rays to select only the points that can be seen. Pros Independent updating and rendering. No resolution limits. Fixed number of points being rendered (< 9M). Cons No filtering. Viewport resolution changes are ill defined. Back-to-front alpha bending is ill defined. Space: O(n) Time: Average: O(R lg(n) + R) should be possible Worst: O(R * n), where R is the number of pixels

Ray Cast Filtering – Ray Casting S1: Test if the ray on the node’s splitting plane and store. S2: If S1 is false, traverse the child closest to the origin. Else, if there is an equal-axis child closer to the origin, traverse the or-equal child. S3: If S2’s child returns a point, return that point. S4: Test if the ray hit this node’s point. ((𝑓 ≥ 0) && (𝑔 < 0)) || ((𝑓 < 0) && (𝑓2 ≥ 𝑔)) S5: If S4 is true, return this node’s point. S6: If S1 is false, return empty. Else, Test if the ray intersection with the node’s axis. S7: If S6 is true, traverse remaining child. S8: If S7’s child returns a point, return that point. S9: Return empty. 𝑓 = 𝐸 ∙ 𝐷 𝑔 = ‖𝐷‖2 ∙ (‖𝐸‖2 − 𝑟2) 𝐸 = 𝑂 − 𝐶 r C O D 9 8 7 6 5 4 3 2 1

QSplat – Data Structure Bounding sphere hierarchy Property: Parent less detailed than children Memory Layout Nodes arranged breadth first in a contiguous memory region Bounding sphere nodes Requires 32 bits without color; 48 bits with color Use special relations instead of pointers (Rusinkiewicz 00)

QSplat – Algorithm Based off the work of Szymon Rusinkiewicz and Marc Levoy Level of detail scaling Rough details are painted over with finer details Quality / speed tradeoff High quality while stationary Low quality while moving Refinement Improve quality in subsequent stationary frames Uses feedback system

Metrics *Memory/HDD Footprint *Load Time Size of backend Graphics Performance Backend Performance Frame Rate (Hz) The time it takes to draw a frame *Update Rate (Hz) The time it takes to update to a new static camera setting Fragment Coverage (%) The percentage of unique pixels that were determined Response Time (ms) The time it takes to begin updating update *Memory/HDD Footprint Size of backend *Load Time Time it takes to load X numbers of points into the backend * Our primary concerns for our prototypes

Results of prototypes - Splatting Node Size: 72 byte Packing: 8 – 8-byte pointers, 4-byte color, 4-byte normal Load Time 2,000,000 pts: 6 sec 10,000,000 pts: 32 sec Update Time 10,000,000 pts: ~ For 1 billion points (estimated): Memory Size: Load Time: 1-2 hrs Update Time: ~

Results of prototypes – Ray Casting Node Size: 20 bytes Packing: 2 – 4-byte pointers, 3-byte color, 3-byte normal, 6-byte point (3 – half floats) Load Time (Simulated): 2,000,000 pts: 5 sec 20,000,000 pts: 53 sec Ray Cast (Simulated): 2,000,000 pts: 2 sec 20,000,000 pts: 14 sec 200,000,000 pts: 44-183 sec Best Case for 20,000,000 pts: 0 sec For 1 billion points (estimated): Memory Size: 20 GB Load Time: 1-2 hrs Update Time (at 1080p fully coverage): 4 yrs (1 min / cast)

Results of prototypes - QSplat Node Size: 6 bytes Packing: 13 bits for position & radius, 3-bit pointer, 14-bit normal, 2-bit normal cone width, 16-bit color Load Time 10,000,000 pts: 20 seconds Update Time 10,000,000 pts: speed/quality tradeoff For 1 billion points (estimated): Memory Size: 6 GB Load Time: 3-4 hrs Update Time: speed / quality tradeoff

Results of prototypes - Summary What We Learned: Best option is QSplat We will need to create the structure at a separate time and store it in permanent storage We will want to cache splats to the GPU We will want separate rendering and updating Alg. Mem. Size Load Time Up-date Octrees 27 GB 1-2 hrs ~ Ray Casting 20 GB 4 yrs QSplat 6 GB 3-4 hrs Vari-able

System Description State Diagram Class Diagram

Accomplishments Figured out the problem. Researched multiple solutions. Selected the three most promising solutions. Prototyped our solutions. Selected the solution to be fully implemented based on our findings. We design an architecture for our application. Created our Project Plan, Design Document, and this presentation. We still alive.

What’s Next? Fully developing our Qsplat implementation. Creating our application. Testing our implementation Redesigning everything to improve performance Design a GPU-side solution Graduate and then sleep and then going to work.

Conclusion What is Siemens PLM Software? What is a Point Cloud? What are We Doing? What are the Requirements? What Solutions are There? How are We Going to Do It? What Have We Done? What Will We Do? © Kean Walmsley http://through-the-interface.typepad.com/.a/6a00d83452464869e2013485dc799d970c-pi

Questions Auto animation http://www.bnl.gov/bnlweb/history/higinbotham.asp http://en.wikipedia.org/wiki/File:Pong.png © LucasArts X-Wing