1. Sort-First, Distributed Memory Parallel Visualization and Rendering Wes Bethel, R3vis Corporation and Lawrence Berkeley National Laboratory Parallel Visualization and Graphics Workshop Sunday October 18, 2003, Seattle, Washington

2. 2 The Actual Presentation Title: Why Distributed Memory Parallel Rendering is a Challenge: Combining OpenRM Scene Graph and Chromium for General Purpose Use on Distributed Memory Clusters Outline: Problem Statement, Desired Outcomes Sort-First Parallel Architectural Overview The Many Ways I Broke Chromium Scene Graph Considerations Demo? Conclusions

3. 3 Motivation and Problem Statement The Allure of COTS solutions: Performance of COTS GPUs exceed that of custom silicon. Attractive price/performance of COTS platforms (e.g., x86 PCs). Gigabit Ethernet is cheap: $100/NIC, $500 8-port switch. Can build a screamer cluster for about $2K/node. We’re accustomed to “nice, friendly” software infrastructure. E.g., hardware accelerated Xinerama. Enter Chromium – the means to use a bunch of PCs to do parallel rendering. Parallel submission of graphics streams is a “custom solution,” and presents challenges. Want a flexible, resilient API to interface between parallel visualization/rendering applications and Chromium.

4. 4

5. 5 Our Approach Distributed memory parallel visualization application design: amortizes expensive data I/O and visualization across many nodes. The scene graph layer mediates interaction between the application and the rendering subsystem: portability, “hide” the icky parallel rendering details, provides an infrastructure for accelerating rendering. Chromium provides routing of graphics commands to support hardware accelerated rendering on a variety of platforms. Focus on COTS solutions: all hardware and software we used is cheap (PC cluster) or free (software). Focus on simplicity: our sample applications are straightforward in implementation, easily reproducible by others and highly portable. Want an infrastructure that is suitable for use regardless of the type of parallel programming model used by the application: No “parallel objects” in the Scene Graph!!!!!

6. 6 Our Approach, ctd.

7. 7 The Many Ways I Broke Chromium Retained mode object “namespace” collisions in parallel submission. Broadcasting: how to burn up bandwidth without even trying! Scene Graph Issues: to be discussed in our PVG paper presentation on Monday.

8. 8 The “Collision” Problem Want to use OpenGL retained-mode semantics and structures to realize performance gains in DM environment Problem: “Namespace collision” of retained-mode identifiers during parallel submission of graphics commands. Example: The problem exists for all OpenGL retained mode objects: display lists, texture objects and programs. The problem extends to all OpenGL retained mode objects: display lists, texture object id’s, programs. Process A: GLuint n = glNewList(1); printf(“ id==%d\n”); // id==0 // build list, draw with list Process A: GLuint n = glNewList(1); printf(“ id==%d\n”,n); // id == 0 // build list, draw with list

9. 9 Manifestation of Collision Problem Show image of four textured quads when the problem is present.

10. 10 Desired Result Show image of four textured quads when the problem is fixed.

11. 11 Resolving the Collision Problem New CR configuration file options: shared_textures, shared_display_lists, shared_programs When set to 1, beware of collisions in parallel submission. When set to 0, collisions resolved in parallel submission. Using shared_* to zero will enforce “unique retained mode identifiers” across all parallel submitters. Thanks, Brian!

12. 12 The Broadcast Problem What’s the Problem?: Geometry and textures from N application PEs is replicated across M crservers. Bumped into limits of memory and bandwidth. To Chromium, a display list is an opaque blob of stuff. Tilesort doesn’t peek inside a display list to see where it should be sent. Early performance testing showed two types of broadcasting: Display lists being broadcast from one tilesort to all servers. Textures associated with textured geometry in display lists was being broadcast from one tilesort to all servers.

13. 13 Broadcast Workarounds (Short Term) Help Chromium decide how to route textures with the GL_OBJECT_BBOX_CR extension. Don’t use display lists (for now). Immediate mode geometry isn’t broadcast. Sorting/routing is accelerated using GL_OBJECT_BBOX_CR to provide hints to tilesort; it doesn’t have to look at all vertices in a geometry blob to perform routing decisions. For scientific visualization, which generates lots of geometry, this is clearly a problem. Our volume rendering application uses 3D textures (N**3 data) and textured geometry (N**2 data), so the “heavy payload” data isn’t broadcast. The cost is immediate mode transmission of geometry (approximately 36KB/frame of geometry as compared to 160MB/frame of texture data).

14. 14 Broadcast Workarounds (Long Term) Funding for Chromium developers to implement display list caching and routing, similar to existing capabilities to manage texture objects. Lurking problems: “Aging” of display lists: the crserver is like a “roach motel”: display lists check in, but they never check out. Adding retained mode object aging and management to applications is an unreasonable burden (IMO). There exists no commonly accepted mechanism for LRU aging, etc. in the graphics API. Such an aging mechanism will probably show up as an extension. Better as an extension with tunable parameters than requiring applications to “reach deeply” into graphics API implementations.

15. 15 Scene Graph Issues and Performance Analysis Discussed in our 2003 PVG Paper (Monday afternoon). Our “parallel scene graph” implementation can be used by any parallel application, regardless of the type of parallel programming model used by the application developer. The “big issue” in sort-first is how much data is duplicated. What we’ve seen so far is about 1.8x duplication was required for the first frame (in a hardware accelerated volume rendering application). While the scene graph supports any type of parallel operation, certain types of synchronization are required to ensure correct rendering. These can be achieved using only Chromium barriers – no “parallel objects” in the scene graph are required.

16. 16 Some Performance Graphs Bandwidth vs. Traffic

17. 17 Parallel Scene Graph API Stuff Collectives rmPipeSetCommSize(), rmPipeGetCommSize() rmPipeSetMyRank(), rmPipeGetMyRank() Chromium-specific rmPipeBArrierCreateCR() Creates a Chromium barrier, number of participants is set by value specified using rmPipeSetCommSize() rmPipeBarrierExecCR() Doesn’t block application code execution. Used to synchronize rendering execution from rmPipeGetCommSize() streams of graphics commands.

18. 18 Demo Applications: Parallel Isosurface

19. 19 Demo Application: Parallel Volume Rendering

20. 20 Demo Application: Parallel Volume Rendering with LOD Volumes

21. 21 Conclusions We met our objectives: General purpose infrastructure for doing parallel visualization and hardware accelerated rendering on PC clusters. The infrastructure can be used by any parallel application, regardless of the parallel programming model. The architecture scaled well from one to 24 displays, supporting extremely high-resolution output (e.g, 7860x4096). We bumped into network bandwidth limits (not a big surprise). Display lists are still broadcast in Chromium. Please fund them to add this much-needed capability, which is fundamental for efficient sort-first operation of clusters.

22. 22 Sources of Software OpenRM Scene Graph: www.openrm.org Source code for OpenRM+Chromium applications: www.openrm.org. www.openrm.org Chromium: chromium.sourceforge.net.

23. 23 Acknowledgement This work was supported by the U. S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under SBIR grant DE-FE03- 02ER83443. The authors wish to thank Randall Frank of the ASCI/VIEWS program at Lawrence Livermore National Laboratory and the Scientific Computing and Imaging Institute at the University of Utah for use of computing facilities during the course of this research. The Argon shock bubble dataset was provided courtesy of John Bell and Vince Beckner at the Center for Computational Sciences and Engineering, Lawrence Berkeley National Laboratory.