1. Ultra-Scale Visualization with Open-Source Software Berk Geveci Kitware Inc.

2. Background

3. Kitware Is A software company creating open-source collaboration platforms which are used globally for research teaching commercial application. This software is created by internationally recognized experts in extended communities using a rigorous, quality-inducing software development process.

4. Technical Portfolio

5. Software Process

6. Heart and Soul: Software Quality Process

7. In The Beginning There Was VTK

8. From Ohloh: Very large, active development team: Over the past twelve months, 66 developers contributed new code to VTK. This is one of the largest open-source teams in the world, and is in the top 2% of all project teams on Ohloh. VTK Development Team and many others...

9. VTK Development Team

10. Then Came ParaView

11. ParaView Architecture

12. Support for Large Displays

13. ParaView is VTK ParaView is developed by the VTK team The ParaView team develops in the VTK repository : all development benefits the whole community ParaView leverages all features of VTK

14. ParaView is a Framework ParaView extends VTK to provide: Client-server computing State management Python modules Application/GUI framework ParaView framework can be used to develop other applications ParaView can be embedded in other application and frameworks

15. Short Demo

16. VTK and ParaView Funding Army Aeroflightdnymics Directorate National Alliance for Medical Computing and many others...

17. Problem Statement

18. The Good

19. Ability to Compute

20. Ability to Communicate

21. Remote Access

22. The Bad Ability to Compute >> Ability to Transfer >> Ability to Store/Read

23. State of Large Data Vis

24. Parallel Visualization

26. Visualization Resources LANL Vis Cluster (2001): 128 dual-core nodes with NVIDIA Quadro FX 540 Sandia Red Rose (2005): 264 dual-core nodes with NVIDIA Quadro FX 3400 TACC Longhorn (2009): 256 8 core nodes with 2 NVIDIA Quadro FX 5800s

27. General Purpose Tools EnSight ParaView VisIt FieldView …

28. Specialized Tools VAPOR ViSUS...

29. 1 billion cell asteroid detonation simulation (AMR) ½ billion element weather simulation (rectilinear) 150 million element fire simulation (unstructured)

30. Where to Next?

31. Large Data, Small Bandwidth and Small Vis Resources Vis on the supercomputer Batch Interactive Co-Processing Multi-Resolution Streaming

32. Ability to Compute

33. Vis on Supercomputer

34. Compiling Parallel IO Scalability Software Rendering Compositing Client/server Resource allocation

35. Client/server Resource Allocation Interactive on Supercomputer

36. ParaView on Supercomputers Blue Gene L and P Cray Xt 3, 4 and 5 AIX Linux Visualization of VPIC results on Kraken (image courtesy of Bill Daughton, LANL)

37. Co-Processing

38. 11

39. Run-Time Visualization and Setup

40. Objectives Develop an extensible and flexible co- processing library Develop CFD specific analysis and visualization algorithms Develop tools for visualizing, analyzing and managing extracts Integrate run-time visualization with co- processing

41. Workflow

46. Setup Use ParaView Load and visualize extracts or previous simulation output for context Interactively build processing/visualization pipeline using available algorithms Save configuration file for co- processing library

47. Library Architecture Based on VTK/ParaView C, Fortran and Python bindings Built-in Python interpreter (optional) Full demand-driven pipeline Distributed computing with MPI Extract generation Rendering (off-screen with Mesa)

48. Connection with the Simulation simulation_initialize() coprocessor_initialize() for t in time_steps: do_compute() if coprocess_needed(): do_coprocess() coprocessor_finalize() simulation_finalize()

49. Connection with the Simulation Adapter or converter? Simple memory layout (structured data) We used converters for vectors, adapter for scalars Can we use adapters for both? Complex memory layout (unstructured grid connectivity) We used converters Possible to use generic dataset framework

50. Decision Making Based on Features Steer processing based on emerging features For example: Change extraction parameters Extract more or less often Demand-driven pipeline help Python bindings are ideal

51. Simulations – the More the Merrier Overflow (overset curvilinear) Phasta (unstructured) Acusolve (unstructured) Helios (unstructured - AMR hybrid) CTH (AMR) NPIC (structured)...

54. Run-Time Visualization and Setup

55. What Is Different? Extensible and flexible VTK API Python Scalable Leverages a larger system (ParaView) Tools for visualizing, analyzing and managing extracts

56. Management and Visualization of Extracts Extracts are saved multi- resolution Viewer streams extracts based on camera position Extracts are registered with a database (with meta-data) for data management

57. Multi-Resolution Streaming

58. Data-Parallel Processing

59. Streaming

65. Prioritized Streaming

72. Adaptive Streaming

81. Collaboration and Web Visualization

82. Remote Access

83. Collaboration

84. Web Visualization

85. Architecture

86. Comparative Visualization

87. Engine inlet with aggressive bend Active control Parameters: duct geometry, location of jet, gap width, jet waveform, jet frequency,...

88. 180180180180

89. Ouch!

90. Data and Meta-Data Management

91. Batch Analysis

93. Loading Meta-Data in ParaView

94. Analysis and Subsetting of Meta- Data

96. Aperture for Data- Analysis

97. Conclusions

98. Compute power is growing fast Network speed is growing fast Researchers can simulate more Visualization resources are not scaling (budget) It is about insight Multiple solutions: In-situ Multi-resolution streaming Vis on supercomputer It is not always about size: Collaboration Ensemble analysis Need customized solutions Need frameworks to make it easy to build them Multiple slides from these

99. Ability to Compute and Communicate

100. Bottlenecks

101. We need multiple solutions!

102. Interactive on Supercomputer

103. Co-Processing

104. Multi-Resolution Streaming

105. It’s About Collaboration

106. We need frameworks to build specialized tools

107. General Purpose Tools EnSight ParaView VisIt FieldView …

108. Specialized Tools VAPOR ViSUS...

109. ParaView is a Framework ParaView is VTK

110. The End