Laurent Castanié (Earth Decision / Paradigm – INRIA Lorraine – Project ALICE) Christophe Mion (INRIA Lorraine – Project ALICE) Xavier Cavin (INRIA Lorraine – Project ALICE) Bruno Lévy (INRIA Lorraine – Project ALICE) Distributed Shared Memory for Roaming Large Volumes

2 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

3 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

4 Reservoir scale volume 300x400x400 (~50 MB) Km 2 Targeted ROI 1000x1000x1000 (~1 GB) Targeted volume 4000x5000x5000 (~100 GB) Reservoir scale ROI 100x200x200 (~4 MB) Targeted ROI ~ 250x typical reservoir scale ROI REGIONAL SCALE Km 2 Introduction Interpretation scales in Oil and Gas EP

5 Introduction OOC visualization on a single workstation 8% 6% 0.5% 100 GB Data Volume (Disk) 512 MB Graphics Card (V-RAM) 8 GB Workstation (RAM)

6 Introduction OOC visualization on a single workstation Bhaniramka and Demange, IEEE VolVis 2002, OpenGL Volumizer Plate et al., VISSYM 2002, Octreemizer Castanie et al., IEEE Visualization 2005 VolumeExplorer (coupling OOC visualization and data processing) Probe-based roaming systems with LRU volume paging

7 Efficient solution up to GB, however: ROI size is limited to the amount of graphics memory available; Performance decreases rapidly when the size of data on disk increases over 30 GB. => How to SCALE our solution up to GB ? Distributed Hierarchical Cache System (DHCS) on COTS cluster Introduction OOC visualization on a single workstation

8 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

9 SLAVE NODE MASTER NODE 1. Segmentation 2. Distribution 3. Rendering 4. Composition Distributed volume rendering Sort-last parallel volume rendering

10 Distributed volume rendering Pipelined binary-swap compositing P0P0 P1P1 P2P2 P3P3 Result on MASTER node Ma et al., IEEE CG&A 1994 Binary-swap compositing Cavin et al., IEEE Visualization 2005 Cavin et al., Eurographics PGV 2006 DViz pipelined implementation

11 ROI size several GBs => ~15-20 fps One virtual graphics card 8 GB DViz Master Distributed volume rendering Pipelined binary-swap compositing 16 nodes with GeForce 6800 ULTRA MB

12 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

13 Very low disk to memory bandwidth Faster transfers through the network? ? Distributed data management Limited disk to memory bandwidth Graphics Card (V-RAM) Workstation (RAM) Data Volume (Disk) Gigabit Ethernet Network ~1 GB/s~50 MB/s

14 4x faster transfers through the network 50 MB/s Network Disk 220 MB/s Distributed data management Disk Vs Network bandwidth 120 MB/s500 MB/s

15 Fully dynamic memory state that must be kept up-to-date ~50 MB/s ~220 MB/s 11 2 Distributed data management Our fully dynamic implementation (DHCS) 8 GB

16 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

17 30 copies of the Visible Human data set = 5580x5400x3840 ~ 107 GB ROI 1000x1000x1000 ~ 1 GB => Real-time rendering and volume roaming at full resolution at 12 fps on average on a 16-node cluster Results Real-time rendering and volume roaming Full resolution volume roaming Full resolution volume rendering

18 Outline Introduction Large volumes in the Oil and Gas EP domain Previous work: single-workstation cache system COTS cluster solution: DHCS Distributed volume rendering Distributed data management Real-time roaming in gigantic data sets with DHCS Conclusions

19 Volume visualization of ~100 GB Volume roaming a ROI of several GBs Cluster-based hierarchical cache system Distributed volume rendering Distributed data management Compression techniques Better load balancing of the communications on the network Pre-fetching strategies to hide disk access Other use cases: Combination of multiple attributes of 100 GB each Real-time full resolution volume slicing at slices per second Conclusions

20 Acknowledgements This work involved and has been supported by: Earth Decision (now part of Paradigm) LORIA / INRIA – Project ALICEhttp://alice.loria.frhttp://alice.loria.fr DVizhttp:// Region Lorraine (CRVHP) GOCAD consortiumhttp://

21 Distributed data management System architecture

22 Distributed data management System flow chart

23 Optimal transfer rate with GL_BGRA format Distributed data management Memory to graphics memory bandwidth

24 Distributed data management GL_BGRA texture packing

25 GL_RGBA8 only CPU bound up to 32x32x16 Distributed data management Frame rate against the brick dimensions CPU boundGPU bound CPU boundGPU bound

26 2xGigabit Ethernet and 512 KB pages => 220 MB/s 220 MB/s114 MB/s Distributed data management Network bandwidth

27 8 GB ~220 MB/s Limited by the global amount of main memory on the cluster Distributed data management Classical static DSM implementation 8 GB Initialization with a static resident set on each node DeMarle et al. 2004

28 Each node with its own buffer state 1 buffer transferred at step 0 2 buffers transferred at step 1 2 i buffers transferred at step i Each node with the entire cluster state Distributed data management Binary all-to-all communication