1. Real-time Visualization of Massive Imagery and Volumetric Datasets Thesis Defense by Ian Roth

2. Overview Introduction Introduction Out-of-core software architecture Out-of-core software architecture –Asynchronous data I/O engine –Hierarchical caching Interactive visualization techniques Interactive visualization techniques –Viewing modes for image generation –Per-tile rendering methods –User interface and navigation modes Performance and benchmarking Performance and benchmarking Features and extensions Features and extensions Conclusions and future work Conclusions and future work

3. Kolam Software Package Developed at University of Missouri, Columbia Developed at University of Missouri, Columbia Version 1.0 by Joshua Fraser (2000 – 04) Version 1.0 by Joshua Fraser (2000 – 04) –Written in C, OpenGL and GLUT Version 2.0 by Ian Roth (2001 – 04) Version 2.0 by Ian Roth (2001 – 04) –Written in C++, OpenGL and Qt Additional code by Jared Hoberock (2001 – 02) Additional code by Jared Hoberock (2001 – 02) –Dataset I/O utilities Other contributors Other contributors –Dr. K. Palaniappan, Ian Scott, Dave Metts, Vidyasagar Chada, Mike Sullivan, Ryan Calhoun

4. Motivation Massive datasets Massive datasets –Too large to fit entirely in memory –Examples: IKONOS/MODIS panchromatic/multispectral satellite imagery IKONOS/MODIS panchromatic/multispectral satellite imagery CT/MRI/PET scans, Visible Human Project CT/MRI/PET scans, Visible Human Project Out-of-core approach Out-of-core approach –Require segmentation (tiling) for efficient roam operations –Require multiple resolutions (LODs) for efficient zoom operations

5. Definitions Image or Dataset Image or Dataset –Encapsulation of a pyramid file (disk cache) – I r (x) : image at resolution r, pixel coordinate x Layer Layer –A dataset with associated viewing parameters Associated colormaps, heightmaps, etc. Associated colormaps, heightmaps, etc. Stores x/y offset and scale for embedded datasets Stores x/y offset and scale for embedded datasets Colormap Colormap –Maps dataset values to RGB color values Heightmap Heightmap –Maps image pixels to height values

6. Kolam Software Architecture Engine Engine –Based on the workpile design pattern –Multiple worker threads process requests from application and perform data I/O operations Application Application –Displays data and GUI components –Determines which tiles are visible and at what resolution –Posts request which are handled by engines worker threads Application Extensions Application Extensions –Plugins: dynamically linked libraries –Provide additional GUI components, dataset I/O routines, image processing methods, image viewing and projection modes, etc.

7. Kolam Engine Asynchronously processes Asynchronously processes requests from application Proceedure: Proceedure: 1.Worker threads are initially idle 2.For each frame, the application determines visible tiles and posts requests 3.Each posted request awakens a worker thread 4.Repeat forever for each awakened thread 1. Worker thread attempts to pop request from queue; if no requests are on the queue, then exit loop. 2. Worker thread reserves a free slot in cache 3. Worker thread performs tile I/O 4. Application is notified of new data in cache 5.Worker thread enters idle state

8. Kolam Engine Versions Version 1.0 Version 1.0 –Each layer has its own resources Per layer: cache space, threads, tile lookup table, requests Per layer: cache space, threads, tile lookup table, requests –Fine for non-embedded datasets Version 2.0 Version 2.0 –Engine has global resources, shared by layers Engine: cache space, threads, requests Engine: cache space, threads, requests Per layer: tile lookup table Per layer: tile lookup table –Allows active layers to seize resources from inactive layers

9. Hierarchical Caching Three levels: Three levels: –Disk cache Stores tiles on disk Stores tiles on disk Original source may be Original source may be row-major order image file on disk or tiles streamed over network –Main memory cache Stores fixed number of tiles in memory Stores fixed number of tiles in memory –Texture cache Stores fixed number of tiles on graphics hardware Stores fixed number of tiles on graphics hardware Passed directly to real-time display component Passed directly to real-time display component

10. Disk Cache Pyramid file format Pyramid file format –Additional resolutions appended to end of image file –Requires additional space (no more than 2 original) 1D image takes 2 space of original 1D image takes 2 space of original 2D image takes 4/3 space of original 2D image takes 4/3 space of original 3D image takes 8/7 space of original 3D image takes 8/7 space of original Wavelet encoded file format Wavelet encoded file format –Additional resolutions embedded in image –Requires no additional space –Complex to encode/decode, especially in n-D –Examples: JPEG 2000 JPEG 2000 MrSID (LizardTech) MrSID (LizardTech)

11. Main Memory Cache Asynchronous I/O Asynchronous I/O –Multiple read/write threads –Allows display thread to continue working while data is loading –Spreads out I/O lag time over shorter intervals Caching: allocates/deallocates memory for tile data Caching: allocates/deallocates memory for tile data –Cache methods available to application developers checkout tile, checkin tile, add request, query tile/request, invalidate tile/request checkout tile, checkin tile, add request, query tile/request, invalidate tile/request –Variations of the checkout method basic checkout, checkout first available, wait for tile basic checkout, checkout first available, wait for tile Paging: swaps old data for new Paging: swaps old data for new –Temporal strategies – view projection independent LRU priority queue – O(# of layers) LRU priority queue – O(# of layers) –Spatial strategies – view projection dependent Brute force distance calculation – O(# of cached tiles) Brute force distance calculation – O(# of cached tiles) Modular ROI array – O(1) Modular ROI array – O(1) Graph traversal algorithms – O(# of cached tiles) Graph traversal algorithms – O(# of cached tiles)

12. Texture Cache Similar to main memory LRU cache Similar to main memory LRU cache –Replace allocator object Synchronous I/O Synchronous I/O –Adequate for single data display window Can be implemented asynchronously Can be implemented asynchronously –Useful for multiple displays One-to-one correspondence between thread and context One-to-one correspondence between thread and context Each display is essentially synchronous Each display is essentially synchronous Frame rate not bounded by longest display time Frame rate not bounded by longest display time Implemented by OpenRM Scene Graph Implemented by OpenRM Scene Graph

13. Interactive Visualization Components Viewing modes Viewing modes –Orthogonal projection –Oblique projection (separate source tree) –Spherical projection –Arbitrary geometry projection (future work) –Volumetric visualization (separate source tree) Tile renderers Tile renderers –Raster renderer –Texture renderer –Terrain renderer –Ray casting renderer (separate source tree) Navigation modes Navigation modes –Roam, zoom, and tilt –Fly mode (separate source tree) –Embedded dataset positioning –Image crop (future work) –Geometry and terrain manipulation (future work)

14. Visualization Running Time Measured by number of tiles processed per frame Measured by number of tiles processed per frame Definitions Definitions – n(I) : total number of tiles in image I – m(I, t) : number of visible tiles in image I at time t Processing O(n) tiles per frame is too much Processing O(n) tiles per frame is too much Processing (m) tiles per frame is the goal Processing (m) tiles per frame is the goal Analytic solution: Analytic solution: – O(m) : single resolution solution – O(m + lg n) : multiple resolution solution –Not suitable for explicitly defined projections, i.e. triangular meshes –View projection dependent, often difficult to implement Iterative solution: Iterative solution: – O(m lg m + lg n) : single/multiple resolution solution –Relies on quadtree/octree/scenegraph traversal to iteratively refine solution from coarse to fine resolutions –View projection independent, relatively east to implement

15. Visible rectangular region width/height: Visible rectangular region width/height: Visible rectangular region x/y offset: Visible rectangular region x/y offset: Resulting region R is in tile coordinates at scale Resulting region R is in tile coordinates at scale Orthogonal Projection Analytic Solution t : 2D translation/position vector t : 2D translation/position vector in original resolution pixel coords in original resolution pixel coords z : pixel zoom factor z : pixel zoom factor {w T, h T } : tile width/height {w T, h T } : tile width/height {w V, h V } : viewport width/height {w V, h V } : viewport width/height Global image resolution: Global image resolution: Scaling matrix (to tile coordinates): Scaling matrix (to tile coordinates):

16. Oblique Projection Analytic Solution Single resolution solution Single resolution solution –Intersect view frustum planes with image plane Result is a polygon in the image plane Result is a polygon in the image plane Clip polygon to image boundaries Clip polygon to image boundaries –Run polygon fill algorithm to find tiles in polygon at a fixed resolution Flood fill algorithm (recommended) Flood fill algorithm (recommended) –First use Bresenham to discretize polygon to tile coords Scanline fill algorithm Scanline fill algorithm Multiple resolution solution Multiple resolution solution –First subdivide frustum into regions Calculate distance from eye p when z = 1 Calculate distance from eye p when z = 1 : FovY/2, r : resolution, d : distance from eye : FovY/2, r : resolution, d : distance from eye Assume pixel aspect ratio = 1 Assume pixel aspect ratio = 1 d and p both measured in pixels d and p both measured in pixels –Perform single resolution solution for each section

17. Oblique Projection Iterative Solution Quadtree traversal Quadtree traversal Determine if tile is visible Determine if tile is visible –View-frustum culling test Performed in either R 3 or canonical view volume (CVV) space Performed in either R 3 or canonical view volume (CVV) space Test 4 tile vertices w/ 6 frustum planes (24 tests) Test 4 tile vertices w/ 6 frustum planes (24 tests) Test 4 tile edges w/ 6 frustum planes (24 tests) Test 4 tile edges w/ 6 frustum planes (24 tests) Test 12 frustum edges w/ tile plane (12 tests) Test 12 frustum edges w/ tile plane (12 tests) Maximum of 60 tests per tile Maximum of 60 tests per tile Determine tile resolution Determine tile resolution –Project to viewing plane and calculate quadrilateral area –Caclulate resolution at sample points

18. Spherical Projection Analytic Solution Convert Euler coords to spherical Convert Euler coords to spherical – u : [0, 2 ], v : [- /2, /2] Intersect view frustum planes with sphere Intersect view frustum planes with sphere –Result will be up to 6 circles in 3D space n : unit normal of plane P, p : any point on P n : unit normal of plane P, p : any point on P c c/s : circle/sphere center, r c/s : circle/sphere radius c c/s : circle/sphere center, r c/s : circle/sphere radius Find distance d from c s to plane along n Find distance d from c s to plane along n Project circles of intersection onto image plane Project circles of intersection onto image plane –Parametric equations for circle and sphere t : [0, 2 ], x : vertex in R 3 t : [0, 2 ], x : vertex in R 3 n : unit normal of circle plane, p : up vector n : unit normal of circle plane, p : up vector a = n p, b = n a a = n p, b = n a –Solve for u and v –For each circle Pick sample t values Pick sample t values u(t) and v(t) are points in image plane u(t) and v(t) are points in image plane –Compute polygon intersections Tesselate non-convex polygons Tesselate non-convex polygons Perform pairwise convex polygon intersection tests Perform pairwise convex polygon intersection tests

19. Spherical Projection Iterative Solution Determine visible tiles Determine visible tiles –Tiles no longer planar, can be composed of many polygons Use bounding spheres to approximate curved tile shapes Use bounding spheres to approximate curved tile shapes Similar to vertex/frustum intersection test Similar to vertex/frustum intersection test –Hidden surface removal Back-face culling methods dont work on curved surfaces Back-face culling methods dont work on curved surfaces Additional clipping plane to divide sphere into front- and back- facing regions Additional clipping plane to divide sphere into front- and back- facing regions –Orthogonal to vector v –Need to determine distance from eye Determine tile resolution Determine tile resolution –Projecting and calculating area is difficult for curved tiles Overlapping regions not accounted for Overlapping regions not accounted for –Caclulate resolution at sample points

20. Arbitrary Geometry Projection Iterative Solution Image projected onto arbitrary triangular mesh Image projected onto arbitrary triangular mesh Similar to spherical projection Similar to spherical projection –Requires mapping from mesh vertices to u/v coords Determine visible tiles Determine visible tiles –Use hierarchy of bounding spheres –Hidden surface removal ID image : a lookup table of visible triangles ID image : a lookup table of visible triangles Determine tile resolution Determine tile resolution –Caclulate resolution at sample points

21. Volumetric Visualization Iterative Solution Octree traversal Octree traversal Determine if tile (cube) is visible Determine if tile (cube) is visible –View-frustum culling test Performed in either R 3 or canonical view volume (CVV) space Performed in either R 3 or canonical view volume (CVV) space Test 8 tile vertices w/ 6 frustum planes (48 tests) Test 8 tile vertices w/ 6 frustum planes (48 tests) Test 12 tile edges w/ 6 frustum planes (72 tests) Test 12 tile edges w/ 6 frustum planes (72 tests) Test 8 frustum vertices w/ 6 tile planes (48 tests) Test 8 frustum vertices w/ 6 tile planes (48 tests) Test 12 frustum edges w/ 6 tile planes (72 tests) Test 12 frustum edges w/ 6 tile planes (72 tests) Maximum of 240 tests per tile Maximum of 240 tests per tile Using bounding spheres only 6 tests required, but less accurate Using bounding spheres only 6 tests required, but less accurate Optimization: use spheres during octree traversal, use cubes on leaf nodes Optimization: use spheres during octree traversal, use cubes on leaf nodes Determine tile resolution Determine tile resolution –Project to viewing plane and calculate convex hull (polygon) area –Caclulate resolution at sample points

22. Raster Renderer Renders images directly to framebuffer Renders images directly to framebuffer Advantages Advantages –Simple to implement –No additional texture/VRAM cache required –No additional overhead from texture load time –Tile data changes are applied instantly (on next display update) No need to flush texture cache No need to flush texture cache Disadvantages Disadvantages –Can only be used in orthogonal viewing mode –Not all affine transformations supported Rotations and shearing Rotations and shearing Only 2D transformations Only 2D transformations –Can be slow on some graphics hardware

23. Texture Renderer Stores tile data in texture/VRAM memory on graphics hardware Stores tile data in texture/VRAM memory on graphics hardware Advantages Advantages –Often faster than rendering directly to framebuffer –Allows any affine transformation to be applied to a tile –Allows hardware-accellerated interpolation to be applied when rendering –Can be used by all viewing modes Volumetric viewing mode requires 3D texture support, otherwise must use ray casting renderer Volumetric viewing mode requires 3D texture support, otherwise must use ray casting renderer Disadvantages Disadvantages –Wasted space when compositing color channels Luminance component always stored in red channel in OpenGL Luminance component always stored in red channel in OpenGL Must use RGB color textures for color masking to be effective Must use RGB color textures for color masking to be effective –Need to flush texture cache when tile data changes or global filters are applied Texture cache Texture cache –Similar to main memory LRU cache Synchronous rather than asynchronous Synchronous rather than asynchronous Can reuse some code by replacing the allocator object Can reuse some code by replacing the allocator object Texture borders Texture borders –Use OpenGL GL_CLAMP_TO_EDGE texture parameter Widely supported, produces artifacts when z  1 Widely supported, produces artifacts when z  1 –Use OpenGL texture borders Not widely supported, often slow Not widely supported, often slow Tradeoff between efficiency and storage Tradeoff between efficiency and storage –Simulate texture borders (not implemented) Tile size: 2 n – 2, Texture coords: [1 / 2 n, (2 n – 1) / 2 n ] Tile size: 2 n – 2, Texture coords: [1 / 2 n, (2 n – 1) / 2 n ] Requires major change to file format Requires major change to file format Tradeoff between efficiency and storage Tradeoff between efficiency and storage

24. Terrain Renderer Similar to texture renderer Similar to texture renderer –Also converts a heightmap to visible terrain –Rendering a single tile requires neighboring heightmap tiles to be available –Two possible implementations Triangular mesh terrain Triangular mesh terrain –Problems with seams across tiles of disparate resolutions Bump-mapped terrain (Ian Scott) Bump-mapped terrain (Ian Scott) –Horizon still looks flat –Requires fragment shader support (GPU) –Requires additional texture memory

25. Ray Casting Renderer Used by volumetric view mode Used by volumetric view mode Can be implemented on GPU Can be implemented on GPU –Graphics hardware must support 3D textures and fragment shaders Proceedure Proceedure 1.Project tile onto viewing plane 1. To find regions of plane that intersect tile 1.Intersect 6 tile planes with viewing plane 2.Map polygon vertices to voxel coordinates To find regions of plane that do not intersect tile 1.Project 8 tile vertices onto viewing plane 2. Compute convex hull of result (a polygon) 3. Traverse each pixel of polygon in voxel space –Flood fill algorithm –Scanline fill algorithm (recommended) 2.For each pixel of polygon 1. If pixel coordinate in voxel space intersects tile 1.Tile voxel coordinate = pixel coordinate in voxel space modulo tile width/height/depth 2.Can Otherwise 1.Tile voxel coordinate = intersection of ray from eye through pixel with frontmost tile plane 2. Traverse ray from eye through pixel starting with voxel coordinate 1.Accumulate voxel data along ray

26. Navigation Modes Roam, zoom and tilt Roam, zoom and tilt –Drag image using combination of mouse buttons Fly mode Fly mode –Constant velocity based on ray from screen center to mouse cursor Embedded dataset positioning Embedded dataset positioning –Position, resize and rotate a single dataset Image crop Image crop –Drag edges of dataset to change size without stretching Geometry manipulation Geometry manipulation –Move individual vertices of geometry –Select group of vertices to move at once –Use mass-spring system to pull connected vertices –Use finite element method to simulate objects with physical properties

27. Optimizations Hiding disk latency Hiding disk latency –Search upward through quadtree for equivalent coarser resolutions –Never see blank tiles, see blurred tiles instead –Implemented in checkout first available cache checkout method Maintaining constant FPS Maintaining constant FPS –Maintain maximum number of visible tiles –Replace group(s) of 2–4 tiles with a single coarser tile while roaming and/or zooming Prioritizing requests Prioritizing requests –Maintain list of visible tiles for current and previous frames –Compare lists Remove requests for tiles that are no longer visible Remove requests for tiles that are no longer visible Move requests for tiles that are no longer visible to lower priority queue Move requests for tiles that are no longer visible to lower priority queue Prefetching Prefetching –Load constant size radius of tiles from around ROI when idle –Load adjacent tiles from coarser/finer resolutions when idle –Use motion vectors from recent user navigation to predict next ROI

28. Performance Interactivity Interactivity –FPS & # of dropped requests w/ varying number of layers –FPS & # of dropped requests w/ varying cache sizes –FPS w/ varying # of (embedded/non-embedded) layers –Hardware-specific tests FPS w/ and w/o colormaps FPS w/ and w/o colormaps FPS of raster vs. texture mode FPS of raster vs. texture mode Cache efficiency Cache efficiency –I/O latency w/ and w/o compression –Paging time analysis File format efficiency File format efficiency –Compression ratios

29. Performance – Multithreading Avg. FPS 1 Thread 69.6 2 Threads 48.6 4 Threads 54.4 8 Threads 47.5 16 Threads 52.5 32 Threads 47.4 Avg. Requests Dropped 1 Thread 6.93 2 Threads 0.574 4 Threads 2.69 8 Threads 0.0 16 Threads 1.98 32 Threads 0.0164

30. Performance – Cache Size Avg. FPS 8 MB Cache 75.5 16 MB Cache 74.6 32 MB Cache 72.8 64 MB Cache 68.5 128 MB Cache 68.8 256 MB Cache 67.0 Avg. Requests Dropped 8 MB Cache 13.7 16 MB Cache 10.9 32 MB Cache 10.5 64 MB Cache 5.90 128 MB Cache 3.66 256 MB Cache 1.54

31. Application Extensions Plugins: dynamically linked libraries Plugins: dynamically linked libraries –Plugins have access to all application data –Plugins can attach processing routines to predefined breakpoints in the code –A list of only the attached routines is processed at each breakpoint Typical extension categories Typical extension categories –Image/dataset file formats –GUI components –Viewing modes –Tile renderers –Navigation modes

32. Example Plugins Apply Colormap Apply Colormap –Provides GUI control similar to layer dialog –Uses prerender breakpoint to apply colormap to image before display Histogram Display & Enhancement Histogram Display & Enhancement –Provides GUI control w/sliders –Uses layer initialized breakpoint to spawn new thread to compute histogram of overview –Uses wait for tile tile checkout method to avoid busy waiting –Uses prerender breakpoint to apply histogram enhancement to image before display

33. Conclusions and Future Work Conclusions Conclusions –A stable and basic visualization tool –Many features yet to be implemented Need funding for full time programmers! Need funding for full time programmers! SBIR Proposal SBIR Proposal –Extension to n -dimensional data View slices of both sparse and dense datasets View slices of both sparse and dense datasets –Adoption of HDF5 file format Chunking, grouping, n -D support Chunking, grouping, n -D support Possibly provide our own extensions to the format Possibly provide our own extensions to the format –Image projection onto arbitrarily large triangular meshes Unified framework for sphere mapping, terrain, etc. Unified framework for sphere mapping, terrain, etc. Fast image registration Fast image registration Incorporate modeling and deformation tools Incorporate modeling and deformation tools –Network streaming of tile data Similar to Keyhole software Similar to Keyhole software

34. Similar Large Data Visualization Software ESRI – ArcGIS ESRI – ArcGIS –Full-featured GIS package –May not handle large images (bigger than available RAM/virtual memory) Keyhole – EarthViewer Keyhole – EarthViewer –Network streaming of remote sensing & vector data LizardTech – GeoExpress View LizardTech – GeoExpress View –MrSID file format & image viewing software –Achieves high compression ratios at expense of quality Sensor Systems – Remote View Sensor Systems – Remote View –Image processing and analysis software used in DoD –Doesnt handle large images (bigger than available RAM/virtual memory) EPFL – Real-time Visible Human Navigator EPFL – Real-time Visible Human Navigator –Only works on visible human dataset SGI – OpenGL Performer/Volumizer SGI – OpenGL Performer/Volumizer –C/C++ API for 2D/3D large data visualization –IRIX/Linux platforms only Kitware – VTK (Visualization Toolkit) Kitware – VTK (Visualization Toolkit) –C++/Tcl/Java/Python API for 2D/3D large data visualization –Large mesh dataset support based on multiple LODs, no inherent datset segmentation

35. Screenshot – Display Windows

36. Screenshot – GUI & Colormaps

37. Animation Embedded Datasets Embedded Datasets Layers & Colormaps Layers & Colormaps Spherical View Mode Spherical View Mode

38. IBM T221 Display