Image Fusion In Real-time, on a PC

Goals Interactive display of volume data in 3D –Allow more than one data set –Allow fusion of different modalities (e.g. CT+PET) –Allow simultaneous display of all data sets

Considerations Must run on consumer level hardware –APIs such as OpenGL™ Volumizer already available on High End machines ($$$$$) –Must run at interactive frame rates –Must allow flexibility in visualization of data

Requirements Allow for: –Arbitrary viewpoints –Arbitrary “slicing” to show internal detail –Flexible transfer functions User adjustable for: –Colors –Transparency –Arbitrary mixing of multi modal data sets

Problems More $$$$ = More flexibility, so: Consumer cards have less than 128 MB of shared memory (texture, frame buffer etc) PCs have more main memory but it is slow Don’t have X units working together –Need processing power, (lots of it)

Workarounds Limited memory on video card –Roam around a high resolution data set by selecting smaller areas that fit in available memory Still slow, because when the viewpoint moves new data needs to be loaded (from main memory) –Down sample data to fit in memory Loss of detail

Workarounds –Compress textures Implementations will allow for a 8:1 compression of textures Will quality be affected? How much? Will it compromise flexibility? How much? Processing power needed –Use programmability of newer “GPU”s to leverage available high speed memory –Do transfer functions on the GPU? In parallel?

Details Allow user to choose blending between the data sets –Assign colors, transparencies and mixing values –Each data set must have its own transfer function(s) Registration –Not handled

Rendering How? –Volume data  Volume textures (3D Textures) –“Project” onto 2D screen by overlaying slices –Slices are perpendicular to the screen –Slices are cut from the volume at fine enough intervals and blended together to produce a single viewpoint image –Repeat for different views (rotations/scale) etc

Rendering More datasets –Could render image for each data set and overlay on top (multipass = slow) –Render once, applying both textures at once Limited to 2 textures on most cards –Use “shaders” to program the GPU + Very flexible, 2+ textures at once - Flexibility?

Rendering Lighting –Current rendering mode does not help with lighting –Must be done at a lower level Surfaces –No real “surfaces” generated, so lighting is hard to do, must be done at the voxel level –Lighting at voxel level, will show appearance of “surfaces”

Conclusions Limited Memory & Large data sets Want lighting & lack of “surfaces” But we have a programmable GPU to our advantage –Dataset manipulation in fast video memory = Faster rendering Alternative rendering methods? –Most are CPU bound

References – – – – – –