1/13 A MJPEG Encoder for the NOW to visualize and navigate on data- intensive scenarios José María González

2/13 The problem Problem: How to be able to monitor through interactive visualization enormous amounts of image data that are being treated in a remote (parallel) machine from an external box that controls the data process behavior. Two issues: –link from the parallel machine to the controller is usually low bandwidth (at least << parallel machine internal links) –display performance

3/13 The solution Solution: On-the-fly degradation (“downgrade”) of the data trading off visualization quality to permit the adaptation to the control box display performance and link bandwidth. Some example numbers: –ATV: 1920x1080, 30 fps, 1.5 B/pixel, 93 MB/s, cannot be pumped into a normal LAN –CIF at 30 fps, 4.6 MB/s neither –ATV with MJPEG 20/1 compression (q~75) plus frame decimation (7.5 fps) get possible bandwidth = 1 MB/s

4/13 More issues Interactivity Scalability (no centralized coordination point) Flexibility Solution: Parallelize the degradation process!

5/13 Parallelization

6/13 Parallel Architecture

7/13 An interesting application A virtual-reality system that permits a remote user to navigate on very large image assets (satellite, astronomical, geographical, medical, …) interactively. The user asks for images depending on its position. The system reads the data needed and encodes it in parallel, obtaining a video stream that fits in the user link.

8/13 Implementation Three kinds of processes: –readers –encoders –multiplexer Use of River, a data-flow programming environment and an I/O substrate for clusters of computers.

9/13 Implementation (cont.) Network of Workstations River ReaderEncoderMux. Controlling scripts River modules

10/13 Performance results Disk performance ~ IJG JPEG Encoder performance ~ 2-3 MB/s n/n/1Configuration n/n/1 Be careful with file caches! (cleaner process)

11/13 Performance results (cont.) TBD ATV CIF QCIF

12/13 Conclusions Efficient, scalable MJPEG encoder (1 MB/s of reading throughput per pair of nodes) Use of a parallel architecture to permit visualization of huge amounts of image data is an interesting tool to permit interactive monitoring of parallel machines. Design and development of a virtual-reality system based on the parallel MJPEG encoder

13/13 Future work Output bandwidth adaptive to remote link MPI implementation instead of River Display app. problems: RTP / MPEG Prefetching policies Add DQ Finish the implementation of the system (some parts are still simulated).