VOMegaPlot Efficient Plotting of Large VOTable Datasets

VOPlot  VOPlot is a tool for visualizing astronomical data that is available in the VOTable format.VOTable  VOPlot reads the xml file in order to load entire data into memory and then processes it to draw various types of plots.  This approach of loading the entire data into memory cannot be used for VOTable files that are very large.

Approach for VOMegaPlot  VOMegaPlot preprocesses the xml file to create intermediate files which are subsequently used for plotting.  Entire data is divided into fixed sized blocks and individual blocks are loaded into memory thus reducing the memory requirement.  The number of intermediate files created is equal to the number of columns present in the xml file.

Pre-processing operation Creation of array blocks Col 1Col m ……. 1 2 n.... Col 2 File 2 ……. Original xml file with m columns and n rows Block 1 File 1 Block 2 Block k Block 1 Block 2 Block k File m Block 1 Block 2 Block k Intermediate files on disk

Algorithm for drawing a scatter plot 1)Input the columns to be plotted, say A vs. B 2)Load a set of corresponding blocks for both columns, A and B. 3)Take corresponding data elements from both the blocks and plot them. 4)After plotting all the points, discard the blocks. 5)If there exist more blocks of data repeat step 2, else stop.

Advantages  The complexity for plotting is to O(2n) where n is the no. of rows. This complexity is independent of the no. of columns in the xml file.  If the user has to plot only a subset of data (as in case of zoom operation) then there exists another set of files which can be used for this purpose.

Dealing with subset of data  Data for every column is stored in an indexed fashion.  This helps in accessing the subset of data without having to go through the entire set of data.  As a result, operations like zoom become much faster.

Pre-processing operation Creation of tree blocks Col 1Col m ……. 1 2 n.... Col 2 Indexed File for col Indexed File for col ……. Original xml file with m columns and n rows Intermediate files with indexed data 0.1– – Indexed File for col m

Pre-processing operation Creation of tree blocks (contd) … Indexed file for a column

Results TychoTycho-2UCAC2 Data size1 million rows and 56 columns 2.5 million rows and 32 columns 48.3 million rows and 9 columns Pre- processing time 18 minutes30 minutes3 hours 26 minutes Plotting time for scatter plot 9 seconds22 seconds5 minutes 46 seconds

Features of VOMegaPlot  Scatter Plot with zoom, reverse axis and logged axis  Projection Plot  Density Plot  Histogram

Scatter Plot Tycho-1 catalogue ( RA vs. Vmag)

Density Plot Tycho-1 catalogue ( RA vs. Vmag)

Density Plot Tycho-2 catalogue ( DEC vs RA)

Scatter Plot UCAC2 Catalogue (2m_J vs. U2Rmag)

Density Plot UCAC2 Catalogue (2m_J vs. U2Rmag)

Future Enhancements  Support for reading data stored in binary format  Block level compression while creating intermediate files  Client Server version

References  VOTable  VOPlot  VOMegaPlot tm tm  IUCAA  Persistent Systems Pvt. Ltd.

Sample VOTable Back