Presentation is loading. Please wait.

Presentation is loading. Please wait.

End-to-End XML Data Description: XDF PI. Ed Shaya Brian Thomas, Zhenping Huang, James Gass, James Blackwell, Gail Schneider (RITSS) NASA ATR: Cynthia.

Similar presentations


Presentation on theme: "End-to-End XML Data Description: XDF PI. Ed Shaya Brian Thomas, Zhenping Huang, James Gass, James Blackwell, Gail Schneider (RITSS) NASA ATR: Cynthia."— Presentation transcript:

1

2 End-to-End XML Data Description: XDF PI. Ed Shaya Brian Thomas, Zhenping Huang, James Gass, James Blackwell, Gail Schneider (RITSS) NASA ATR: Cynthia Cheung

3 Introduction Time to move beyond just moving data arrays from machine to machine. –Develop Data Model that is rich enough to standardize: Complex Structures, Axial/Coordinate information, Error bars, named subsets/sections/tiles, arbitrary tuples, physical units. Allow browsers, visualization programs, image processing packages to automatically display complex data properly. –Use OO inheritance so that ES, Astro, Space Physics, Biology etc. all use same basic schema. Share / Reuse software across disciplines Exchange data across disciplines –Use one basic schema from spacecraft to publishing house. Thereby, maintain metadata.

4 Introduction (cont’d) Use XML –Get in sync with world wide programming effort and standardization efforts of the w3c Browsers, XML Tools (parsing, transformation, styling, management, validation, web publishing), and Semantic Web Effort (RDF, DAML). –Get in sync with publishing houses They use SGML, a superset of XML. –Make use of XML Schema Inheritance/Extension. Eg. Basic XDF underlies CDFML and FITSML Eg. FITSML underlies GalaxyDB.

5 XDF - eXtensible Data Format An XML Data Description/Format –Can create 100% XML tagged data documents –Can wrap legacy data formats. Includes fixed- width ASCII, delimited ASCII, or binary data. –Data can be included in document file or separate. XML Logical document can be physically many files Package them together with zip, tar, jar, Web Folder, or SOAP. Designed with Data Object Model. – Hierarchical structure of N-dimensional. – Axis/coordinate information standardized. –Tuples (vectors, complex) handled by adding a dimension. –Tables and images are conceptually merged. –Regions within a cube can be named/addressed.

6 XDF - eXtensible Data Format (cont’d) Error bars, limits and uncertainties are standardized. Notes can be added to any item including a datum in an external binary file. Tokens are defined. Tables are special case of N-dimensional dataCubes. –Slices intercepting a fieldAxis can have local units, notes, and special values. –Traditional 2d tables are special cases of these. A Perl and Java Package implements an API. –Conversion from XDF to legacy formats (FITS, CDF, HDF) made easy by connecting Java IO package.

7 XDF - Arrays within Arrays Array cells can contain references to other arrays. Allows for Hierarchical Mesh for rapid spatial search. At bottom are files each pertaining to small part of space. Also, can append arrays to arrays by using align and append attributes of array.

8

9 Jansky … …

10 (Not Your Father’s) Arrays Axis Element An axis has a name, a unique id (used later to establish read/write ordering), units and values. The values are a list of numbers or names, but there is a shortcut (used here) for generating a series. Later versions will have arbitrary functions. km s

11 (Not Your Father’s) Arrays Read Element Read elements give the general layout of the data. ‘For’ elements tell how to loop through the data by specifying the ‘inner’ (faster) axis and ‘outer’ (slow changing) axis. The dataFormats are pre-specified, so need only to specify spaces and CRs.

12 (Not Your Father’s) Arrays Tuples and Tables One fieldAxis is allowed in an array. DataFormat and units are now specified for each field. Related fields are grouped by fieldGroup. degrees degrees...

13 Parameters A single value is carried by a parameter element. Attributes: special = (infinite | infiniteNegative | noData | notANumber | underflow | overflow) inequlity = ( lessThan | lessThanOrEqual | greaterThan | greaterThanOrEqual | upperErrorValue | lowerErrorValue ) xs:date It happened on or before this date.

14 Using the XDF API // create 2D (10x10)structure import gov.nasa.gsfc.adc.xdf.*; public Structure create2DXDFStructure () { Array XDFarray = new Array(); Structure XDFObject = new Structure(); // tack array into structure XDFObject.addArray(XDFarray); // add axes to the array Axis axis0 = XDFarray.addAxis(new Axis()); Axis axis1 = XDFarray.addAxis(new Axis()); // add Axis values for(int I=0; I < 10; I++) { String axisValueName = new String(I); axis0.addAxisValue(new Value(axisValueName)); axis1.addAxisValue(new Value(axisValueName)); }

15 Using the XDF API // we will add integer data to whole array. // simply set to IntegerDataFOrmat.. XDFarray.setDataFormat(new IntegerDataFormat()); //Now add our integer data Locator myLocation = XDFarray.createLocator(); for(int I=0; I < 100; I++) { XDFarray.setData(myLocation, I); myLocation.next(); } // finished! Return structure w/ array inside return XDFStructure; } i

16 FITSML, CDFML, and IML inherit XDF Upgrades for FITS (astronomy) and CDF (space physics) using XDF as core. Preserve Keywords and vocabularies (just add tags). XDF is rich enough to preserve familiar structures. Wrap, not disassemble, FITS files. FITS-to-FITML Converter (Java FITSIO-to-jXDF Package).

17 FITSML John Smith VLA

18 FITSML After Parse Perl Smith VLA

19 XML Dataset Archive Metadata about published data sets. – abstract, title, description, authors, references, etc. Inherits XDF for tables, spectra, and images. XSLT allows transformation to: – web pages, pdf pages, next schema version, and present query responses. Investigating QXL, XQUERY, PDOM, XML-DB.

20 Article Markup Language Replace LaTeX for manuscript preparation. Separates content from display. A more natural interface with SGML at publishers. Equation editors for MathML. XML tables, images, and graphs (SVG, XDF). –Retain metadata throughout process. Use CSS2 or XSLT to convert back to LaTex or pretty HTML or PDF hardcopy.

21 Summary ADC is doing its best to keep up with tremendous advancements in IT, particularly XML. It is reasonable now to have an end-to-end XML based data handling system from ground-receivers to archive to publication. Complete metadata historical records can be carried along. Real reuse of tools, especially visualization is finally practical through common XML bases. True interoperability between scientific disciplines and the public is within reach.

22 Where to get more information


Download ppt "End-to-End XML Data Description: XDF PI. Ed Shaya Brian Thomas, Zhenping Huang, James Gass, James Blackwell, Gail Schneider (RITSS) NASA ATR: Cynthia."

Similar presentations


Ads by Google