1. Data Format Description Language (DFDL) WG Martin Westhead EPCC, University of Edinburgh M.Westhead@epcc.ed.ac.uk

2. Overview Aims Approach Documents Discussion

3. Aims A language for describing formats –Transparency of physical representation –Unambiguous (persistent) description of data –Standard implicit XML “view” of data –Generic tools (browsing, conversion) Existing work –BinX http://www.edikt.org/binx/http://www.edikt.org/binx/ –BFD part of the Scientific Annotation Middleware project (http://www.scidac.org/SAM/)http://www.scidac.org/SAM/ –ESML http://esml.itsc.uah.edu/http://esml.itsc.uah.edu/

4. Basic Mechanism XML description of structure Description can be annotated –byteOrder=“littleEndian” In DFDL description language extensible 10010010 10111011 10001001 10111011 01011011 11011011 10010011 10010010 10010010 10111011 10001001 10111011 01011011 11011011 10010011 10010010 10010010 10111011 DFDL description Data file

5. Approach Separate out structure and semantics Tried to avoid a data model General structural language –Repetition –Pointers –References to data –New structures can be built (compositionality) Semantics –Hard to express so…we don’t –General labeling –Label semantics define elsewhere (ontologies) –Labels can be added (extensibility)

6. Structure – arbitrary labels fooSet fooPair foo bunchThings thing0 1 1 0 0 1 1 1 bunchThings............ foo...... fooPair......

7. Structure – example labels complex Array complex float byte bit0 1 1 0 0 1 1 1 byte............ float...... complex......

8. Structural language Formal semantics –Structured binary sequence –Defines hierarchical structure over underlying sequence of binary values Language for describing hierarchical structure –Repetition Explicit number repeats Termination characters –Data reference Conditionals Data size –Pointers Scope –As general as possible but –Must be concise and implementable

9. Ontologies Define mappings binary structure language primitive SDL define binary structure Core API: –getAsInt, getAsFloat, getAsByte… –getAsIntArray, getAsFloatArray… For each structure, ontology defines semantics for each method

10. Ontology 2 short … byte byte byte … 01101001 00100101 00110100 … Ontology defines: –New structures –New XML associated with them –Mappings from new structures to language primitives via core API –API extensions –Relationships between structures (RDF/OWL)? Java primitives

11. Current documents SDL formal definition XML representation of SDL Primitives ontology Basic structures ontology Primer All strawmen known to be incomplete/flawed aim to kick start discussion

12. Discussion issues Transformations –Low level: encodings, compression, blocking –High level: filtering out formatting Concept of type needed –distinct from binary representation Programming language independence Pointer semantics Expressive power of SDL vs. implementability –Layered standard (?)

13. Current proposals More transformation oriented view –as opposed to representation oriented Introduction of data model 01001001… dfdl:char Java charJava byteC char bindings representations