Inference of Concise DTDs from XML data Geert Jan Bex 1 Frank Neven 1 Thomas Schwentick 2 Karl Tuyls 3 1 Hasselt University and Transnational University of Limburg 2 Dortmund University 3 Maastricht University and Transnational University of Limburg

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

Aims & requirements Problem: infer DTD from XML corpus Requirements: –Concise: humans can interpret/validate –Work on large data sets –Work on small data sets –Robust to noise DTD XML

Why DTD inference? Schema inference –≈ 50 % of XML documents : no schema [Barbosa et al. 2005] –≈ 66 % of DTDs and XSDs : not valid [Bex et al. 2005] –Improving existing schemas –“Noisy” XML documents ≈ 90 % of XHTML docs : not valid Related work –Fails on real-world, large data sets –Results not concise

Why schemas? Validation : efficiency, security Optimization : search, processing Static analysis, type checking (e.g., XQuery) Software development : modeling, OR-mapping Integration : (meta-)data sources Schema matching Semantics

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

XML documents book title author author year … ………… book title editor year isbn … …… Learning regular expression from set of strings title (author + + editor + ) year isbn?

Learning automata? Well studied, but… Learning automata ≠ learning regular expressions ((b?(a+c)) + d) + e

abbb + abbd + acd + ac –most specific regex for S (a + b + c + d)* –most general regex for S Learning regular languages? S = { abbb, abbd, acd, ac } ??? < < a (b* + c) d? ? generalization vs. specificity positive examples only! Impossible… in general

Subclasses S ingle O ccurrence R egular E xpressions –99 % of regular expression in DTDs/XSDs CHA in R egular E xpressions –90 % of regular expression in DTDs/XSDs Infer with iDTD Infer with CRX 

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

SOREs What’s a SORE header. protein. organism. reference*. comment*. genetics*. complex*. function*. classification?. keywords?. feature*. summary. sequence authors. citation. volume?. month?. year. pages?. (title + descr)?. xrefs? title. (author. affiliation?) +. abstract … and what’s not title. ((author. affiliation) + + (editor. affiliation) + ). abstract duplicate element names

Sample  SOA W = {bacacdacde, cbacdbacde, abccaadcde} b a c e d S ingle O ccurrence A utomaton 2T-Inf [Garcia & Vidal 1990]

Sample  SOA SOA size –|  | + 2 states – O (|  | 2 ) transitions Complexity of algorithm – O (||W||) –streaming Algorithm sound –W  L(SOA) in general: |S| |L(SOA)| <<

SOA  SORE: R EWRITE b a e d c optional b a e d c b? disjunction a, c e d b? a+c concatenation b?, a+c e d b? (a+c) e d ((b? (a+c)) + self-loop b? (a+c) ((b? (a+c)) + d) + e

R EWRITE : properties Theorem –R EWRITE transforms SOA into equivalent SORE for sufficient data, reports failure otherwise (sound & complete) –Complexity: O (|  | 4 ) SORE size –|  | symbols – O (|  |) operators

R EWRITE + repairs = iDTD W = {bacacdacde, cbacdbacde} b a ce d no rules apply !!! almost disjunction a, c b a e d c ((b? (a+c)) + d) + e Fix: enable-disjunction enable-optional

iDTD: properties Theorem –iDTD transforms SOA into SORE such that L(SOA)  L(SORE) iDTD can be parameterized for performance

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

CHAREs Definition: A chain regular expression is a sequence of factors f 1,…,f n such that no alphabet symbol occurs more than once and a factor is one of (a 1 + … + a k ) (a 1 + … + a k )? (a 1 + … + a k ) + (a 1 + … + a k )* CRX derives CHAin Regular Expressions C hain R egular expression e X traction

CHAREs What’s a chain header. protein. organism. reference*. comment*. genetics*. complex*. function*. classification?. keywords?. feature*. summary. sequence authors. citation. volume?. month?. year. pages?. (title + descr)?. xrefs? … and what’s not title. (author. affiliation?) +. abstract title. ((author. affiliation) + + (editor. affiliation) + ). abstract not a factor duplicate element names

CRX run: pre-order relation a b c c d e c c c a d b f e g b f h i Sample W Pre-order relation  W a b b c c d d e c a a d b f f e e g f h h i a b cf e dg hi

a  W b and b  W c then a  W c CRX run: transitive closure a b c c d e c c c a d b f e g b f h i Sample W f e dg hi a b c

CRX run: transitive closure a b c c d e c c c a d b f e g b f h i Sample W f e dg hi a b c a,b,c equivalence class a  W b and b  W a then a  W b Symbol occurs in exactly one equivalence class

CRX run: folding a b c c d e c c c a d b f e g b f h i Sample W f e dg hi a,b,c predecessor setsuccessor set partial order  W pred(  ) = {  ’ |  ’  W  } succ(  ) = {  ’ |   W  ’}

CRX run: folding a b c c d e c c c a d b f e g b f h i Sample W eghi a,b,c d,f partial order  W pred(  ) = {  ’ |  ’  W  } succ(  ) = {  ’ |   W  ’}  W : partial order  W

CRX run: multiplicity & RE a b c c d e c c c a d b f e g b f h i Sample W e g hi a,b,c d,f + ? ? ?? e?.. h?i?. g?.. (d + f)(a + b + c) + Chain Regular Expression topological sort

CRX algorithm: properties Optimality:  W linearly ordered   CHARE r, W  L(r) and L(r)  L(r W ): r W = r Performance : O (||W|| + |Σ| 3 ) Training set size: Any CHARE r can be learned from {w | w  L(r)   w’  L(r): |w|  |w’| + 2}

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

Related work XTRACT [Garofalakis et al. 2000] –Pioneer –More general than iDTD –Focuses on regular expressions that don’t occur in real DTDs  no concise schemas Trang: roughly equivalent to CRX –Inconsistent results

Data Real world regular expressions –SOREs –Non SOREs Real world data when available Synthetic data otherwise

real world data

real world regexes

Experiments: generalization CRX iDTD no repairs

Experiments: generalization CRX iDTD

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

Extensions Incremental computation –new data  update internal representation (SOA or partial order) Noise –Support for element name too small  ignore element –SOA: support for edges too small  delete edges before repair Numerical predicates –Bookkeeping: minOccurs, maxOccurs Generating XSDs –Infer data types (integer, double, date,…)

Outline Goals & motivation Problem setting iDTD: Sample  SOA  SORE CRX: Sample  CHARE Experiments Extensions Conclusions

iDTD + CRX –learns robust class of regexes from positive examples –complete in their target class for sufficient data –deals with insufficient data –performs well on real world data –runs efficiently Future work: inferring XML Schemas