Xyleme, A Dynamic Warehouse for the XML data of the Web Grégory COBENA INRIA & Xyleme SA ( ) Serge Abiteboul, INRIA & Xyleme SA ( )

2 Organization 1. The Web and XML 2. Xyleme 3. Data Acquisition and Maintenance XML Repository, Semantic Data Integration and Query Processing 4. Query Subscription Conclusion

Xyleme, The Web and XML

4 The Web today Terabytes of data A lot of public pages –1 billion in [06/2000] –several millions of servers Private web: not publicly available pages Deep web: data hidden behind forms

5 HTML = Hypertext Language Ref Name Price X23 Camera R2D2 Robot Z25 PC Information System HTML The X23 new camera replaces the X22. It comes equipped with a flash (worth by itself $ ) and provides great quality for only $. Text + presentation Where is the data ? hard

6 XML = Semistructured Data Ref Name Price X23 Camera R2D2 Robot Z25 PC Information System camera … Robot …... XML Data + Structure Semistructured: more flexible easy

7 XML : Tree Types Semantics and structure are in paths –product-table/product/reference –product-table/product/price product designationdescription price reference product-table

Xyleme, A Dynamic Warehouse for the XML Data of the Web Xyleme

9 Xyleme Research Project Xyleme at INRIA ( ) : Explore XML + Web + SGBD to make the Web a Knowledge Database INRIA –Sophie Cluet: Databases (OQL…) –Serge Abiteboul: semi-structured data + web –Guy Ferran: ex O 2 Technology Mannheim University –Guido Moerkotte Université d’Orsay –Marie Christine Rousset CNAM –Dan Vodislav

10 Xyleme Company Started September 2000 (25 employees end of 2001) Market Challenges: –Few XML documents available on the Web (because of weak software support) –Company is focusing on private XML: Press, Editors, Financial Data, Biology… –Technology: Scalability for large amount of data Internet (+focus) / Intranet support Monitoring and Version Management Heterogeneous Data Integration

11 Architecture Cluster of PCs Developed with Linux and C++ Communications –local: Corba –external: HTTP Distribution between autonomous machines Now Web Services

12 Repository and Index Manager Change Control Query Processor Semantic Module User Interface Xyleme Interface Functional Architecture I N T E R N E T Web Interface Acquisition & Crawler Loader

13 Index I N T E R N E T Change Control and Semantic Integration Change Control and Semantic Integration ETHERNETETHERNET Repository RepositorryRepository Loader |Query Architecture Acquisition and Maintenance Acquisition and Maintenance

Xyleme, Data Acquisition and Maintenance, Page Importance

15 Goals Discover XML pages on the web that are of interest for customers –For this crawl the web (HTML+XML) Maintain them up to date Do this under bounded resources: –Memory for known URLs –Bandwidth

16 Life Cycle of a page in Xyleme The URL of D is discovered as a link in another page (or published by a customer) The page scheduler decides to read D –The meta data of D is read type, last_date_update... –The document D is loaded The document D is re(read) regularly

17 Main Issues Loading of pages –we can load up to 5 millions of pages/day on a standard PC –main cost is Internet connection Metadata management (access to disk) Page scheduling –decide which page to read or refresh next

18 Page Importance Definition: Important pages are linked to by important pages Offline algorithm (used by Google) Our Online algorithm (M. Preda, S. Abiteboul, G. Cobena) –does not require to maintain graph information –faster convergence with focused crawling

Xyleme, ( XML Repository, Semantic Data Integration and Query Processing )

20 Querying Language Today: A mix of OQL and XQL We are currently moving to X-Query (which is also a mix of OQL and XQL…) Select boss/Name, boss/Phone From comp in BusinessDomain, boss in comp//Manager Where comp/Product contains “Xyleme”

21 Web Heterogeneity Semantic domains, e.g., cinema Many possible types for data in this domain, many DTDs Semantic Integration –one abstract DTD for the domain –gives the illusion that the system maintains an homogeneous database for this domain 1 domain = 1 abstract DTD

22 Indexing Standard inverted index –word  documents that contain this word Xyleme index –word  elements that contain this word document + element identifier Goal: more work can be performed without accessing data

Xyleme, Change Control

24 The Web changes all the time Data acquisition + maintenance –keep the warehouse up-to-date Version management –representation and storage of changes Change monitoring –query subscription

25 Subscription Language SQL-like language based on ‘atomic events’. Combines the use of monitoring queries and continuous queries. The language can be extended by adding new types of atomic events. Uses the XML Query Language for continuous queries. “Querying the XML Documents of the Web”, V. Aguilera, S. Cluet, F. Boiscuvier, Tech. Report

26 subscription myPaintings % what are the new painting entries in Musee d’Orsay site monitoring newPainting select URL where URL extends and contains “Monet” % manage the changes in the expositions continuous delta Exposition select... from... where when monthly notify daily% send me a daily report Example Atomic events

27 Step 1: Atomic Event Detection metadata manager HTML parser XML loader document & alerts d/46 complex event detection atomic event 46: URL matches pattern atomic event 67: XML document contains the tag with the value “Monet” 5 millions of pages/day d d/46,67 loading

28 Step 2: Complex Event Detection HTML parser XML loader complex event detection complex event 12: 67 & 46 (XML document contains the tag with value “Monet” and URL matches pattern Millions of alerts of pages/day Millions of subscriptions

29 triggers notification/monitoring Step 3: Notification Processor Reporter continuous queries complex event detection clock notification/results Millions of notifications/day alerts

30 SQL Architecture Xyleme Alerter Web Browser Xyleme Reporter Xyleme Subscription Manager Complex Event Detection Subscription Manager Reporter Trigger Engine Xyleme Query Processor SQL documents

31 Complex Events Algorithm The formal problem is NP-hard We proposed several possible algorithms Experimental (simulation) values proved the effectiveness of our solutions The Hash-Tree based algorithm is well suited for our application: – 10 million Complex Events – 1 million Atomic Events – 100 Atomic events detected per document 0.8 ms to process a document. ~2 million documents per day.

32 Alerters Each Alerter can be viewed as a plug-in that acts on a document flow. All sorts of Atomic events can be detected: URL pattern detection, Keywords, XPath expressions, Page rank… Can be distributed.

33 Some Advanced Alerts Process document flow (single pass) Full strings –Context Stack –Reversed look-up XML Alerts –Reversed XPath expressions –Dual context stack for ‘/’ and ‘//’

34 Versions Objectives: –Temporal Queries (persistent identification of nodes) –Version some documents or some sites (store a ‘delta’) –Change Monitoring (query changes) We proposed a representation of changes “Change-Centric Management of Versions” (VLDB 2001) We developed a Diff algorithm for XML “Detecting Changes in XML Documents”, G. Cobena, S. Abiteboul, A. Marian ICDE 2002 (San Jose)

35 Conclusion & Prospectives Focus crawling on important pages –Refine notion of importance –Improve important pages discovery Improve Change control accuracy –Semantic web –Real-time advanced processing

Xyleme, Merci