the Need for Data Integration Data, Data everywhere: the Need for Data Integration Nicolas Spyratos Professor Emeritus University of Paris South France «Data, data everywhere» : The Economist, February 25, 2010

the relevant questions What is data integration? collecting and combining information from multiple sources into a single information source Why is it needed? to get more informative answers to important questions and/or analyse it to make decisions How is it done? following a well disciplined approach not necessarily using computers What are the technical problems when using computers to do it? many, difficult and costly What are the prerequisites for data integration? datasets should be open and preferably linked

Example: writing a summary report on rice-production, transportation and commercialization (using available information from Japan and Vietnam) datasets translation integration decision making japanese Japan thai thai minister of agriculture thai vietnamese Vietnam one important difficulty though: the data sources are autonomous, heterogeneous and geographically dispersed

sharing the integrated information minister of agriculture Japan minister of transport minister of commerce Vietnam this is the concept of data integration independently of whether we use computers or not

let’s summarize (before going to computer-assisted integration) Japan Vietnam translators need to know the language of the dataset and the language of the integrator integrator specialists decision makers datasets we can now replace all intermediate activities with software modules and either store the knowledge of the integrator in a database (called a data warehouse) or “simulate” it by a software (called a mediator)

using computers for data integration – the data warehouse approach (data in advance) production of “goods” processing/transport wholesaler distribution (to retailers) consumption metatada dataset-1 Translator-1 . Integrator database Data Mart Translator-n Data Mart dataset-n datasets databases, file systems twit sets, XML docs, etc. translators extract/transform integrator filters/loads data warhouse stores integrated data and answers queries specialists filter/answer decision makers and analysts Real world example: the Walmart data warehouse contains 2,5 petabytes of data

a few remarks about data warehouses a data warehouse is above all a database but of a specific nature as : its users are mainly analysts and decision makers (i.e. non computer specialists) it is accessed in read-only mode (usually through data marts) updates happen only at the source datasets and propagated to the data warehouse periodically they store mostly historical data (usually records), therefore the data volumes are orders of magnitude higher than in traditional databases (the Wallmart data warehouse stores 2.5 petabytes of data, i.e. 167 times the information contained in all the books in the US Library of Congress)

translators extract/transform using computers for data integration – the mediator approach (data on demand) dataset-1 Translator-1 .. . . software module Translator-n dataset-n datasets databases, file systems twit sets, XML docs, etc. translators extract/transform mediator query decomposition synthesis of answers decision makers and analysts Real world example: mediating a car dealers network

a few remarks on mediators a mediator is not a database but a software modulethat allows querying multiple sources its users are mainly analysts and decision makers (i.e. non computer specialists) it answers queries of its users users can not update through the mediator (as is the case with data warehouses) they do not store data, they just answer queries the translators are complex pieces of software and writing generic translators is hard

prerequisites for data integration (whether in data warehousing or mediating) a minimal requirement for data integration is that the datasets should be collections of data, published or curated by a single agent, and available for access or download in one or more formats Example of such a dataset: The Credit Institutions Register It is published by the European Banking Authority (EBA) and contains a list of credit institutions to which authorization has been granted to operate within the European Union and European Economic Area countries (EEA).  if the datasets to be integrated are also linked and open then integration can release social and commercial value (ex: through data mining in integrated datasets)

linked data linked data is about publishing and connecting structured data on the Web, using standard Web technologies (such as HTTP, RDF and URIs) to make the connections readable by computers, enabling data from different sources to be connected and queried allowing for better interpretation and analysis an open dataset is a collection of data that can be freely used, modified, and shared by anyone for any purpose most datasets of the web are linked (ex: DBPedia)

open data a dataset is called open if it can be freely used, modified, and shared by anyone for any purpose most datasets of the web are not open (or if they are then their quality is low) the following site contains a list of open datasets most of which have been closed! https://blog.bigml.com/list-of-public-data-sources-fit-for-machine-learning/ however, within controlled user communities, openness is extremely useful (ex: collaborative working environments, big companies government agencies)

concluding remarks data integration is a basic tool in a large number of social and commercial activities (e.g. hotel or airplane bookings, e-learning, digital libraries, e-Government etc.) data warehouses and mediators constitute the common supporting technology for data integration data integration is especially important to governments, where large amounts of data reside in isolated information silos linking, integrating and opening government data can help drive the creation of innovative business and services that deliver social and commercial value

thank you for your attention