State of the Nation in Data Integration for Bioinformatics Carole Goble and Robert Stevens Presented by: Daya Wimalasuriya

2 Outline Introduction Why integration of bioinformatics sources is especially hard Use of traditional data integration techniques in bioinformatics Use of new integration techniques Where to go from here?

3 Introduction As a discipline, bioinformatics is based on a range of diverse, complex and distributed data resources (900 or more). Because of the existence of such a large number of sources, data integration is critically important in this field. Integration of these sources is a challenging task. It is said that bioinformaticians should be a little ashamed by this situation.  It has been stated that a “Bioinformatics Nation” should be developed from the current set of competing “Princely States”.

4 Introduction (contd.) Why so many sources?  The Web makes it (too) easy to publish data.  Being a resource provider is one way to make a reputation.  Each new sub-discipline develops its own data representations skewed to its biases.  Each type of data has many resources which have many overlaps. In comparison, areas such as particle physics have few centralized data resources.

5 Introduction (contd.) Additional problems of these sources  Each group is highly autonomous and routinely create different data resources and designs.  Different interfaces are provided by different groups (e.g., “flat files”, XML, APIs)  Users are often independent and decoupled from data providers.  Many groups don’t have the expertise or the resources needed to survive (only about 18% have a sustained future)

6 Special Challenges of DI in BI When compared with fields such as astronomy and particle physics, bioinformatics data are not very large. The problem is the complexity of data, arising from several factors  describing a sample and its originating context  diversity of sources of a sample  large number of inter-links, etc.

7 How to Handle the Complexity of Data? Common, shared identities and names: “A biologist would rather share their toothbrush than their gene name” Shared semantics  Ontologies can help but political and theoretical wrangling hinder their development Shared and stable access mechanisms  In 2007, BioMART altered its interfaces four time breaking any client software that used them

8 How to Handle the Complexity of Data? (contd.) Adhering to standards a blue collar science Explicitly stating collection policies and governance Balancing “curation” with ease of use These issues have to be handled while keeping the freedom of rapid innovation.

9 Different Data Integration Techniques used with Bioinformatics Sources

10 Traditional Data Integration Techniques Link Integration (Search):  Directly cross-references a data entry in a data source with another entry in another data source.  Implemented using hyperlinks.  Widely used by bioinformatics systems such as SRS, Entrez and Integr8.  This technique actually represents interlinks created in a haphazard manner.  Vulnerable to name changes, updates, etc.

11 Traditional Data Integration Techniques (contd.) Data Warehousing (Materialization):  Data are extracted, cleaned and stored in a separate, integrated database.  Some people believe that this is the only data integration technique that actually works.  Requires a pre-determined encompassing model.  Involves a high initial cost as well as high maintenance costs; hard to change; commonly decoupled from data providers;  Often result in “data mortuaries”.

12 Traditional Data Integration Techniques (contd.) View Integration (Mediation):  Data is still in the source databases but a virtual warehouse is constructed using mappings.  Uses models such as Global-As-View (GAV) and Local-As-View (LAV).  Popular among database theorists and vendors.  Developing a global model is costly, mappings are often brittle, results in a complex environment.  Automated processes are necessary to make this method practically useful.

13 Traditional Data Integration Techniques (contd.) Integration Application (Ad hoc methods)  Applications specifically designed for a particular integration task.  Generally uses a combination of integration techniques and provides more options for the user.  Avoids the “Big I Challenge” Workflows coordinate a transient workflow between data services and analytical tools and expose the integration methods.

14 New Data Integration Techniques Service Oriented Architectures:  Include technologies such as CORBA and Web Services.  Data Integration has to be achieved by “plumbing” these services.  CORBA is generally considered too heavy despite its technical sophistication.  SOAP based web services has shown promise but have problems such as poor documentation.  These services are necessary to do away with the widespread practice of “screen scrapping”

15 New Data Integration Techniques (contd.) Mashups:  A Web 2.0 idea based on taking data from more than one web-based resource to build a new web- based application (e.g., combing a feed of earthquake measurements with Google maps)  Delivered through the Web, open and light.  Platforms such as Microsoft Popfly and Yahoo! Pipes are already available.  Has been used in applications such as tracking the spread of aviation flu.

16 Mashups (contd.) Emphasizes the role of the user in creating a specific, light-touch, on-demand integration. Relies on the existence of APIs and light- weight tools for development. Preferred by bioinformaticians over heavier general engineering solutions. Just as vulnerable as other integration techniques to identity clashes and concept ambiguities.

17 New Data Integration Techniques (contd.) Semantic Web applications are also expected to help the integration of bioinformatics sources. They are called “smashups”. Smashups should use ontologies and support reasoning. The communication mechanisms may be the same as those used by mashups. (e.g., AJAX in the client side)

18 Architecture of Smashups

19 Requirements of Smashups Simple and stable APIs that can be used by third parties Publishing data as RDF and supporting SPARQL endpoints (or supporting conversion to these formats from databases) Clarifying the semantics Tackling the problem of object reconciliation.

20 Where to go from here? Better naming standards and handling object reconciliation is essential for the success of any data integration technique in bioinformatics. Most promising techniques:  Web Services  Mashups (Web 2.0 Applications)  Smashups (Semantic Web Applications)

21 Thank You! Questions?