1. A water information R & D alliance between the Bureau of Meteorology and CSIRO’s Water for a Healthy Country Flagship Vocabulary Services, RDF, SKOS and REST Peter Fitch

2. Outline Outline the problem Background on Linked data RDF SKOS REST – Linked Data API Vocabulary Service What is it How to develop a vocabulary service Test case with USGS code list Demo

3. Warning Frequent use of XML

4. Motivations Xlink is all well and good, but the real problem is what is at the end of the link and how to use it. Agreed, and I wish I knew more about the semantic technologies.

5. The need for semantic context From Lemon OSDM Linked Data workshop 2010 Semantic Context Black and White Bessie Good Milker

6. Machines need it to. From Lemon OSDM Linked Data workshop 2010

7. Information Needed Internal Structure-the information model Supported functions – the operations Semantics What are the concepts What are the vocabularies How are they related Where are they defined Where did it come from? How was it created? Current Metadata Adapted from Lemon OSDM Linked Data workshop 2010 Semantic Context

8. The need for semantic information in Hydro-Domain data exchange

9. Don’t Information Models solve the problem?

10. Take a closer look http://www.bom.gov.au/std/water/xml/wio0.2/property//bom/WaterCourseLevel_m

11. The O word You need an ontology! O What? I know one O word and its not that. I better find out more.

12. 4 Rules of Linked Data TBL – key take home! 1.Use URIs as names for things 2.Use HTTP URIs so that people can look up those names. 3.When someone looks up a URI, provide useful information, using the standards (RDF*, SPARQL) 4.Include links to other URIs. so that they can discover more things. Comment: So by following rule 3, we might be able to get some useful information, we still need semantic context though. Tim Berners-Lee http://www.w3.org/DesignIssues/LinkedData.html

13. Linked data quality scheme Sir Tim BL RatingDescription ★ Available on the web (whatever format), but with an open license ★★ Available as machine-readable structured data (e.g. excel instead of image scan of a table) ★★★ as (2) plus non-proprietary format (e.g. CSV instead of excel) ★★★★ as (2) plus non-proprietary format (e.g. CSV instead of excel) All the above plus, Use open standards from W3C (RDF and SPARQL) to identify things, so that people can point at your stuff ★★★★★ All the above, plus: Link your data to other people’s data to provide context Tim Berners-Lee http://www.w3.org/DesignIssues/LinkedData.html

14. Intro to RDF RDF is a data model for describing resources Resource Description Framework SubjectObject Predicate ThingsHave properties Property Value The object of one statement can become the subject in another. The set of linked statements, forms a directed graph Subject, Object and Predicate are all Resources* A set of Subject, Predicate, Object entities is called a Triple

15. RDF Example Remember - Resources are URI’s Peter – http://www.csiro.au/people/PeterFitch.html hasColleague – http:/mydefinitions/defintions#hasColleague Nate - http://cida.usgs.gov/professional-pages/booth.html Peter Nate hasColleague Subject: Peter Predicate: hasColleague Object: Nate

16. RDF Landscape Basic resource descriptions RDF Express resources as classes, with properties and class relationships RDFS OWL Web Ontology Language Exact description and relationships SKOS Simple KOS Simple description and relationships Expressivity Basic Building Blocks SPARQL RDF Query SPARQL RDF Query

17. Intro to SKOS SKOS : Simple Knowledge Organizational System. KOS- provides semantic context. Built on RDF and RDFS Designed to bridge current chaotic, poorly described web, and full sematic web – OWL. See SKOS primer at http://www.w3.org/TR/skos-primer Limited vocabulary eg: skos:ConceptScheme skos:Concept skos:prefLabel skos:scopeNote And some limited standard relationships skos:exactMatch skos:narrower, skos:broader Allows for limited inference Because of its limited vocabulary, really useful for Thesauri, classification lists, taxonomies etc.

18. SPARQL Queries Purpose: query a RDF triple store, works by matching triples to patterns. example: select ?concept where { ?concept rdf:type skos:Concept} Return me all concepts which are of rdf:type skos:Concept Other Queries CONSTRUCT – returns a rdf graph ASK – returns bool if triple is matched DESCRIBE – returns a graph describing a resource.

19. Intro to Linked Data API Familiar with RESTful services right?? LD API designed as a bridge between the complexity of SPARQL endpoints, and a standard REST API Provides standard URI matching patterns and additional specification for behaviors eg: /doc/school/12345 should respond with a document that includes information about /id/school/12345’ /doc/school should respond with a document of schools /doc/school/12345.JSON should return with a JSON document.

20. Intro to linked data API URI pattern SPARQL for Result Set SPARQL for view on Result Set Response in RDF, Turtle, etc.

21. Vocabulary Service In semantic web, vocabulary is defined as a set of URI’s Functionally we want: Ability to look up definitions of terms and or code lists – skos:Concept, skos:definition, skos:prefLabel Ability to resolve synonyms skos:exactMatch, skos:broader, skos:narrower Ability to deal with different langauages skos:preflabel lang=en Standard API – Linked data API and REST Standard Information/Data Model - SKOS

22. Simon Cox vocab proposal Proposal by S. Cox https://www.seegrid.csiro.au/wiki/Siss/VocabularyService3

23. Vocab development process 1.Select code list or vocab for service 2.Map code list to skos 3.Check code list for web compatibility and harmonise with other code lists or vocabs 1.eg: use a standards units vocab 2.remove any non conforming content. 4.Convert code list to SKOS RDF 5.Validate RDF using W3C RDF validator 6.Import to Triple Store 7.Publish Service 8.Use: Link to in documents!

24. Case study USGS Parameter Code List Proof of Concept Code List is a CSV table of parameter codes.

25. Mapping code list to SKOS Parameter CodeSKOS Mapping Parameter Code Listskos:ConceptScheme Parameter Codeskos:Concept Group Nameskos:broader Parameter Nameskos:prefLabel cas Nameskos:exactMatch srs Nameskos:broadMatch Unitsneed additional relationship usgs:hasUnits

26. Content conformance-harmonization Issues need skos:Definition – Parameter Name? Invalid characters for web in Parameter Name eg &, - & < > Units – non standard representation, eg Mi2 (square Miles),mgd (Million G per day),%, nu (number of bad characters TX by DCP) Fix up Leave as a literal?

27. Comments on code list Conflation of information Chromium(VI), water, unfiltered, recoverable, micrograms per liter Observable phenomena – Chromium(VI) Procedure – unfiltered/recoverable media – water Units – ug/L Phosphorus, suspended sediment, total digestion, dry weight, percent Observable phenomena - Phosphorus procedure total digestion (but not linked to standard method) media – suspended sediment units dry weight percent. Some meaningless codes - Precipitation, cumulative at given time, location 6, inches

28. Duplication Turbidity, water, unfiltered, broad band light source (400-680 nm), detection angle 90 +/- 30 degrees to incident light, nephelometric turbidity units (NTU) Turbidity, water, unfiltered, laboratory, Hach 2100AN, nephelometric turbidity units USEPA method 180.1??? Are they the same? Why not link to standard methods? Needs work by domain experts to resolve.

29. Conversion to SKOS-EXcel2SKOS.Net utility to convert Table into skos using Nvelocity Spreadsheet Template Office Interop Formatter Excel2Skos

30. Nvelocity Template-Mapping to SKOS #foreach($row in $excelsheet) Definition of parameter code $row.get($code) $row.get($name) #if($row.get($casrn)!= "") #end #if($row.get($srsname) != "") #end #end Classes passed in Globals – ConceptScheme definitions excelsheet – 2D table of values.

31. Conversion to skos

32. Converting list to RDF

33. RDF Validation http://www.w3.org/RDF/Validator/

35. Import to Triple store

36. Test Services Developed REST services using Microsoft WCF dotnetRDF and NVelocity libraries Test API /Vocab/ParameterCodeList – respond with a document of skos:ConceptScheme /Vocab/ParameterCodeList/ParameterCode – respond with the first page of parameter codes – my implementation returns all! /Vocab/ParameterCodeList/ParameterCode/{ID}

37. Process reminder Sesame RDF Triple store SPARQL API USGS Code List Harmonise and Map Excel2RDF RDF Validator ValidateRDF Load dotnetRDF WCF REST TestServices http://localhost:8080/VocabService/ParameterCodes/

38. Demo

39. Next Steps Try the Auscope tooling the process is the same, uses sesame rdf store has diferent tooling for Excel to RDF Different service interface, LD not quite ready. If have time, we should set up a test service before I leave. Below is example of what LD vocabs in WaterML2.0 might look like.

40. Conclusions The need for sematic context to assist with data integration is pressing. Vocabularies are foundation services and need to be put in place for data mediation. Technologies and approaches are now mature enough to use RDF, SKOS, SPARQL, LD API There is tooling available through AUSCOPE, but it needs assessment. USGS & CIDA has the opportunity to make a range of standard vocabularies available for the hydro community.

41. Pillars and foundations of Interoperability System of Systems Interoperablity Identity and Registration Service Standards Application Schema Network Standards Community Profiles Feature Catalogs Agreed Vocabularies and Ontologies Semantic Brokering

42. Final word I don’t look it, but I’m so happy, I know what is at the end of the xlink! I don’t look it, but I’m so happy, I know what is at the end of the xlink!

43. Thank you Business Unit Name Peter Fitch Program Leader Environmental Information Systems Phone: +61 2 6246 5763 Email: peter.fitch@csiro.au Web: www.csiro.au/clw/eis Contact Us Phone: 1300 363 400 or +61 3 9545 2176 Email: Enquiries@csiro.au Web: www.csiro.au

44. Lessons of Climate Gate Theft of e-mails from UEA Nov 2009 E-mails indicated manipulation of data, and suppression of raw data Investigations found methods dis-organised bunker mentality lack of transparency Researchers promised to improve scientific data management open access to data Improve transparency climatic research unit, University of East Anglia

45. From RDF Primer W3C

46. SWT in other domains Eco-Informatics Bio-Informatics Eco-Informatics Bio-Informatics HWB – WIRADA Symposium August 2011

47. If Insert presentation title

48. Terminology Activity – single process block which can perform a useful task and which can be linked to another process block Workflow – a linked set of process blocks HWB – WIRADA Symposium August 2011