Presentation on theme: "NeSS Data Exchange 2.0. Timeline for NeSS Data Exchange Started as a pilot called Cafeteria in 2006 Rejuvenated by joint initiative with CLG 2007 Beta."— Presentation transcript:
Timeline for NeSS Data Exchange Started as a pilot called Cafeteria in 2006 Rejuvenated by joint initiative with CLG 2007 Beta April 2008 Release 1.0 December 2008 Release 1.1 April 2009 Release 1.2 November 2009 Release 2.0 February 2010 Eventually to be replaced by WDP API
Cafeteria Pilot system built in 2006 Web service layer over NeSS database Generic NeSS-ML output XML No “discovery” Used by iGather (Scisys) Left on hold until decision to commence NeSS Interoperability project which was part- funded by CLG
4 CLG’s Future Vision: The “Data Web” Places dB DCSF DCLG ONSAudit Commission LA LSP LA LSP LA LSP Internet Hubs LA LSP Central Services Reg Obs Methods System to System GSi OGDs Inspectorates Author : M Stone v 1.6 30/01/08
NDE’s Role To supply CLG’s Data Interchange Hub To support other Local Information Systems National usage secondary consideration Web 2.0 apps also secondary consideration Service can be used live - “hunter” Or to accumulate data - “gatherer”
Web Site vs. Web Service BrowserWeb SiteData Access Component DB Browser Spread Sheet Other AJAX Excel Macro Java.Net etc. NDE Web Service Data Access Component DB Fig. 1 Web Site Fig. 2 Web Service
XML Formats Looked at SDMX – too complex, problems with geography, output hard to consume Also considered keeping NeSS-ML but it was considered too abstract and generic to be easily consumed Went with LGDX (was PIE) to be consistent with CLG hub – also simple and rigid so easy to consume. Only downer is it cannot support cross-tabs. Retained NeSS-ML for internal use and potential transform to other formats in future.
Discovery Half of NDE is the Discovery Service This allows you to ask questions of the NeSS database. Often this will be a string of queries…. Get list of subjects User selects “Health” Get datasets for subject “Health” User selects “General Health UV20” Get variables for dataset UV20 User clicks i button next to “Not Good Health” Get metadata for this variable
Delivery The other half is the Delivery Service This delivers the data payload as a self- documenting chunk of XML. Initially this required a complex request to be built up from discovery calls. Simplified in Release 1.1 Initially only LDGX format SDMX and RDF added in Release 1.2
Clients ONS supplied sample clients to show how the service can be used with different languages Some interest from commercial companies to supply “middleware” Access initially via SOAP only (REST-like version to follow) Authenticated access via manual registration only. Valuable at first but not seen as so important now.
Pros and Cons of NDE POWERFUL – everything the NeSS website can do you can do with NDE, including useful geographical functions COMPLEX – big manual, many operations, understanding of NeSS database required RELIABLE – 99.6% uptime INCOMPATIBLE – with some newer systems, notably Adobe Flex and Ruby
Where are We Now? NDE has had generated only moderate interest so far, and there are currently no live applications using it as far as we know (though SASPAC beta looks promising). Versions 1.2 and 2.0 look to redress some of the “Cons” and provide alternative output formats (corporate requirement, not from NeSS) Small budget for improvements but committed to continued support for foreseeable future. NDE is proving to be a useful toe in the water for the WDP API that will eventually replace it.
What’s New? VERSION 1.2 Adobe Flex support (+demo) SDMX output RDF output (experimental) VERSION 2.0 Lite Service
Adobe Flex Premier Data Visualisation Tool NDE 1.1 does not have the correct configuration to allow Flex applications to call it. Changes made to 1.2 to make it work. Demonstration “dashboard” application supplied. Uses SOAP service, will supply a REST-like version with 2.0
SDMX International Data Interchange Format Being adopted by ONS, NOMIS and many overseas organisations such as Eurostat Can comprehensively model all data Flexible and Complex Generated on-the-fly by NDE Message contains definitions and data Harder to consume than LGDX Sample Output sdmx_sample2.xmlsdmx_sample2.xml
RDF Building block for the Semantic Web Triples consisting of subject, predicate and object Format favoured for Tim Berners-Lee’s UK Public Data Project Generated on-the-fly by NDE Experimental format based on SCOVO Could supply complete geographic hierarchies like OS Parishes
RDF Example Alan Pardew (subject) is manager of (predicate) Southampton FC (object). Markus Liebherr (subject) is chairman of (predicate) Southampton FC (object). Using this method everything on the www is interlinked Good for metadata Possibly too verbose for large datasets – NeSS has about 10,000,000,000 triples. Sample file ness_transformed_to_rdf.xmlness_transformed_to_rdf.xml
Lite Service Simplified, fewer operations REST or SOAP Low security Optional registration Aimed at web 2.0 and other lightweight users Possibility for reduced functionality
Simplified Operations Hide some of the complexities of the NeSS database Cut down the perceived number of operations by adding a few more! Split into “Core” and “Advanced” operations Rationalise the area search into a single operation Make the user documentation less daunting
SOAP Web Service The Simple Object Access Protocol Standard for web services for many years Usually operates via an HTTP POST Operations defined via a WSDL document Many languages such as Java and VB can use the WSDL to create objects SOAP can also be used for AJAX applications via XMLHTTP with raw XML but it is clunky
REST Web Service (HTTP GET) Not RESTful by the textbook definition, but calls are made via a simple URL, e.g. http://www.neighbourhood.statistics.gov.uk/int erop/NeSSLiteDiscoBindingPort/GetAreaChil dren?Areaid=565457http://www.neighbourhood.statistics.gov.uk/int erop/NeSSLiteDiscoBindingPort/GetAreaChil dren?Areaid=565457 Can only accept simple parameters (strings and numbers) No auto-generation of objects from WSDL Preferred to SOAP for many modern web platforms
Security: Lite Service URI Lite Discovery Endpoint Lite Delivery Endpoint OPERATION BOMXRender Request HTTPS
NDE Lite - Other Despite no need to register, users can still sign up for notifications and tech support There will be no loss of functionality except for the complex getDataCube function Current plan is to run NDE Lite as a separate application in parallel with the current service. Turn off old service once users have had plenty of time to migrate