Presentation is loading. Please wait.

Presentation is loading. Please wait.

Nikos Manolis Agro-Know Technologies Tutorial on data aggregation and accessing datasets.

Published byAmy Sharp Modified over 8 years ago

Similar presentations

Presentation on theme: "Nikos Manolis Agro-Know Technologies Tutorial on data aggregation and accessing datasets."— Presentation transcript:

1 Nikos Manolis Agro-Know Technologies Tutorial on data aggregation and accessing datasets

2 Slide 2 of 63 There is a lot of data

3 Slide 3 of 63 Need for data aggregation and harmonization

4 Slide 4 of 63 Objectives This presentation aims to provide information on:  How to use a service for aggregating datasets  How to get already processed datasets  How to search processed datasets with a search API Educational – GLN API (21008 res) Bibliographic – ABN API (451602 res)

5 Slide 5 of 63 The agDataHarvester service Implements the OAI-PMH protocol to harvest metadata records from open data providers – REST-based API – Harvested dataset available through HTTP

6 Slide 6 of 63 agDataHarvester parameters { "document_type": "harvesting_target", "harvesting_target": { " name ":"Repository name", " description ":”Short Repository Description", " url ":"OAI-PMH target URL", " type ":"metadata format prefix", " frequency ":hours }

7 Slide 7 of 63 param.json { "document_type": "harvesting_target", "harvesting_target": { "name":"Indian Academy of Science", "description":"Indian Academy of Science", "url":"http://repository.ias.ac.in/cgi/oai2","http://repository.ias.ac.in/cgi/oai2 "type":"mets", "frequency":24 } curl -X POST -d@INDUS.json http://'demo001':aginfra@agro.ipb. ac.rs/agcouchdb curl -X POST -d@ param.json tp://'demo001':aginfra@agro.ipb.ac.rs/agcouchdb { "ok": true, "id": " 5c56a3fa18fa21d2a85fd63cc9eb78ac ", "rev": "1- 19ef1210376df8f1695a32b53ecb963a" }

8 Slide 8 of 63 Get details on the dataset http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list?dataset.process_parameter_id= 5c56a3fa18fa21d2a85fd63cc9eb78ac

9 Slide 9 of 63 Get details on the dataset http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_view/list_by_process?key=agdataharvester {" id ": " 6796259b52d79e4797e210c06e6a0aee ", "key": "6796259b52d79e4797e210c06e6a0aee", "value": { "_id": "6796259b52d79e4797e210c06e6a0aee", "_rev": "1-d55d7bc90d26db64dae328c9328e4e4a", "document_type": "harvesting_target", "harvesting_target": { "name": “ WorldBank ", "description": "The World Bank - Open Knowledge Repository", "url": ""https://openknowledge.worldbank.org/oai/request", "type": “mets", "frequency": 24 }, "document_publisher": { "address": "83.212.96.169", "author": "demo001", "utc_datetime": "Wed Dec 11 11:58:45 2013", "utc_timestamp": 1386763125 }

10 Slide 10 of 63 The agWorkflow service http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list? dataset.process=agworkflow&dataset.type=oai_lom&dataset.accuracy=true I want all datasets with educational resources processed by the agINFRA powered aggregation workflow ! http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list? dataset.process=agworkflow&dataset.type=oai_agris&dataset.accuracy=true I want all datasets with bibliographic resources processed by the agINFRA powered aggregation workflow !

11 Slide 11 of 63 Is there a way to search on available datasets ?

12 Slide 12 of 63 Search API REST-based queries over harmonized information (result of metadata processing) Two data models supported – akif: describing educational resources for agriculture, http://54.228.180.124:8080/search-api/v1/akif/?q=* – agrif: describing bibliographic resources for agriculture (mainly from FAO’s data), http://212.189.145.241:8080/search-api/v1/agrif/?q=*

13 Slide 13 of 63 Search options Simple search http://BASE_URL/search-api/v1/akif/?q=tomato Searching within specific fields http://BASE_URL/search- api/v1/akif/?languageBlocks.en.description=tomato Temporal http://BASE_URL/search-api/v1/akif/?creationDate=2013-04-16 Fetching specific items http://BASE_URL/search-api/v1/akif/COLLECTION/20296

14 Slide 14 of 63 Managing results Sorting results e.g ?q=*&sort_by=creationDate&sort_order=desc Facets e.g ?facets=set&facet_size=3 Pagination e.g ?q=sea&page_size=25&page=3 Full Documentation : 54.228.180.124:8080/search-api/

15 Nikos Manolis Agro-Know Technologies manolisn@agroknow.gr

16 Slide 16 of 63 … … demo001…demo005 // aginfra

17 Slide 17 of 63 View all harvested datasets http://agro.ipb.ac.rs/agcouchdb/_design/harvesting_targets/_view/list

Download ppt "Nikos Manolis Agro-Know Technologies Tutorial on data aggregation and accessing datasets."

Similar presentations

Large-scale (meta)Data Aggregators & Infrastructure Requirements the case of agriculture Nikos Manouselis Agro-Know Technologies & ARIADNE Foundation

Large-scale (meta)Data Aggregators & Infrastructure Requirements the case of agriculture Nikos Manouselis Agro-Know Technologies & ARIADNE Foundation
Theo van Veen, Koninklijke Bibliotheek The European Library: opportunities for new services.

Theo van Veen, Koninklijke Bibliotheek The European Library: opportunities for new services.
Digital Repositories – Linked Open Data – the possible Role of D4Science Workshop, 16-17 December 2010, FAO use cases A tool to create Linked Data providers.

Digital Repositories – Linked Open Data – the possible Role of D4Science Workshop, December 2010, FAO use cases A tool to create Linked Data providers.
Applying preservation metadata to repositories For JISC KeepIt course on Digital Preservation Tools for Repository Managers Module 3, Primer on preservation.

Applying preservation metadata to repositories For JISC KeepIt course on Digital Preservation Tools for Repository Managers Module 3, Primer on preservation.
Supported by EU projects 12/12/2013 Athens, Greece Open Data in Agriculture Hands-on with data infrastructures that can power your agricultural data products.

Supported by EU projects 12/12/2013 Athens, Greece Open Data in Agriculture Hands-on with data infrastructures that can power your agricultural data products.
Breakout Session 5: Collaboration Between DPs and SPs Protocol simple, providing a service more difficult… …because of lack of networking among DPs and.

Breakout Session 5: Collaboration Between DPs and SPs Protocol simple, providing a service more difficult… …because of lack of networking among DPs and.
June 22-23, 2005 Technology Infusion Team Committee1 High Performance Parallel Lucene search (for an OAI federation) K. Maly, and M. Zubair Department.

June 22-23, 2005 Technology Infusion Team Committee1 High Performance Parallel Lucene search (for an OAI federation) K. Maly, and M. Zubair Department.
Overview of key concepts and features

Overview of key concepts and features
Discove r Humanities and Social Science Electronic Thesaurus - HASSET Faceted search HASSET is the subject thesaurus that the UK Data Service uses to index.

Discove r Humanities and Social Science Electronic Thesaurus - HASSET Faceted search HASSET is the subject thesaurus that the UK Data Service uses to index.
National Science Digital Library (NSDL) Core Infrastructure Metadata Repository (“union catalog”) Naomi Dushay Cornell University.

National Science Digital Library (NSDL) Core Infrastructure Metadata Repository (“union catalog”) Naomi Dushay Cornell University.
Application Standards for ‘Push’ Content and Streaming Media Hadi Partovi Microsoft Corporation.

Application Standards for ‘Push’ Content and Streaming Media Hadi Partovi Microsoft Corporation.
GL12 Conf. Dec. 6-7, 2010NTL, Prague, Czech Republic Extending the “Facets” concept by applying NLP tools to catalog records of scientific literature *E.

GL12 Conf. Dec. 6-7, 2010NTL, Prague, Czech Republic Extending the “Facets” concept by applying NLP tools to catalog records of scientific literature *E.
Using the DPLA API as Community Reps Webinar August 19, 2014 A PLATFORM TO BUILD UPON Danielle Cunniff

Using the DPLA API as Community Reps Webinar August 19, 2014 A PLATFORM TO BUILD UPON Danielle Cunniff
ETD Repositories Using DSpace Software Andrew Penman The Robert Gordon University 27 th September 2004.

ETD Repositories Using DSpace Software Andrew Penman The Robert Gordon University 27 th September 2004.
PerfSONAR Client Construction February 11 th 2010, APAN 29 – perfSONAR Workshop Jeff Boote, Assistant Director R&D.

PerfSONAR Client Construction February 11 th 2010, APAN 29 – perfSONAR Workshop Jeff Boote, Assistant Director R&D.
4th project meeting 27-29/05/2013, Budapest, Hungary FP 7-INFRASTRUCTURES-2011-2 programme 283770 agINFRA agINFRA A data infrastructure for agriculture.

4th project meeting 27-29/05/2013, Budapest, Hungary FP 7-INFRASTRUCTURES programme agINFRA agINFRA A data infrastructure for agriculture.
AGRIS Multi-Host Search System: Using Dublin Core to homogenise distributed databases Frehiwot Fisseha FAO/WAICENT AGRIS/CARIS and Documentation Unit.

AGRIS Multi-Host Search System: Using Dublin Core to homogenise distributed databases Frehiwot Fisseha FAO/WAICENT AGRIS/CARIS and Documentation Unit.
SWORD Stories - Easy Deposit Cutting Through Repositories’ Red Tape Sarah Currier Consultancy | E-Learning * Resource Sharing * Web 2.0 * Metadata * Repositories.

SWORD Stories - Easy Deposit Cutting Through Repositories’ Red Tape Sarah Currier Consultancy | E-Learning * Resource Sharing * Web 2.0 * Metadata * Repositories.
LBTO IssueTrak User’s Manual Norm Cushing version 1.3 August 8th, 2007.

LBTO IssueTrak User’s Manual Norm Cushing version 1.3 August 8th, 2007.
Metadata Harvesting The Hague, 13 & 14 January 2009 Julie Verleyen Scientific Coordinator, Europeana Office EuropeanaLocal Knowledge Sharing Workshop.

Metadata Harvesting The Hague, 13 & 14 January 2009 Julie Verleyen Scientific Coordinator, Europeana Office EuropeanaLocal Knowledge Sharing Workshop.

Similar presentations

Ads by Google