1. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam The SEALS Platform for Scholarly Research Asun Gómez-Pérez asun@fi.upm.es Ontology Engineering Group Universidad Politécnica de Madrid Acknowledge to: Miguel Esteban Gutiérrez, Raul García-Castro, Nandana Mihindukulasooriya and the SEALS consortium 11.10.2014 1 http://www.seals-project.eu/

2. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 2 Ed Hovy Requirements A kind of framework that graphically depicts the whole scholarly environment/workflow - on this, a set of markers, like Post Its notes, that each indicate a technology available for use at the relevant point - also on this, another set of Post Its notes that each indicate some kind of bottleneck or gap A kind of system that helps me to reproduce experiments that appear in research papers and compare my technology with other technologies 11.10.2014 2 Asun Gómez-Pérez Requirements

3. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 3 11.10.2014 3 Scenario Tool1 Tool4 Tool5 Tool1 Tool4 Tool5    Tool1 Tool2 Tool1 Tool2 Tool3 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5 Tool1 Tool2 Tool6 Tool2 Tool6 Tool7 Tool6 Tool7 Scholarly workflows How do I know which tools are appropriate?: Reproduce Experiments (unbiased, repeatable, …) Tools Test data Evaluations Results My tool My results

4. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam Heterogeneity in the tools to be compared Data Sets – Levels of complexity – Scalability – Synthetic, hand-crafted and real world data Tools – Execution environment – Functionalities – Criteria to evaluate: scalability, interoperability, etc. Metrics – Precision/recall, information losses of data, etc. Evaluations – Fully automatic – The user is in the loop – Hybrid approach – Extrinsic and intrisic Starting point – From scratch – Extend and adapt preliminar approaches 11.10.2014 4

5. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam Test data selection Metrics selection Execution requirements selection Evaluation services implementation Interpretation selection Goals selection Tool features selection Tools selection Evaluation workflows implementation Assumptions selection Criteria selection Dimensions Evaluation design

6. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam SEALS platform for technology evaluation SEALS Platform Features 6 Open (everybody can use it) Scalable (to users, data size) Extensible (to more tests, different technology, more measures) Sustainable (beyond SEALS) Independent (unbiased) Repeatable (evaluations can be reproduced)

7. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 7 Test Data Evaluations Tools The SEALS platform 7 Results 11.10.2014 Semantic Technology Recommendation Framework Results Comparisons

8. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 8 SEALS Service Manager Runtime Evaluation Service SEALS Portal Test Data Repository Service Tools Repository Service Results Repository Service Evaluation Descriptions Repository Service Technology Providers Evaluation Organisers Technology Adopters Software agents SEALS Repositories SEALS Logical Architecture 11.10.2014 8 Virtualization: two execution environments

9. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam Evaluation Execution Process at 10,000ft 9 ER 10.07.2012

10. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam Evaluation services 10 Exploit results 11.10.2014 Tools Test data Evaluations Results Reproduce experiments My tool My results My test data My results Tools Evaluations Use your dataset to evaluate others tool Tools Test data My evaluation My tool My results My test data Define your own experiment

11. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 11 Achievements 11.10.2014 Specification of the Platform Architecture Workflows Implemented in BPEL Test Data Repositor y Service Tools Repositor y Service Results Repositor y Service Evaluation Descriptions Repository Service Implementation and Evaluation of Repositories http://www.seals-project.eu/ontologies/ SEALS entities (dataset, tools, evaluations and results) are described using Ontologies Semantic Technology Recommendation Framework Results Comparisons

12. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam 12 SEALS support to scholarly workflows 12 SEALS Service Manager Runtime Evaluation Service Test Data Repository Tools Repository Results Repository Evaluation Descriptions Repository SEALS Repositories Scholarly Workflow Repository Scholarly Portal Tool1 Tool2 Tool1 Tool2 Tool3 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5 Tool1 Tool2 Tool6 Tool2 Tool6 Tool7 Tool6 Tool7 Tool1 Tool4 Tool5 Tool1 Tool4 Tool5   

13. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam Thank you for your attention!

14. Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam A non exhaustive list of wishes Repositories – Adapt/Extend Data, tools, evaluation and result repositories – A scholarly workflow repository (workflow itself, inputs and expected outputs) Web application for WF visualization, including Post Its and recommendations Definition and execution of Evaluations – User friendly WF editors: BPEL4People – From Fully automatic to the user is in the loop Recommenders – Qualitiative models 11.10.2014