1. © 2013 HP development company L.P. The Contrail Demonstrator and other use cases Christian Temporale, Hewlett Packard christian.temporale@hp.com 1 contrail is co-funded by the EC 7th Framework Programme under Grant Agreement nr. 257438 contrail-project.eu

2. © 2013 HP development company L.P. Why use cases? To evaluate, test and demonstrate the technology developed in the core research Work Packages. – Doing experiments and extensive tests on the CONTRAIL system – Report usability issues, suggest improvements, identify bottlenecks – Demonstrate that the project have both a scientific and business impact Identify new requirements Refine existing requirements – Users’ feedback collection and analysis 2

3. © 2013 HP development company L.P. Contrail use cases Distributed Provision of Geo-referenced Data – Implementation of a Virtual 3D Tourist Guide with an embedded 3D Digital Globe, that let end users to navigate through data provided by federated Content Providers Multimedia Processing Service Marketplace – Implementation of a web portal - that serves as interface for end users, whereas the backend interact with federated Content and Technology providers – offering multimedia services to end users Scientific Data Analysis – Implementation of a scientific data analysis service, mainly oriented to neutron scattering community Electronic Drug Discovery – Bioconductor (†) – The Electronic Drug Discovery enables companies in the life science sector to use modern bioinformatics tools/applications on a federated cloud system 3

4. © 2013 HP development company L.P. UC1 - Distributed Provision of Geo-referenced Data Virtual 3D Tourist Guide (Google Earth like) – Users navigate the 3D digital globe via web browser – Detailed maps and points of interest Data set (from Data providers) – Geo-spatial data provided by SDIs (Spatial Data Infrastructures) – Geo-referenced data provided by Content Providers SDIs and Content Providers – Become part of the Contrail Federated Cloud – Get IT infrastructure from the Contrail Federated Cloud 4

5. © 2013 HP development company L.P. UC1 – Data and content layers 5

6. © 2013 HP development company L.P. UC1 - Distributed provision of geo-referenced data Contrail Federated Cloud Private cloud Internet Public cloud Private Cloud Data Provider VIN Application Provider Configuration file Data Provider 6

7. © 2013 HP development company L.P. UC1 - Architecture 7

8. © 2013 HP development company L.P. UC1 – Deployment in Contrail 8

9. © 2013 HP development company L.P. UC1 – Main benefits from Contrail Fully decentralized approach to 3D Virtual World rendering Enables Data providers to autonomously manage their proprietary data in the federation Choice of location storage inside the cloud Elasticity: scale services for traffic spikes – Peak load management Reduction of initial investments in IT infrastructures for small data providers Quick and efficient processing of complex raw geospatial data Support of constraints on data storage Data protection 9

10. © 2013 HP development company L.P. Transform the selected Use Case into a market oriented demonstrator, open to final users Prove the concept of Contrail in a nearly production scenario, with all security and usability features At the end of the project the demonstrator will become full featured and can be easily transformed in commercial products by the industrial partners Goals of the demonstrator 10

11. © 2013 HP development company L.P. UC2 - Multimedia processing service marketplace: end user standpoint end user sign-up to System subscribe to services use contents download streaming search contents advanced search enrich contents 11 contrail-project.eu contents proposition advanced features

12. © 2013 HP development company L.P. UC2 - Actors Multimedia Marketplace Content Provider Technology Provider searches / uses / enriches contents administers users, services, contents requests digital transformation provides adapted contents Final user Administrator searches / requests contents provides contents contrail-project.eu 12

13. © 2013 HP development company L.P. UC2 - Architecture web server Advanced content search system Content providers Technology providers Client Data mining system s Presentation layer (V) Business layer (C) DAO layer (M) Media digital processing broker User data processing + AC File Payments No SQL DB 13

14. © 2013 HP development company L.P. UC2 - Advanced content search component 14 contrail-project.eu images DB (faces, keys) Face detection, face recognition Model training Advanced content search system New content (video) Image and video processing Machine learrning

15. © 2013 HP development company L.P. UC2 - Advanced content search – Example 1 15 contrail-project.eu Marketplace Business layer Advanced Content Search images DB DoctorStrangelove.avi Marketplace front end “Peter Sellers” Content Provider Doctor Strangelove link machine learning Media digital processing broker Marketplace DB 314 1 2 3 4 5 6 Sellers.jpg Name, code NameOCV ID Sellers314 ContentTags Doctor Strangelove Bombs, Sellers update tags retrieve name 7 8 9 Get contents by tags (Sellers) FaceID 314 Doctor Strangelove 1 2 3 4 5 6 video recognition request front-end activities back-end pre-processing

16. © 2013 HP development company L.P. UC2 - Advanced content search – Example 2 16 contrail-project.eu Marketplace Business layer Advanced Content Search images DB DoctorStrangelove.avi Marketplace front end Content Provider Doctor Strangelove link Media digital processing broker Marketplace DB 314 NameOCV ID Sellers314 ContentTags Doctor Strangelove Bombs, Sellers retrieve name Get contents by tags (Sellers) FaceID 314 Doctor Strangelove image recognition request Update DB Sellers 11 12 1 2 9 8 7 6 5 4 3 10

17. © 2013 HP development company L.P. Data mining: Apache Mahout 17 contrail-project.eu Mahout is a rich set of distributed scalable machine libraries that extracts insights from “Big data” Mahout’s core algorithms for clustering, classification and batch-based collaborative filtering are implemented on top of Hadoop using Map Reduce Mahout features used in the demonstrator: Recommendation mining - based on users’ behavior Data classification – based on existing categorized contents

18. © 2013 HP development company L.P. UC2 - System deployment (3 rd rel.) - Current plan s Presentation layer (V) Business layer (C) DAO layer (M) Media digital processing broker User data processing + AC File Payments MySQL SQL DB Technology Provider contrail DaaS Content Provider WS Advanced content search HTTP Data mining system Contrail Fed. / conpaas Contrail distr. storage GAFS Contrail VEP 18 SQL server manager conpaas SQL service conpaas web application hosting service conpaas map- reduce service conpaas KV store

19. © 2013 HP development company L.P. UC2 - Deployment examples DB deployment on ConPaas Web application deployment on ConPaas 19

20. © 2013 HP development company L.P. UC2 - Main benefits from Contrail Security – Confidentiality and protection of whatever users’ data (contents, personal data, usage data) – Legal constraints on data storage (e.g. geographical location) Reliability – Content storage reliability (→ SLA) – VM reliability (→ SLA) Performance – Bandwidth availability (→ SLA) – System response time (→ SLA) Contrail Federation – Technology providers immediately enabled to sell their know-how directly to external operators (e.g. the Marketplace) without needing to invest on development of products accessible by end users – Content providers need trustable storage infrastructures to preserve their data (their business comes from the commercial value of their contents) 20 contrail-project.eu

21. © 2013 HP development company L.P. UC2 - Other Contrail benefits ConPaaS services – Marketplace deployment (elasticity + scalability) Structured/distributed storage + GAFS – Contents localization – Legal restrictions on stored contents SQL servers (High Level Services) – No need to set up new DBs. Map/Reduce services – Advanced usage data processing – Extraction of statistical information from log files – Search indexes building VIN / VCP – Advanced configuration of network topology 21 contrail-project.eu

22. © 2013 HP development company L.P. UC3 - High Performance Real-Time Scientific Data Analysis Analyses data from neutron scattering experiments on biological samples at the ISIS facility at STFC Runs a large number of model-fitting simulations to determine geometrical shapes of chemical samples Presents a figure of merit for each simulation marking its closeness to the experimental data Small Angle Neutron Scattering 22

23. © 2013 HP development company L.P. UC3 - Neutron source data Complex parameter space for modeling analysis of biological structures 1,000 to 100,000 simulations to determine the molecular structure of biological samples Currently, only the most experienced researchers know how to navigate the parameter space 23

24. © 2013 HP development company L.P. UC3 - Scientific data Experiments are expensive – Booked months in advance – Need to locate “interesting” parameters – Feed experiences back into model Manual search can take weeks – Large amount of time taken in trial-and-error – “Close enough” may result in only 10% of the experimental data being used – wasteful 24

25. © 2013 HP development company L.P. UC3 - Architecture SDA portal: – Simulations of target structures based on complex parameter exploration – Statistical analysis of generated data, allowing ranking on multiple aspects – Feedback to user as to whether more experimental data is required in order to fully categorize the sample 25

26. © 2013 HP development company L.P. UC3: System deployment ConPaaS Bag-of-Tasks service Function: Run a simulation task for each combination of parameters ConPaaS Map/Reduce service Function: Analyze the simulation results and provide a ranking compared to experimental data User defines parameters and experimental data Location of simulation results in GAFS is a parameter to Map/Reduce GAFS is used to store experimental data, simulation results, ranked data, and metadata (parameter settings, etc) 26

27. © 2013 HP development company L.P. UC3 - Main benefits from Contrail “Horizontal scaling” – Allocate additional VM instances as needed (not like Grid, previously used) – Estimate expected completion time Predicable turnaround time → productivity Performance – Grid technologies did not guaranteed system responsiveness Single-Sign On ConPaaS – Bag-of-Tasks – Web application – MapReduce GAFS for sharing data and simulation output Infrastructure reliability Security SLA Enforcement 27

28. © 2013 HP development company L.P. WP12: Use Cases and Applications – CONTRAIL features in Use Cases Features exploited in release 2.0 of Use Cases FunctionalityUse Case 1Use Case 2Use Case 3Use Case 4 Cloud Federation SLA Network Virtualization (VIN) Storage Virtualization (GAFS) VEP Computation Virtualization (IaaS) ConPaaS SQL (High Level Services) ConPaaS Web Servers (Runtime Environment) ConPaaS Task Farming (Runtime Environment) Monitoring Identity Management (AuthN, AuthZ, Identity redirection) Security and QoP Map/Reduce Accounting 28

29. © 2013 HP development company L.P. WP12: Use Cases and Applications – CONTRAIL features in Use Cases Features exploited in release 3.0 of Use Cases FunctionalityUse Case 1Use Case 2Use Case 3 Cloud Federation SLA Network Virtualization (VIN) Storage Virtualization (GAFS) VEP Computation Virtualization (IaaS) ConPaaS SQL (High Level Services) ConPaaS Web Servers (Runtime Environment) ConPaaS Task Farming (Runtime Environment) Monitoring Identity Management (AuthN, AuthZ, Identity redirection) Security and QoP Map/Reduce Accounting 29

30. © 2013 HP development company L.P. Thank you! 30

31. © 2013 HP development company L.P. Funded under: FP7 (Seventh Framework Programme) Area: Internet of Services, Software & Virtualization (ICT- 2009.1.2) Project reference: FP7-IST-257438 Total cost: 11,29 million euro EU contribution: 8,3 million euro Execution: From 2010-10-01 till 2013-09-30 Duration: 36 months Contract type: Collaborative project (generic) contrail is co-funded by the EC 7th Framework Programme 31 http://contrail-project.eu contrail-project.eu

32. © 2013 HP development company L.P. Backup slides