1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Information Services et al. Standup Chair: Morris Riedel With thanks to Sergio Andreozzi..... and the audience 11/21/2016 Amsterdam, 2011-05-131

2. www.egi.eu EGI-InSPIRE RI-261323 Summary

3. www.egi.eu EGI-InSPIRE RI-261323 Overview and Focus 12/05/2011EGI User Virtualization Workshop3

4. www.egi.eu EGI-InSPIRE RI-261323 Scope Information about services living within a VM instance Community-specific  out of scope Perspective: Information about VM instances itself Infrastructure  in scope Information for ‘Scheduling’ (overlaps with Monitoring) Perspective: Discovery Infrastructure  in scope How do we find systems where a dedicated VM image can run? Avoid overlaps (we came back to this again and again…) Monitoring break-out (e.g. health-states, who is running an appliance) Very closely related, also to accounting (tracking resource usage?)

5. www.egi.eu EGI-InSPIRE RI-261323 Common Understanding Discover Information about Hypervisor type/virtualization technology Need careful investigation to understand overlaps with monitoring Assuming there will be a ‘submission of VMs to sites possible’… ‘Scheduling related parameters useful to perform decisions’ Different capabilities often related to compute ones… CPU/GPU, # cores, cpu hardware architectures supported (x86, etc.), Machine speed (i.e. GhZ), bandwidth, network type, memory, disk space, IP address, firewall rules, running time) (Also discussed in VM Mgt break-out) End-user information? More scientific end-user input required… Bottom line: basically all that has been useful in Grids !

6. www.egi.eu EGI-InSPIRE RI-261323 Other Related Work and Topics Available VM appliances  VM Image Repository Also information about hypervisor types/ virtualization technologies Answers: ‘what images work with what VM management environments OVF will be a key for this – but will all support it? Related but not directly clear and discussed Pricing (Understanding costs of running a VM image on different hosts) Security info: Am I allowed to instantiate a VM via a certain endpoint? Information protection: who is allowed to see what information? Consumers of the information End-users and Brokers/meta-schedulers (e.g. WMS, others) Also the ‘services’ itself work with this information ‘locally’ or to be able to contact ‘related services’… setup working interconnections?

7. www.egi.eu EGI-InSPIRE RI-261323 Standards (1) Are there standards we should be using? Information Model Context Many Specifications have covered the same things (mostly compute-based parameters) OCCI defines key/value pairs for VM OVF covers specification of VMs (also network/storage) Work within DMTF (Virtual system profile, Virtual system virtualisation profile, etc.) Leverage GLUE 2.0 with additions? Endpoint to discover ‘VM Management endpoints’ ExecutionEnvironment to discover types of instances where to deploy VM images

8. www.egi.eu EGI-InSPIRE RI-261323 Standards (2) GLUE2 Execution Environments…

9. www.egi.eu EGI-InSPIRE RI-261323 Standards (4) Are there standards we should be using? Information Discovery Interface context OCCI has a generic discovery interface Including resource description based on key/value pairs Some mandatory: e.g. related to # cores, cpu speed CDMI has a discovery service for data Can express capabilities of the storage system (e.g., type of FS)

10. www.egi.eu EGI-InSPIRE RI-261323 Standards (4) OCCI attributes & states (  Monitoring?!)…

11. www.egi.eu EGI-InSPIRE RI-261323 Best practices Are there best practices we should adopt? From ‘Early Cloud Work’ StratusLab will address during 2 nd year (federation of cloud systems) Based on OpenNebula (related to monitoring) Info about the physical system (what is available/used) (Ganglia for status  Monitoring?!) From Grid For information model, evaluate type of info captured in GLUE 2.0 Difficult to upgrade information models Due to dependencies on usage/sensors; re-use experience and try to get it right for the ‘emerging VM work’ Quality of data more than completeness Not all information was always really used (  Investigate which ones)

12. www.egi.eu EGI-InSPIRE RI-261323 Software Maturity & Availability What is the maturity of the software? What is the availability of the software? How to deliver the information? Traditional models like in Grids now (hierarchy of BDII’s) Alternative and complementary with messaging: ActiveMQ How to aggregate/store/query info BDII/OpenLDAP Evaluate approach of repository as DB vs. in-memory Understand scaling from P2P vs. centralized vs. hierarchical  important when EGI users towards 100.000

13. www.egi.eu EGI-InSPIRE RI-261323 Priorities What are the priorities? 1.Overall about Information Model: What capabilities need to be represented 1.Compute The following are basically considered as part of compute: VM Appliances information Type of VMs you can use 2.Storage 3.Network 2.Then work on the ‘transport of information’ via useful systems Take into account lessons learned from Grids with scalability, deployment time, etc. 3.Understand overlaps with monitoring and set boundary

14. www.egi.eu EGI-InSPIRE RI-261323 Gaps, Issues & Concerns Where are the gaps, issues & concerns? Issue is how to we come to a consensus of which exact capabilities are needed (one agreed profile?) (De-facto) standards define the same things, is this something we can steer or is it rather something ‘out of our control zone’  SDOs? Understanding moves of IEEE Cloud standardization ‘lighthouse’ ‘information isolation’ for users about usage of the infrastructure when performed in monitoring Amount of resources available to a particular user Basically part of the information model, but we need to understand where to set the boundary or granularity level ()

15. www.egi.eu EGI-InSPIRE RI-261323 Work items and Next Steps What work is needed to remove these? Filter for GLUEx-based information based on lessons learned Evaluate what is really used in production Understand what information is just overhead Evaluate how to simplify the information needed at finer grain level They have been modeled in Grid but somewhat difficult to capture in a meaningful way Explore “Freshness” of information; should we have a policy in the profile like “info should be not less than 5 minutes’”? Or do we have to think about different models? Current ‘database-based models’ vs. in-memory models

16. www.egi.eu EGI-InSPIRE RI-261323 Work needed to remove the gaps From the technical perspective, moving towards virtualization does not show ‘massive big problems’ Issue with revising the info model, the process can take time, but manageable Analyse overlaps and differences between Grid and Cloud standards related to info model/discovery Hopefully leverage common meetings of SDOs (e.g. next week DMTF/OGF/SNIA will meet to discuss integration of OCCI/CDMI/OVF) Start from there for the further work Minimum profile based on outcome or investigate common profile within the community Then filter which capabilities have been really used in Grids and add necessary new ones

17. www.egi.eu EGI-InSPIRE RI-261323 Notes from the session

18. www.egi.eu EGI-InSPIRE RI-261323 Scope Services, VM images, ? We want to discover info about: Available VM appliances Scheduling related parameters useful to perform decisions Hypervisor type/Virtualization tech. CPU/GPU, # cores… (note from VM Mgt: Bandwidth, network type, memory, disk space, IP address, # cores, CPU, firewall rules, running time) AuthZ info (can I instantiate a VM through a certain endpoint?) Pricing To investigate overlapping of scheduling related info with monitoring requirements (Monitoring: who is starting/running VMs)

19. www.egi.eu EGI-InSPIRE RI-261323 Consumers of the Information End-users Brokers/meta-schedulers (e.g., WMS) Other services

20. www.egi.eu EGI-InSPIRE RI-261323 Standards Information Model Leverage GLUE 2.0 (?) Endpoint to discover VM Management endpoints ExecutionEnvironment to discover types of instances where to deploy VM images OCCI defines key/value pairs for VM OVF covers specification of VMs/network/storage DMTF Virtual system profile Virtual system virtualisation profile Information Discovery Interface OCCI has a generic discovery interface + resource description based on key/value pairs Some mandatory: e.g. related to # cores, cpu speed CDMI has a discovery service for data Can express capabilities of the storage system (e.g., type of FS)

21. www.egi.eu EGI-InSPIRE RI-261323 Best Practices From Cloud StratusLab will address during 2 nd year (federation of cloud systems) Based on OpenNebula (related to monit.) info about the physical system (what is available/used) Ganglia for status From Grid For info model, evaluate type of info captured in GLUE 2.0 Difficult to upgrade information models due to dependencies on usage/sensors; re-use experience and try to get it right Importance for info service, quality of data more than completeness

22. www.egi.eu EGI-InSPIRE RI-261323 Guidelines Evaluate how to simplify the information needed at finer grain level They have been modeled in Grid but somewhat difficult to capture in a meaningful way “Freshness” of information; should we have a policy in the profile like “info should be old less than 5’”?

23. www.egi.eu EGI-InSPIRE RI-261323 Software How to deliver the information Messaging: ActiveMQ How to aggregate/store/query info BDII/OpenLDAP … Evaluate approach of repository as DB vs. in-memory P2P vs. centralized vs. hierarchical

24. www.egi.eu EGI-InSPIRE RI-261323 Priorities Information Model What capabilities need to be represented Compute VM Appliances or as part of VM registries Type of VMs you can use Storage Network

25. www.egi.eu EGI-InSPIRE RI-261323 Gaps Isolation between users about usage of the infrastructure Amount of resources available to a particular user

26. www.egi.eu EGI-InSPIRE RI-261323 Work needed to remove the gaps About isolation of info, not high-priority From the technical viewpoint, moving towards virtualisation does not show big problems Issue with revising the info model, the process can take time Analyse overlaps and differences between Grid and Cloud standards related to info model/discovery maybe leverage common meetings of SDOs (next week DMTF/OGF/SNIA will meet to discuss integration of OCCI/CDMI/OVF) Minimum profile