2. Understanding the Universe with help from OpenStack, CERN and Budapest
Tim Bell
29/05/2013
CERN OpenStack CEE Day

3. CERN was founded 1954: 12 European States
“Science for Peace”
Today: 20 Member States
~ 2,300 staff
~ 1,000 other paid personnel
> 11,000 users
Budget (2013) ~1,000 MCHF
Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom
Candidate for Accession: Romania
Associate Members in Pre-Stage to Membership: Israel, Serbia
Applicant States for Membership or Associate Membership: Brazil, Cyprus (awaiting ratification), Pakistan, Russia, Slovenia, Turkey, Ukraine
Observers to Council: India, Japan, Russia, Turkey, United States of America; European Commission and UNESCO
29/05/2013
CERN OpenStack CEE Day

4. What are the origins of mass ?
29/05/2013
CERN OpenStack CEE Day

5. Matter/Anti Matter Symmetric?
29/05/2013
CERN OpenStack CEE Day

6. Where is 95% of the universe?
29/05/2013
CERN OpenStack CEE Day

7. 29/05/2013
CERN OpenStack CEE Day

8. Over 1,600 magnets lowered down shafts and cooled to -271 C to become superconducting. Two beam pipes, vacuum 10 times less than the moon
29/05/2013
CERN OpenStack CEE Day

9. 29/05/2013
CERN OpenStack CEE Day

10. Collisions
29/05/2013
CERN OpenStack CEE Day

11. July 4, 2012
29/05/2013
CERN OpenStack CEE Day

12. 29/05/2013
CERN OpenStack CEE Day

13. Initial data reconstruction Data distribution
Tier-0 (CERN):
Data recording
Initial data reconstruction
Data distribution
Tier-1 (11 centres):
Permanent storage
Re-processing
Analysis
Tier-2 (~200 centres):
Simulation
End-user analysis
Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid
In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs
The Worldwide LHC Computing grid is used to record and analyse this data. The grid currently runs over 2 million jobs/day, less than 10% of the work is done at CERN. There is an agreed set of protocols for running jobs, data distribution and accounting between all the sites which co-operate in order to support the physicists across the globe.
OpenStack London December 2012
Tim Bell, CERN

14. The CERN Meyrin Data Centre
29/05/2013
CERN OpenStack CEE Day

15. 29/05/2013
CERN OpenStack CEE Day

16. A Big Data Challenge Nearly 100PB storage today
LHC produces up to 35PB/year today
Peaks of 25GB/second
Data rates double in 2015
Experiments will run for 20 years and data needs to be preserved
Exabytes of storage to maintain….
29/05/2013
CERN OpenStack CEE Day

17. 29/05/2013
CERN OpenStack CEE Day

18. New Data Centre Approved
Data centre in Geneva at the limit of electrical capacity at 3.5MW
New centre chosen in Budapest, Hungary
Additional 2.7MW of usable power
Local on-site support for hardware maintenance and installations
29/05/2013
CERN OpenStack CEE Day

19. 29/05/2013
CERN OpenStack CEE Day

20. 29/05/2013
CERN OpenStack CEE Day

21. Good News, Bad News Additional data centre in Budapest now online
Increasing users of CERN’s facilities and higher computing requirements as data rates increase
Staff numbers are fixed, no more people
Materials budget decreasing, no more money
Legacy tools are high maintenance and brittle
How do we maximise our computing resources within these constraints ?
29/05/2013
CERN OpenStack CEE Day

22. Approach Remodel IT services on Cloud layered models
IaaS, PaaS, SaaS
Move to commonly used open source tools
Focus on strong communities and momentum
Implement clouds at scale
Aim for 90% infrastructure virtualised
Exploit ecosystem solutions rather than writing from scratch
29/05/2013
CERN OpenStack CEE Day

23. Training for Newcomers
Buy the book rather than guru mentoring
29/05/2013
CERN OpenStack CEE Day

24. Job Opportunities
29/05/2013
CERN OpenStack CEE Day

25. mcollective, yum Bamboo Puppet AIMS/PXE Foreman JIRA OpenStack Nova
git
Koji, Mock
Yum repo
Pulp
Active Directory /
LDAP
Hardware database
Lemon /
Hadoop /
LogStash /
Kibana
Puppet-DB
29/05/2013

26. Service Models Pets are given names like pussinboots.cern.ch
They are unique, lovingly hand raised and cared for
When they get ill, you nurse them back to health
Cattle are given numbers like vm0042.cern.ch
They are almost identical to other cattle
When they get ill, you get another one
29/05/2013
CERN OpenStack CEE Day

27. CERN Status CERN IT OpenStack Cloud LHC experiment farms
Running Folsom around 500 hypervisors on KVM and Hyper-V
High availability components using load balancing
All Puppet managed to configure OpenStack
Over 100 users creating up to 300 new VMs/day
Cattle service level only
LHC experiment farms
CMS currently running 1,300 hypervisors with 50,000 cores
ATLAS starting to ramp up to a similar size
Other science grid sites moving to private cloud on OpenStack
Brookhaven, IN2P3, FutureGrid, NeCTAR, IHEP, …
29/05/2013
CERN OpenStack CEE Day

28. Block Storage Provider
CERN Network Database
Block Storage Provider
Cinder
Network
Account mgmt system
Compute
Scheduler
Keystone
Nova
Microsoft Active Directory
29/05/2013
CERN OpenStack CEE Day
Horizon
Glance
CERN DB on Demand

29. http://www. eucalyptus
29/05/2013
CERN OpenStack CEE Day

30. 29/05/2013
CERN OpenStack CEE Day

31. Preproduction Service
29/05/2013
CERN OpenStack CEE Day

32. Linux Security Incident
New operational procedures being developed with cloud flexibility
29/05/2013
CERN OpenStack CEE Day

33. Outlook Track stable Grizzly releases in RedHat EPEL Scaling
Up to date but not too close to the leading edge
Scaling
Expect 15,000 hypervisors, 150,000 VMs by 2015
Manageability
Metering, Orchestration with Heat, Bare Metal
Functionality
Load Balancing, High Availability Storage and Pets
29/05/2013
CERN OpenStack CEE Day

34. High Energy Physics Clouds
Distributed cloud computing using HEP and non-HEP clouds
Long-term preservation of software and data of HEP experiments
Simplify the management of heterogeneous in-house resources
Utilize special computing resources attached to the detectors
Use commercial clouds for exceptional computing demands
29/05/2013
CERN OpenStack CEE Day

35. What have we learnt? Automate everything from the beginning
Puppet and Stackforge are a great help
Distributions and appliances make getting started much easier
Constant rate of change requires a different approach
Focus on core technologies and keep up to date
Track new projects but don’t adopt too early unless strategic
Many of our users are cloud aware
Culture changes for legacy application coding and IT services
Communities are major motivators
But administrators need to engage and adapt rather than re-invent
29/05/2013
CERN OpenStack CEE Day

36. Conclusions
CERN IT is re-engineering to deliver additional capacity to 11,000 physicists within fixed resources
Clouds models can simplify current large scale computing infrastructure
OpenStack and its ecosystem allows us to meet this challenge and help others through open source
29/05/2013
CERN OpenStack CEE Day

37. Questions ?
29/05/2013
CERN OpenStack CEE Day