1. The Cambridge Research Computing Service
A brief overview
Dr Paul Calleja
Director of Research Computing
University of Cambridge

2. From EDSAC to Openstack
Cambridge has had an in house research computing development and provision activity in place service for the last 77 years
EDSAC 1949
Darwin 1996
CSD3

3. Strategy – what are we trying to achieve
Deliver cutting-edge research capability using state of the art technologies within the broad area of data-centric high- performance computing, driving research discovery and innovation.
Strategic outcomes
World class innovative data centric research computing provision
Diversity of high value user driven services
Drive research discovery & Innovation within Cambridge and the national science communities that we serve
Delivery economic impact within UK economy

4. Strategy – how do we get there
Continue in house technology innovation, currently focused on :-
Convergence of HPC and Openstack technologies
Next generation tiered storage – strong focus om parallel file systems and NVMe
Large scale genomics analysis software
Hospital clinical informatics platforms
Data analytics and machine learning platforms
Data visualisation platforms
Continue to build best in class, in house capability in:-
System design, integration and solution support
User support
Scientific support RSE (6 FTE)

5. Delivery focus
Driving Discovery, Innovation & Impact

6. Team structure
28 FTE across 6 groups

7. Cambridge research computing capability
Highly resilient HPC DC
100 Cabinets, 30 Kw water cooled
racks, 1000Kw IT Load
People
28 FTE technical team
Skill focus in :-
HPC system integration
Large scale storage
Openstack development
& deployment
Scientific support
Systems
3.7 PF (2500 servers X86 + GPU),
280 node Hadoop system
23 PB storage + Intel Lustre & tape
Value of equipment in service ~£20M

8. Research computing usage and outputs
1600 active from 387 research groups from 42 University departments + National HPC users
Usage growth rate is 28% CAGR year on year for last 9 years, growth rate is expected to grow with Openstack usage models
Research computing services support a current active grant portfolio of £120 – which represents 8% of the Universities annual grant income
Underpinning 2000 publications over the last 9 years, current output ~300 per year

9. CSD3 Platform
1152

10. Peta-4 & Wilkes leading UK academic systems
Largest open access academic X86 system in UK
core skylake nodes – cores 2.0 PF
Fastest academic supercomputer in the UK
KNL 341 nodes 0.5 PF
We can run this as a single heterogeneous system yielding 2.4 PF
Wilkes-2
Largest open access academic GPU system in UK
1.2 PF (20% over design performance)
360 P100

11. Solid state I/O accelerator

12. What does it look like ?
24 Dell EMC PowerEdge R740xd
Each with 24 Intel SSD P4600
0.5 PB of Total Available Space
500 B/s R 350 W
Number 2 in I/O 500
Integration with SLURM and flexible storage orchestrator to be reconfigured to provide maximum performance

13. Cambridge HPC services
Central HPC data analytics service and Service
Pay per use service of large central HPC and storage systems
X86, KNL, GPU
Research computing cloud (new in 2018)
Infrastructure as service
Clinical cloud VM service
Scientific Openstack cloud for IRIS
Secure data storage and archive service (new in 2019)
NHS IGT
ISO 27001
Data Analytics Service (beta)
Hadoop / Sparc

14. Cambridge HPC services
Bio-lab
Develop, deploy and support Open-CB next gen genomics analysis platform
Deploy and support Biocomputing scientific gateways
Deploy and support wide range of medical imaging, microscopy and structure determination platforms
Scientific computing support
Team of scientific programme experts that provide in depth application development support to users.
Very flexible support model
Able to pool fractional FTE funds from grant to part time FTE on long term basis

15. Cambridge HPC services
HPC and Big Data innovation lab
Holds a large range of test /dev HPC Data analytics hardware
Specific Lab engineering resource
Open to third party use
Used to drive HPC and Big Data R & D for RCS and our customers
Strong industrial supply chain collaboration
Strong user driven inputs
Outputs POC’s, case studies and white papers
Drives innovation in research computing solution development and usage for both The University and wider community
System design, procurement and managed hosting service for group owned resources

16. Cambridge HPC services
HPC and Big Data innovation lab
Holds a large range of test /dev HPC Data analytics hardware
Specific Lab engineering resource
Open to third party use
Used to drive HPC and Big Data R & D for RCS and our customers
Strong industrial supply chain collaboration
Strong user driven inputs
Outputs POC’s, case studies and white papers
Drives innovation in research computing solution development and usage for both The University and wider community
System design, procurement and managed hosting service for group owned resources

17. IRIS at Cambridge
36 32 core skylake nodes, 384 GB RAM, dual low latency 25g ethernet, OPA
1 PB lustre
Provisioned as core hours – 2,270,592 per quarter
Bare metal, Openstack VM’s, Openstack slurm as a service
Available in next few weeks fr on boarding IRIS users

18. The EPSRC Tier 2

19. CSD3 EPSRC Tier 2 ecosystem
Oxford, £3M, 22 8way DGX 492 TF, 426 T500, 1PB PFS
EPCC £2.4M 8960 core Broadwell FDR, 300 TF, 0.5 PB PFS,
£5M EPSRC, £1.2 M DiRAC, £2.8M Cam, 24,000 core Skylake, 360 P100, 341 KNL, 5 PB PFS, 10 PB tape, 1PB SSD, 80 nodes Hadoop – total 3.1PF (SKX+KNL =1.7PF 75 T500) GPU system 1.2 PF 100 T500) – OPA EDR islands 2:1
CSD3
UCL £4M core Broadwell 523 TF 395 T500, OPA islands 3:1, 1PF PFS
Loughborough, £3.2M, 1400 core Broadwell, 499 TF 395 T500, OPA islands 3:1
Bristol, £3M, Arm 10,000 core thunderX2 Cray ~300 TF Aries interconnect

20. DiRAC national HPC service
Cambridge are a long standing DiRAC delivery partner
~500 Skylake nodes + 13% of our KNL and GPU system, 3 PB Lustre
Co development partner in DAC and Openstack