QCloud Queensland Cloud Data Storage and Services 27Mar2012 QCloud1

NATIONAL RDSI 6 primary, 4 additional nodes PB by Gb/sec node interconnect Storage for large national collections Shared access by research communities Research data management On-shore data cloud Low-cost research production quality of service 27Mar2012QCloud2

NATIONAL NeCTAR RESEARCH CLOUD 6 primary nodes (maybe 7) 24,000 cores by 2013 On-demand compute capacity for research Hosted shared services for research communities Virtual laboratories Research tools, workflows Uni of Melbourne QCIF/UQ ANU Monash Uni 27Mar2012QCloud3

QCLOUD COMPONENTS RDSI DATASTORE NeCTAR RESEARCH CLOUD QERN CLOUD GENOMICS CLOUD 5x >15x 27Mar2012QCloud4

QUEENSLAND RDSI NODES ~ 30 Petabytes by 2014 Primary node hosted at UQ (embargoed) Additional Node at JCU (awaiting decision) National focus: ecosystems, genomics 75% Merit Allocated Collections 25% “Collection Development” Can add RDSI+ commercial research storage 27Mar2012QCloud5

COLLECTIONS Market Data Collections Active research projects Collaborations across multiple locations Fast disk, flash memory – immediate access Market Data Collections Active research projects Collaborations across multiple locations Fast disk, flash memory – immediate access Vault Data Collections Rarely accessed data archival storage Broad availability Tape, slow disk Vault Data Collections Rarely accessed data archival storage Broad availability Tape, slow disk 27Mar2012QCloud6

RESEARCH COMMUNITY RDSI NODE SERVICES OWNER Qualification Capture Ingest Bulk Update Curation Metadata (with ANDS) Resilience (multiple copies) Management Entitlements to Access Monitoring Support, Help Desk, Training OWNER Qualification Capture Ingest Bulk Update Curation Metadata (with ANDS) Resilience (multiple copies) Management Entitlements to Access Monitoring Support, Help Desk, Training USER Entitlements to Access Discovery Read Update Duplicate Monitoring Support, Help Desk, Training USER Entitlements to Access Discovery Read Update Duplicate Monitoring Support, Help Desk, Training 27Mar2012QCloud7

TARGET RESEARCH USERS/SERVICES POWER USERS PB-scale data collections Low-level access services Specialised treatment POWER USERS PB-scale data collections Low-level access services Specialised treatment LARGE DATA USERS TB-scale data collections Direct data access, FTP Automated allocations LARGE DATA USERS TB-scale data collections Direct data access, FTP Automated allocations LONG-TAIL USERS GB-scale data collections Cloud access Dropbox style on PC Little allocation surveillance LONG-TAIL USERS GB-scale data collections Cloud access Dropbox style on PC Little allocation surveillance DATA COLLECTION SIZE NUMBER OF USERS 27Mar2012QCloud8

QCIF/UQ NeCTAR NODE 4,000 cores, up to 4,000 virtual machines Research service quaility 70% for NeCTAR Allocated Services 30% for Queensland Allocation 27Mar2012QCloud9

SERVICES HOSTED SERVICES Permanent web sites Research communities Virtual laboratories Research tools Collaboration HOSTED SERVICES Permanent web sites Research communities Virtual laboratories Research tools Collaboration COMPUTATION On-demand virtual machines Off-load “small” jobs from HPC/clusters COMPUTATION On-demand virtual machines Off-load “small” jobs from HPC/clusters 27Mar2012QCloud10

COMPUTE USERS/SERVICES LONG-TAIL USERS HOSTED SERVICES 1-4 cores Small HPC jobs LONG-TAIL USERS HOSTED SERVICES 1-4 cores Small HPC jobs NUMBER OF CORES REQUIRED NUMBER OF USERS Tier 1,2,3 HPC More than 16 cores Specialised treatment Tier 1,2,3 HPC More than 16 cores Specialised treatment MEDIUM COMPUTE USERS 4-16 cores Large memory VMs (1TB+) Automated allocations Smaller HPC jobs MEDIUM COMPUTE USERS 4-16 cores Large memory VMs (1TB+) Automated allocations Smaller HPC jobs 27Mar2012QCloud11

EARLY NODE (QERN) pQERN – Operational October 2011 – Pre-RDSI/NeCTAR – Genomics++ QERN – Delivered February – Operational ~April – 330 cores – 0.5 PB disk – RDSI/NeCTAR services – Early adopter researchers – Operational experience 27Mar2012QCloud12

PROJECTED NODE SCALE-UP 27Mar2012QCloud13 PB Cores NeCTAR Research Cloud RDSI Data Storage Today, QERN

PARTICIPANT ROLES 27Mar2012QCloud14 QCIF Governance, Management, Research Support RDSI Lead Agent NeCTAR Lead Agent UQ RCC Management, SysAdmin JCU Management, Operations UQ ITS Acquisition, Operations Partners Research Support Collection & Service Owners & Users

27Mar2012QCloud15

QERN Progress 27Mar2012QCloud16 Familiarisation Hardware installation Software systems Service Definition and Delivery COO and Solution Architect start Service catalogue Software stack Registration processes Trial usage Production Early adopters Basic services Continuing service introduction Primary Node (Jul2012) Mid Feb

QCLOUD PARTNERS QCIF UQ JCU CQU Griffith QUT USQ USC Bond CSIRO NICTA TERN DERM DTMR DEEDI Ergon Energy 27Mar2012QCloud17

FUNDING FOR QCLOUD 27Mar2012QCloud18 Cash $15M RDSI Primary NoDE $1.5M (contracting) NeCTAR Research Cloud $2M (contracting) Smart Futures Co-investment Fund $3.6M (signed) RDSI Additional NoDE $266K (awaiting decision) RDSI ReDS, DaSh, NRN funds $6.5M+ (projected) DERM, DTMR co-investments $600K (offered) QCIF QERN $200K In- kind $5M UQ Data centre facilities, operations and power consumption plus researcher support $2.3M JCU Data centre facilities, operations and power consumption plus researcher support $700K Other members, CSIRO, NeCTAR, NICTA, TERN, DERM, DTMR, Ergon Energy, QCIF researcher support $2M

Agreements Progress 27Mar2012QCloud19 Smart Futures Signed with DEEDI, Jan2012; funds flow in July Contingent on RDSI, NeCTAR RDSI Final adjustments in progress Signature expected within April NeCTAR Research Cloud Feedback provided to NeCTAR Signature expected within April Operations (UQ, JCU) Drafting agreement text RDSI, NeCTAR agreements as back-to-back schedules Data, service owners To be completed Preliminary arrangements for early users

RISK MANAGEMENT – QCIF Power upgrade at UQ DC2 (JCU also?) – Decision deferred to May Transfer to new UQ data centre – PACE building offer replacing DC2 – network Staff and skills availability Software and skills maturity Authentication, entitlements 27Mar2012QCloud20

RISK MANAGEMENT – NATIONAL Levels of interaction: Lead Agents, other nodes National support and help desk structure RDSI ReDS and DaSh Programs 27Mar2012QCloud21

RISK MANAGEMENT – LONG TERM RDSI value proposition to researchers Growth past 2014 Business model for sustainability Commercial service quality 27Mar2012QCloud22

QUESTIONS? 27Mar2012QCloud23