CERN-IT-DB Exabyte-Scale Data Management Using an Object-Relational Database: The LHC Project at CERN Jamie Shiers CERN, Switzerland

EB Scale DBs Overview  Brief introduction to CERN & LHC  Why we have massive data volumes  The role of Object-Relational DBs  A Possible Solution…

CERN - The European Organisation for Nuclear Research The European Laboratory for Particle Physics  Fundamental research in particle physics  Designs, builds & operates large accelerators  Financed by 20 European countries (member states) + others (US, Canada, Russia, India, ….)  ~€650M budget - operation + new accelerators  2000 staff users (researchers) from all over the world  LHC (starts ~2005) experiment: 2000 physicists, 150 universities, apparatus costing ~€300M, computing ~€250M to setup, ~€60M/year to run  10+ year lifetime

EB Scale DBs

airport Computer Centre Geneva  27km 

EB Scale DBs

The LHC machine Two counter- circulating proton beams Collision energy TeV 27 Km of magnets with a field of 8.4 Tesla Super-fluid Helium cooled to 1.9°K The world’s largest superconducting structure

EB Scale DBs The LHC Detectors CMS ATLAS LHCb

EB Scale DBs online system multi-level trigger filter out background reduce data volume from 40TB/s to 100MB/s level 1 - special hardware 40 MHz (40 TB/sec) level 2 - embedded processors level 3 - PCs 75 KHz (75 GB/sec) 5 KHz (5 GB/sec) 100 Hz (100 MB/sec) data recording & offline analysis 1000TB/s according to recent estimates

EB Scale DBs Higgs Search H  ZZ    Start with protons (quarks + gluons) Accelerate & collide Observe in massive detectors

EB Scale DBs LHC Data Challenges  4 large experiments, year lifetime  Data rates: ~500MB/s – 1.5GB/s  Data volumes: ~5PB / experiment / year  Several hundred PB total !  Data reduced from “raw data” to “analysis data” in a small number of well-defined steps  Analysed by thousands of users world-wide

CERN-IT-DB LHC Other experiments LHC Other experiments Moore’s law Planned capacity evolution at CERN Mass Storage Disk CPU

EB Scale DBs RAWRAW ESDESD AODAOD TAG random seq. 1PB/yr 100TB/yr 10TB/yr 1TB/yr Data Users Tier0 Tier1

interactive physics analysis batch physics analysis batch physics analysis detector event summary data raw data event reprocessing event reprocessing event simulation event simulation analysis objects (extracted by physics topic) Data Handling and Computation for Physics Analysis event filter (selection & reconstruction) event filter (selection & reconstruction) processed data CER N

EB Scale DBs LHC Data Models  LHC data models are complex!  Typically hundreds ( ) of structure types (classes)  Many relations between them  Different access patterns  LHC experiments rely on OO technology  OO applications deal with networks of objects  Pointers (or references) are used to describe relations Event TrackList TrackerCalor. Track Track Track Track Track HitList Hit Hit Hit Hit Hit

EB Scale DBs CMS:1800 physicists 150 institutes 32 countries World Wide Collaboration  distributed computing & storage capacity

EB Scale DBs physics group regional group CERN Tier2 Lab a Uni a Lab c Uni n Lab m Lab b Uni b Uni y Uni x Tier3 physics department    Desktop Germany Tier 1 USA UK France Italy ………. CERN Tier 1 ………. The LHC Computing Centre

CERN-IT-DB Why use DBs? OK, you have lots of data, but what have databases, let alone Object- Relational DBs got to do with it?

EB Scale DBs Why Not: file = object + GREP ?  It works if you have thousands of objects (and you know them all)  But hard to search millions/billions/trillions with GREP  Hard to put all attributes in file name.  Minimal metadata  Hard to do chunking right.  Hard to pivot on space/time/version/attributes. Internet site:

EB Scale DBs The Reality: its build vs buy  If you use a file system you will eventually build a database system :  metadata,  Query,  parallel ops,  security,….  reorganize,  recovery,  distributed,  replication,

EB Scale DBs OK: so I’ll put lots of objects in a file Do It Yourself Database  Good news:  Your implementation will be 10x faster (at least!)  easier to understand and use  Bad news:  It will cost 10x more to build and maintain  Someday you will get bored maintaining/evolving it  It will lack some killer features: Parallel search Self-describing via metadata SQL, XML, … Replication Online update – reorganization Chunking is problematic (what granularity, how to aggregate)

EB Scale DBs Top 10 reasons to put Everything in a DB 1.Someone else writes the million lines of code 2.Captures data and Metadata, 3.Standard interfaces give tools and quick learning 4.Allows Schema Evolution without breaking old apps 5.Index and Pivot on multiple attributes space-time-attribute-version…. 6.Parallel terabyte searches in seconds or minutes 7.Moves processing & search close to the disk arm (moves fewer bytes (qestons return datons). 8.Chunking is easier (can aggregate chunks at server). 9.Automatic geo-replication 10.Online update and reorganization. 11.Security 12.If you pick the right vendor, ten years from now, there will be software that can read the data.

CERN-IT-DB How to build multi-PB DBs Total LHC data volume: ~300PB VLDBs today: ~3TB Just 5 orders of magnitude to solve… (one per year)

EB Scale DBs Divide & Conquer  Split data from different experiments  Split different data types  Different schema, users, access patterns,…  Focus on mainstream technologies & low- risk solutions  VLDB target: 100TB databases 1.How do we build 100TB databases? 2.How do we use 100TB databases to solve 100PB problem?

EB Scale DBs Why 100TB DBs?  Possible today  Vendors must provide support  Expected to be mainstream within a few years

EB Scale DBs Decision Support (2000) CompanyDB Size* (TB) DBMS Partner Server PartnerStorage Partner SBC10.50NCR LSI First Union Nat. Bank4.50InformixIBMEMC Dialog4.25ProprietaryAmdahlEMC Telecom Italia (DWPT) 3.71IBM Hitachi FedEx Services3.70NCR EMC Office Depot3.08NCR EMC AT & T2.83NCR LSI SK C&C2.54OracleHPEMC NetZero2.47OracleSunEMC Telecom Italia (DA) 2.32InformixSiemensTerraSystems *Database size = sum of user data + summaries and aggregates + indexes

EB Scale DBs Size of the Largest RDBMS in Commercial Use for DSS Source: Database Scalability Program 2000 Terabytes Projected By Respondents

EB Scale DBs BT Visit – July 2001  Oracle VLDB site: Enormous Proof of Concept test in 1999  80TB disk, 40TB mirrored, 37TB usable  Performed using Oracle 8i, EMC storage  “Single instance” – i.e. not cluster  Same techniques as being used at CERN  Demonstrated > 2 years ago!  No concerns for building 100TB today!

EB Scale DBs Physics DB Deployment  Currently run 1-3TB / server  Dual processor Intel/Linux  Scale to ~10TB in a few years sounds plausible  10-node cluster: 100TB  ~100 disks in 2005!  Can we achieve close to linear scalability?  Fortunately, our data is write-once, read-many  Should be good match for shared-disk clusters

EB Scale DBs 100TB DBs & LHC Data Analysis data: 100TB ok for ~10 years One DB cluster  Intermediate: 100TB ~1 year’s data  ~40 DB clusters  RAW data: 100TB = 1 month’s data  400 DB clusters to handle all RAW data 10 / year, 10 years, 4 experiments

EB Scale DBs RAW Data  Processed sequentially ~once / year  Need only current + historic window online  Solution: partitioning + offline tablespaces  100TB = 10 days data  Ample for (re-)processing  Partition the tables  “Old” data  transportable TBS  copy to tape  Drop from catalog  Reload, eventually to a different server, on request

EB Scale DBs Intermediate Data  ~ TB / experiment / year  Yotta-byte DBs predicted by 2020!  1000,000,000 TB ?Can DBMS capabilities grow fast enough to permit just 1 server / experiment?  ++500TB / year  An open question …

EB Scale DBs DB Deployment DAQ cluster: current data – no history export tablespaces to RAW cluster to/from MSS ESD cluster: 1/year? 1? AOD/TAG 1 total? to RCs to/from RCs reconstructanalysis

EB Scale DBs Come & Visit Us!

EB Scale DBs Come Join Us!

EB Scale DBs Summary  Existing DB technologies can be used to build 100TB databases  Familiar data warehousing techniques can be used to handle much larger volumes of historic data  A paper solution to the problems of LHC data management exists: now we just have to implement it