TiBS Fermilab – HEPiX-HEPNT Ray Pasetes October 22, 2003

Problem AFS full backups > 40 hours UNIX full backups > 20 hours Operator assisted tape mounts Mandate for automated operations

Investigation March – July 2003 Software evaluated included –Veritas Netbackup –SyncSort –IBM/Tivoli TSM –Amanda –Teradactyl/TiBS

Criteria Support for IBM AFS and OpenAFS Reduce backup windows < 24 hours Complete automation Scalable Reliable Affordable

Evaluation AFS/ OpenAFS < 24 Window Automated ScalableReliableAffordable Veritas NoYes No SyncSort NoYes No IBM Yes No Amanda Yes No* TiBS Yes

Notes: Veritas No support for OpenAFS No ongoing development for AFS No bug fixes for current AFS offering Pricing –Per seat, by OS, #CPUs, and CPU type –Number of tape drives Different pricing for SAN attached drives –Lots of add-on feature charges

Notes: SyncSort No support for AFS Pricing –Per seat, UNIX or Windows –Number of tape drive slots in library Different pricing for SAN attached drives –Pricing came in packs May not fit current equipment. I.E. 170 tape slot pack for 120 slot library 25 pack UNIX licenses for 18 UNIX systems

Notes: IBM Actively supporting AFS and OpenAFS Can do single file restores for AFS Pricing –Vague pricing unless we were willing to commit to significant purchase –Per seat, OS, CPU type

Notes: Amanda AFS support enhanced by internal work Pricing –The support effort to maintain the product was not available

Notes: TiBS Support for AFS and OpenAFS Pricing –Client Per OS, irregardless of clients Enterprise (all supported OSes) –Server Per process (number of parallel backups)

Decision July 2000, decision to deploy TiBS Production -- August 2000 –400GB of data, all AFS –Only part of CSI backups migrated Currently, ~1.6TB – 140+ total systems AFS, HFS+, NTFS, UNIX file systems. –IRIX, Linux, OSX, Solaris, Tru-64, W2K, XP –Backups for CSI and SCS migrated Backup Window ~< 5 hours

TiBS: How does it work? TiBS == True incremental Backup System First backup is a full backup over the network All subsequent backups are incrementals over the network Subsequent weekly and full tapes generated from previous Fulls, Weeklies and incrementals at the server, without client participation All data staged to spooling disk first

More on TiBS Incremental data –Mirrored in disk cache –Merged with previous incrementals daily Does allow for direct-to-tape backup –Can not use merge process One-pass restore

Additional Benefits AFS –Reports on volumes that need salvaging –Consistency check between vldb and fileservers –Look up individual AFS files before restoration General –Hostaudit reports any new partitions/disks on a backup client –Merge process auto-checks integrity of tapes –Lost tapes can be reconstructed from previous tapes –User initiated backups -- laptops –Support has been excellent

Cons May require large caching disk –Typically 10% of total data backed up –Cache needs to be at least as large as largest filesystem Requires a larger library –Or more aggressive tape handling –Reliance on previous tapes to create data

Keep in Mind Non-traditional scalability factors – Size of individual filesystems vs. overall storage –Disk/cache and tape speed vs. network speed –CMU-CS backs up 7TB using a single sparc, 512MB memory, 280GB cache, 4DLT8Ks. Higher reliance on previous backups Failed backups have higher significance

More on Teradactyl Roots are from CMU –Birthplace for AFS –CMU ECE and H&SS departments are beta sites Very open to customizing solutions –K5/KCA authentication

Summary Originally purchased for CSI AFS backups and UNIX server backups (20 systems total) Proved to be very scalable on current hardware and has expanded to two groups, 140+ systems We are now backing up desktops, laptops, Windows and Mac OSX as well as AFS and traditional UNIX systems Does not require enterprise version of Linux OS

Additional Information FNAL – –Ray Pasetes –Kevin Hill Vendor –