Presentation on theme: "Scaling Condor on XSEDE for LIGO"— Presentation transcript:
1 Scaling Condor on XSEDE for LIGO Peter CouvaresSyracuse UniversityLIGO Scientific Collaboration
2 Who am I? What is LIGO? Former Condor Team member (‘99-’08). Now at Syracuse University focused on distributed computing problems for the LIGO Scientific Collaboration, and fostering a research computing community at SU more generally.LIGO (the Laser Interferometer Gravitational-Wave Observatory) is a large scientific experiment to detect cosmic gravitational waves and harness them for scientific research.
3 The Project The Charge: The Challenge: demonstrate whether LIGO can effectively utilize XSEDE resources for its large-scale computing. (And if not, why?)The Challenge:LIGO’s existing computing model doesn’t map perfectly to XSEDE.
4 Four Talks The political story (NSF) The cultural story (LIGO + TACC) The architectural story (what we did)The technical story (how we did it)
5 The Political StoryLIGO plans to buy millions of $$$ of computers later this year to be ready for the Advanced LIGO detectors as they come online.LIGO has always done most of its computing “in-house” on dedicated LIGO clusters, with good results – so we haven’t tried very hard (at least not lately) to utilize opportunistic resources we don’t manage ourselves*.* notable exception =
6 The Political StoryBefore writing us a check, the NSF wanted to understand why we only planned to buy our own private clusters, when some other large NSF projects are successfully using (or contributing to) shared resources.Given the size of the check, the NSF also probably wanted to know whether we were doing our computing sensibly, and weren’t building something unnecessarily inefficient or eccentric.WARNING: this is my speculation based on fourth-hand accounts of other people’s guesses. I could be wrong.
7 The Cultural StoryThe NSF asked LIGO to see if it could run some or all of its large-scale computing work on XSEDE.Stampede was the closest thing to an HTC cluster in XSEDE, so the NSF told LIGO and TACC to work together on it.
11 Shotgun WeddingLIGO: “We don’t need a car with 12 cylinders and molybdenum brakes to commute to work. These Hyundais we’ve got lined up are fine.”TACC: “You need how many cars?!?”
12 Shotgun WeddingLIGO: So, we just need Condor everywhere, no firewall, a bunch of yum repos and RPMs installed on all your machines, single-sign on for our users using their LIGO.ORG credentials, and the ability to run VMs as jobs.TACC: Uh, we don’t normally do any of that stuff. And no way are you running VM jobs.
13 Shotgun Wedding TACC: Have you optimized your code? LIGO: Who do they think we are, amateurs? Have we optimized our code! Harrumph!TACC: Here, look at these FFT results.LIGO: Oh. Uh… wow, that’s faster. Nice!
14 The Cultural StoryLike any shotgun wedding, neither party were thrilled to be at the altar under duress.But we got to work, and quickly dropped the grumpiness.The TACC staff turn out to be great to work with, have all kinds of valuable expertise LIGO can use, and have been extremely helpful.Despite the impedance mismatch, together we succeeded in running a production LIGO workflow, at scale, on Stampede.
15 The Architectural Story Key points of contrast between the LDG and Stampede:Central NFS fileservers (LDG) vs. Lustre DFS (Stampede).Persistent compute nodes w/state (LDG) vs. transient/stateless execute nodes (Stampede).LDG uses persistent local disk for distributed checkpointing and Condor loggingNFS for job input and output, local scratch disks for runtime file i/o (LDG) vs. Lustre for everything (Stampede).Condor batch queue system (LDG) vs. SLURM (Stampede).Scientific Linux 6.1 (LDG) vs. CentOS 6.5 (Stampede).Software pre-installed in system locations on dedicated resources (LDG) vs. local builds on shared resources (Stampede).Long running jobs (LDG) vs. 48h maximum (Stampede)
16 Design ChoiceMake LDG look more like Stampede or Make Stampede look more like LDG Given our experience porting a large body of LDG software and workflows to new OS platforms and versions, we knew it took more time than we had, so we started with the latter and worked back the other way when it was necessary or easier.
18 LDG Overlay on Stampede Glide-in Condor pool via SLURMPersistent Condor central managerPersistent login/submit machineMake heavy use of Condor standard universe checkpointing to handle mismatch between SLURM scheduling policies and long running analysis jobs with unpredictable runtimes.Pre-install LIGO software (RPMs) site-wide on Stampede.Use LDR and Globus for data transfer to Lustre via gridftpSetup LIGO web services in dedicated VMData discovery serviceLIGO.ORG protected web site to post analysis resultsEnable access to LDG with XSEDE credentials and vice-a-versaStampede an early XSEDE adopter of CILogon
19 The GoalValidate the ability to transfer simulated aLIGO data from a LIGO Engineering Run to Stampede and confirm that the CBC offline detection pipeline can run and generate the same results on Stampede as the LDG.Select one LDG site (Syracuse) for detailed comparison runs.Start with the Initial LIGO (iLIGO) pipeline and well understood input data.Perform correctness and scaling testsOptimize performanceSwitch to the aLIGO pipeline currently being developedPerform longer running stability testsIn the background allow for other small scale LIGO tests
20 Round 0 Setup systems for testing: Install LIGO software including Condor – PASSAfter a few iterations official releases of LIGO software from package repositories where installed on all Stampede systems.Setup VM for LIGO to install and manage web services – PASSTook a few iterations including installing extra certificates and mailing physical security tokens but straightforward.Minor change to LIGO web services authentication configuration to handle different network topology at TACC.Setup 10G network transfer of LIGO data via Globus and LDR using gridftp – PASSTook a while to track down a performance issue due to mismatched MTU but eventually solved.Manually support registration of CILogon credentials before XSEDE deployed that during the test
21 Round 1 Analyze one-day of LIGO data on Stampede using iLIGO code: Condor glide-in via SLURM – PASSData transfer via LDR – PASSCentral checkpointing – FAILNetwork firewall issues – FAIL
22 Round 2 Analyze two weeks of data: Solved initial firewall problems – PASSImprove security by moving Condor Central Manager to dedicated host (VM) caused new firewall problems – FAILTry solving checkpoint scaling by having parallel checkpoint writing and central resume – FAIL
23 Round 3 Analyze 6 weeks of data: Condor code patch to support parallel checkpoint save/restore to a shared filesystem without persistent checkpoint servers – PASSScaling to 9,000 concurrent jobs with synchronous checkpiont/resume woke up TACC support team at inconvenient hour – MIX>2000 load avg on submit nodeMoved Condor LOCK and LOG files to /dev/shm to reduce load on Submit machine (temporary solution) – PASSScaling to 25k concurrent jobs hit limit of single submit machine at 13k jobs – MIX
24 Future Submit machine scalability (13k != 25k) Several straightforward ways to solveSubmit fewer but multi-core jobsSplit work between multiple Submit machinesFurther investigate/enhance Condor Shadow scalabilityUse a factory to manage glide-ins automatically.What happens when we don’t have a fortuitous alignment of OS?Virtual Machines (not supported on Stampede)Restrict amount of needed software (focus on production rather than development computing)Port necessary packages as opt-in modulesEnhance LIGO packages to be relocatable as more appropriate for a shared resource
25 LessonsIt takes a lot of work to migrate a "big" computing system to a new environment. Something has to give.It can be done.Miron might say we “cheated” by statically reproducing much of our existing environment on Stampede, rather than bringing it with us – but we had a deadline and it’s a big first step.And the cultural accomplishment inside LIGO may end up being bigger than the technical accomplishment…XSEDE sites like TACC have incredibly valuable expertise – you should take advantage of it. Not being HPC-focused, we underappreciated it before this exercise.
26 Lessons Speaking for myself, not LIGO: We should have been more optimistic, and more humble, up front – but we got there.The NSF should be more clear about what’s going on when it arranges this kind of thing, to limit FUD.While LIGO must manage its own significant computing resources for some work (e.g., low-latency analysis, detector characterization, software development, testing, and training students), we can use shared resources like Stampede today for a large fraction of our computing.Longer-term, LIGO should develop its “grid plumbing” to enable more flexible use of other shared resources that can’t be made to look like LDG sites as easily as Stampede.
28 Acknowledgements Apologies in advance to those I surely forgot… LIGO Stuart Anderson, Duncan Brown, Kent Blackburn, Josh Willis, Patrick Brady, many othersTACCYaakoub El Khamra, Luke Wilson, John Cazes, John McCalpin, Bill Barth, Nathaniel Mendoza, many othersCondorGreg Thain, Alan De Smet, many othersNSFFaceless bureaucrats who forced us out of our rut!