1. www.ci.anl.gov www.ci.uchicago.edu Parallel Scripting on Beagle with Swift Ketan Maheshwari (ketan@mcs.anl.gov) Postdoctoral Appointee (Argonne National Laboratory & University of Chicago)

2. www.ci.anl.gov www.ci.uchicago.edu 2 Overview Swift Highlights Swift Performance on Beagle Swift Components Examples Summary of Swift benefits on Beagle

3. www.ci.anl.gov www.ci.uchicago.edu 3 Swift Highlights Swift is a parallel scripting system for distributed computing – For loosely-coupled applications: application and utility programs linked by exchanging files – Can use local and/or distributed resources: PADS, Beagle, TeraGrid, OSG, clouds, etc. Swift is easy to write – Simple high-level C-like functional language – Small Swift scripts can do large-scale work Swift is easy to run – Contains all services for running workflow: a java application, run from a commandline – Preinstalled as Beagle module : module load swift

4. www.ci.anl.gov www.ci.uchicago.edu 4 Submit host (Laptop, Linux server, …) Workflow status and logs ????? Compute nodes f1 f2 f3 a1 a2 Data server f1 f2 f3 Provenance log script App a1 App a2 site list app list File transport Clouds Swift supports clusters, grids, and supercomputers Download, untar, and run Architecture http://www.ci.uchicago.edu/swift/

5. www.ci.anl.gov www.ci.uchicago.edu 5 Swift Performance on Beagle Decision Support System for Agrotechnology Transfer (DSSAT) Application: Soil Analysis DSSAT Numbers : 120,000 App Calls, dataset Beagle Plots: Peak load of 2000+ cores, completed in < 3hours

6. www.ci.anl.gov www.ci.uchicago.edu 6 Swift Components sites.xml tc.data config rawdata = sim(settings); stats = analysis(rawdata); … compile select resources allocate resourceswrite script  execute

7. www.ci.anl.gov www.ci.uchicago.edu 7 Swift Examples (1) Simple Script A simple SwiftScript type file; app (file output) cat(file input) { cat @input stdout=@output; } file data ; file out ; out = cat(data); Commandline $ swift –sites.file sites.xml –tc.file tc.data catsn.swift

8. www.ci.anl.gov www.ci.uchicago.edu 8 Swift Examples (2) Script with mappers and loop A SwiftScript with mappers and loop type file; app (file output) cat(file input) { cat @input stdout=@output; } file out[] ; file data ; foreach j in [1:@toint(@arg("n","1"))] { out[j] = cat(data); }

9. www.ci.anl.gov www.ci.uchicago.edu 9 More Swift Examples and Demos /lustre/beagle/ketan/beagle-swift-training- examples – sites.xml, tc, cf – 01-app.swift : A simple file copy application – 02-function.swift : functions in Swift – 03-wrapper.swift : Calling wrapper script passing Swift parameters – 04-arrays.swift : Arrays in Swift – 05-regexp.swift : Regular expression mappers

10. www.ci.anl.gov www.ci.uchicago.edu 10 Swift Benefits on Beagle Hassle-free SuperComputing – No need to write qsub scripts and aprun command Easy scaling – sites.xml parameters Easy app wrapping – tc.data Parallel Computation – futures, foreach loops

11. www.ci.anl.gov www.ci.uchicago.edu Thanks! Swift Team: Michael Wilde (PI), Daniel Katz (PI), Mihael Hategan, Justin Wozniak, David Kelly, and many others Swift Students: Alberto Chavez, Papia Rizwan, Jon Monette, Tim Armstrong, Glen Hocky Questions? (or later: swift-user@ci.uchicago.edu)swift-user@ci.uchicago.edu or in person, we are located on fifth floor (5.E.10) at Argonne-TCS Swift Homepage: www.ci.uchicago.edu/swift