Distributed Monte Carlo Instrument Simulations at ISIS Tom Griffin, ISIS Facility & University of Manchester

What is Distributed Computing The software we use VITESS Specifics McStas Specifics Conclusions Introduction

What do I mean by ‘Distributed Grid’? A way of speeding up large, compute intensive tasks Break large jobs into smaller chunks Send these chunks out to (distributed) machines Distributed machines do the work Collate and merge the results

Spare Cycles Concept Typical PC usage is about 10% Most PCs not used at all after 5pm Even with ‘heavily used’ (Outlook, Word, IE) PCs, the CPU is still grossly underutilised Everyone wants a fast PC! Can we use (“steal?”) their unused CPU cycles? World Community Grid (

Toolkit e.g. COSM Low level toolkit – source code level integration So time consuming work, for each application Entropia DC Grid Trial run at ISIS two years ago. Some success Company bought out and in limbo (?) United Devices Grid MP What we’re currently using Quite expensive Condor Free (academic research project) In our experience 2 yrs ago, not reliable with Windows Possible Software Implementations

The United Devices System Server hardware We use two, dual Xeon servers client licenses Could (will) easily cope with more clients Software Servers run RedHat Linux Advanced Server / DB2 Clients available for Windows, Linux, SPARCs and Macs Programming MGSI – Web Services interface – XML, SOAP Accessed with C++ and Java classes etc Management Console Web browser based Can manage services, jobs, devices etc

Visual Introduction to the Grid

CPU Intensive Low to moderate memory use Not too much file output Coarse grained Command line / batch driven Licensing issues? Suitable / Unsuitable Applications

Program McStas Job wish_simulation Jobstep Workunit sent to a Device Data Set Data Objects within the Grid

1)Think about how to split your data and merge results 2)Wrap and upload your executable 3)Write the application service Pre and Post processing 4)Use the Grid Fairly easy to write Interface to grid via Web Services So far used: C++, Java, Perl, Fortran, C# How to write Grid Programs

Executable + any dlls etc Standard data files Compression Encryption Capture screen output Set Environmental Variables Command Line Wrapping Your Executable

Pre-processing 1) Partition data 2) Package data partitions 3) Log in to the Grid server 4) Create a Job and Job Step 5) Create a Data Set 6) Create Datas and upload data packages 7) Create Workunits 8) Set the Job running Post-Processing 1) Retrieve results 2) Merge results Application Service

Two scenarios: Single large simulation run Split the neutrons into smaller numbers and execute separately Merge results in some way Many smaller runs Parameter scan Monte Carlo Speed-up Ideas

Easy mode of operation: fixed executables + data files Executables held on server Split command line into bits – divide Ncount Vary the random seed Create data packages Upload data packages VITESS – Splitting It

Use GUI to create instrument – Save As Command “Parameter directory” set to “.” VITESS – Running It Submit program parses bat file Substitutes ‘V’ and ‘P’ Removes ‘header’ and ‘footer’ Creates many new bat files with different ‘--Z’s and

Submit program creates many bat files VITESS – Running It C:\My_GRID\VITESSE\VITESSE\build>Vitess-Submit.exe example_job example.bat req_files 20 logging in to as tom.... Adding Vitesse dataset.... Adding Vitesse datas.... 3e+007 neutrons split into 20 chunks, of -n neutrons Total number of Vitesse 'runs' = 20 Uploading data for run #1... Uploading data for run #2.... Uploading data for run #19... Uploading data for run #20... Adding Vitesse datas to system.... Adding job.... Adding jobstep.... Turning on automatic workunit generation.... Closing jobstep.... All done Your job_id is 4878

Web Interface VITESS – Monitoring It

Download the ‘chunks’ Merge Data files DetectedNeutrons.dat : concatenate vpipes : trajectories & count rate Two classes of files 1D - Values: sum & divide by num chunks- - Errors: square, sum and divide 2D –Sum / num of chunks VITESS – Merging It

Many times faster: linear increase Needs verification runs (x3) Typically 11 (potentially) 30+ times faster 12 hours runs in 1 hour! Very large simulations reach random limits VITESS – Advantages and Problems

VITESS – Some Results 176 hours 59 hours 6hrs 20mins

Different executable for every run Executable must be uploaded at run time Split –n into chunks or run many instances (parameter scan) Create data (+ executable) packages Upload packages McStas – Splitting It

Use McGui to create and compile executable Create input file for Submit program McStas – Running It

Large run Submit program breaks up –n##### Uploads new command line + data + executable Parameter Scan Send each run to a separate machine McStas – Running It

Many output files  Separate merge program PGPLOT and Matlab implemented Very similar PGPLOT 1D – intensities: sum and divide. Errors: square, sum and divide. Events: Sum 2D – intensities: sum and divide. Errors: square, sum and divide. Events: Sum Matlab 1D – Same maths, different format 2D – Virtually the same ‘Metadata’ leave untouched McStas – Merging It

Security: Do we trust users? 100 times faster[?] Linux version much faster than Windows [?] How do we merge certain fields? values = ' e '; statistics = 'X0=3.5418; dX= ; Y0= ; dY=1.0288;'; Some issue related to randomness of moderator file McStas – Advantages and Problems

Expansion Proposal accepted for an additional 400 licenses Giving us a total of 480 Change in licensing model Future Developments - Expansion Completed Funded Seeking funding $50k $45k $50k $83k Bottom Line: Costs Setup, server licenses, 80 client licenses + support – $18k – CMSD Total ≈ $250k

Both run well under Grid MP Submit & Retrieve a few hours work Merge a bit more Needs to merge more output formats [?] Issues with very large simulations More info on Grid MP at Conclusions