1. Simulation in a Distributed Computing Environment
S. Guatelli1, J. Moscicki2, M.G. Pia1
1INFN Genova, Italy
2CERN, Geneva, Switzerland

2. Speed of Monte Carlo simulation
Speed of execution is often a concern in Monte Carlo simulation
Often a trade-off between precision of the simulation and speed of execution
Typical use cases
Semi-interactive response
Detector design
Optimisation
Oncological radiotherapy
Very long execution time
High statistics simulation
High precision simulation
Fast simulation
Variance reduction techniques
(event biasing)
Inverse Monte Carlo methods
Parallelisation
Methods for faster simulation response

3. Features of this study
Geant4 application in a distributed computing environment
Architecture
Implications on simulation applications
Environments
PC farm
GRID
Two use cases: Geant4 Advanced Examples
semi-interactive response (brachytherapy)
high statistics (medical_linac)
By-product: results for Geant4 medical application
Quantitative study
results to be submitted for publication

4. Requirements Architectural requirements
Transparent execution in sequential/parallel mode
Transparent execution on a PC farm and on the Grid
Semi-interactive simulation
High statistics simulation
Geant4 brachytherapy
Execution time for 20 M events: 5 hours
Goal: execution time ~ few minutes
Geant4 medical_linac
Execution time for 109 events: ~10 days
Goal: execution time ~ few hours
Reference: sequential mode on a Pentium IV, 3 GHz

5. Parallel mode: local cluster / GRID
Both applications have the same computing model
a job consists of a number of independent tasks which may be executed in parallel
result of each task is a small data packet (few kb), which is merged as the job runs
In a cluster
computing resources are used for parallel execution
user connects to a possibly remote cluster
input data for the job must be available on the site
typically there is a shared file system and a queuing system
network is fast
GRID computing uses resources from multiple computing centres
typically there is no shared file system
(parts of) input data must be replicated in remote sites
network connection is slower than within a cluster

6. Overview Architectural issues Performance tests Conclusions DIANE
How to dianize a Geant4 application
Performance tests
On a single CPU
On clusters
On the GRID
Conclusions
Lessons learned
Outlook
Quantitative, documented results
Publicly distributed:
DIANE
Geant4 application code

7. DIANE http://cern.ch/DIANE Master-Worker R&D project
Developed by J. Moscicki, CERN/IT
R&D project
started in 2001 in CERN/IT with very limited resources
collaboration with Geant4 groups at CERN, INFN, ESA
succesful prototypes running on LSF and EDG
prototype for an intermediate layer between applications and the GRID
Hide complex details of underlying technology
Master-Worker
architectural pattern
DIANE is a layer which provides applications with a easy and convinient way of execution in the distributed, cluster environment.
THE DESIGN GOALS:
- easy customization and adaptability to different needs
- hide details of underlying technology (allow for easy migration)
- location independant – accessible from anywhere in the network
- limited to Master-Worker model which covers most of the typical jobs and needs in HEP
Parallel cluster processing
make fine tuning and customisation easy
transparently using GRID technology
application independent

8. Practical example: Geant4 simulation with analysis
Each task produces a file with histograms
The job result is the sum of histograms produced by tasks
Master-worker model
client starts a job
workers perform tasks and produce histograms
master integrates the results
Distributed Processing for Geant4 Applications
task = N events
job = M tasks
tasks may be executed in parallel
tasks produce histograms/ntuples
task output is automatically combined (add histograms, append ntuples)
Master-Worker Model
Master steers the execution of job, automatically splits the job and merges the results
Worker initializes the Geant4 application and executes macros
Client gets the results

9. UML Deployment Diagram for Geant4 applications
simulation with DIANE
Completely transparent to the user: same Geant4 application code
G4Simulation class is responsible of managing the simulation
manage random number seeds
Geant4 initialisation
macros to be executed in batch mode
termination

10. Development costs
Strategy to minimise the cost of migrating a Geant4 simulation to a distributed environment
DIANE Active Workflow framework
provides automatic communication/synchronization mechanisms
application is “glued” to the framework using a small Python module
in most cases no code changes to the original application are required
load balancing and error recovery policies may be plugged in form of simple python functions
Transparent adaptation for Clusters/GRIDs, shared/local file systems, shared/private queues
Development/modification of application code
original source code unmodified
addition of an interface class which binds together application and M-W framework
The application developer is shielded from the complexity of underlying technology via DIANE

11. Test results Test on a single CPU Test on a dedicated farm (60 CPUs)
Performance of the execution of the dianized Brachytherapy example
Test on a single CPU
Test on a dedicated farm (60 CPUs)
Test on a farm shared with other users (LSF, CERN)
Test on the GRID (LCG)
Tools and libraries:
Simulation toolkit: Geant4 7.0.p01
Analysis tools: AIDA and PI 1.3.3
DIANE: DIANE 1.4.2
CLHEP:
G4EMLOW 2.3

12. Overhead at initialisation/termination
Test on a single dedicated CPU (Intel ®, Pentium IV, 3.00 GHz)
Study execution via DIANE w.r.t. sequential execution
run 1 event
Standalone application
4.6  0.2 s
Application via DIANE, simulation only
8.8  0.8 s
Application via DIANE,
with analysis integration
9.5  0.5 s
Overhead: ~ 5 s, negligible in a high statistics job

13. The overhead of DIANE is negligible in high statistics jobs
Overhead due to DIANE
Test on a single dedicated CPU (Intel ®, Pentium IV, 3.00 GHz)
Study execution via DIANE w.r.t. sequential execution
Execution time vs.
number of events in the job
The overhead of DIANE is negligible in high statistics jobs
Ratio =
with respect to the number of events

14. Farm: execution time and efficiency
Dedicated farm : 30 identical bi-processors (Pentium IV, 3 GHz)
Thanks to Regional Operation Centre (ROC) Team, Taiwan
Thanks to Hurng-Chun Lee (Academia Sinica Grid Computing Center, Taiwan)
Load balancing: optimisation of the number of tasks and workers

15. Optimizing the number of tasks
The job ends when all the tasks are executed in the workers
If the job is split into a higher number of tasks, the chance that the workers finish the tasks at the same time is a higher
Note: the overall time of the job is determined by the last worker to finish the last task
Worker number
Time (seconds)
Worker number
Time (seconds)
Example of a job that can be improved from a performance point of view
Example of a good job balancing

16. Farm shared with other users
Real-life case: farm shared with other users
Execution in parallel mode on 5 workers of CERN LSF
DIANE used as intermediate layer
Preliminary!
The load of the cluster changes quickly in time
The conditions of the test are not reproducible
Highly variable performance

17. Parallel execution in a PC farm
Required production of Brachytherapy: 20 M events
20 M events in sequential mode :
16646 s (~ 4h and 38 minutes) on a a Intel ®, Pentium IV, 3.00 GHz
The same simulation runs in 5 minutes in parallel on 56 CPUs
appropriate for clinical usage
Similar results for Geant4 medical_linac Advanced Example
production can become compatible with usage for the verification of IMRT treatment planning
sequential execution requires ~ 10 days to obtain significant results

18. Running on the Grid (LCG)
G4Brachy executed on the GRID (LCG)
nodes located in Spain, Russia, Italy, Germany, Switzerland
Conditions of the test
The load of the GRID changes quickly in time
The conditions of the test are not reproducible
Efficiency
The evaluation of the efficiency with the same criterion as in a dedicated farm does not make much sense in this context
Study the “efficiency” of DIANE as automated job management w.r.t. manual submission through simple scripts

19. Test results Execution on the GRID through DIANE,
20 M events,180 tasks, 30 workers
Execution on the GRID, without DIANE
Worker number
Worker number
Time (seconds)
Time (seconds)
Through DIANE:
- All the tasks are executed successfully on 22 workers
- Not all the workers are initialized and used: on-going investigation
Without DIANE:
- 2 jobs not successfully executed due to set-up problems of the workers

20. How the GRID load changes
Execution time of Brachytherapy in two different conditions of the GRID
DIANE used as intermediate layer
Worker number
Worker number
Time (seconds)
Time (seconds)
20 M events, 60 workers initialized, 360 tasks
Very different result!

21. Farm/GRID execution Preliminary indication
Brachy, 20 M events, 180 tasks
Taipei cluster:
29 machines, 734 s ~ 12 minutes
GRID:
27 machines, 1517 s ~ 25 minutes
Preliminary indication
The conditions are not reproducible

22. Lessons learned DIANE as intermediate layer Load balancing
Transparency
Good separation of the subsystems
Good management of CPU resources
Negligible overhead
Load balancing
A relatively large number of tasks increases the efficiency of parallel execution in a farm
Trade-off between optimisation of task splitting and overhead introduced
Controlled and real life situation is quite different in a farm
need dedicated farm for critical usage (i.e. hospital)
Grid
highly variable environment
not mature yet for critical usage
automated management through a smart system is mandatory
work in progress, details still to be understood quantitatively

23. Conclusions
General approach to the execution of Geant4 simulation in a distributed computing environment
transparent sequential/parallel application
transparent execution on a local farm or on the Grid
user code is the same
Quantitative, documented results
reference for users and for further improvement
on-going work to understand details