1. PROOF – Parallel ROOT Facility
Fons Rademakers
Bring the KB to the PB not the PB to the KB
March, 2003
CHEP'03

2. PROOF
Collaboration between core ROOT group at CERN and MIT Heavy Ion Group
Part of and based on ROOT framework
Uses heavily ROOT networking and other infrastructure classes
Currently no external technologies
March, 2003
CHEP'03

3. Main Motivation
Design a system for the interactive analysis of very large sets of ROOT data files on a cluster of computers
The main idea is to speed up the query processing by employing parallelism
In the GRID context, this model will be extended from a local cluster to a wide area “virtual cluster”. The emphasis in that case is not so much on interactive response as on transparency
With a single query, a user can analyze a globally distributed data set and get back a “single” result
The main design goals are:
transparency, scalability, adaptability
March, 2003
CHEP'03

4. Parallel Script Execution
#proof.conf
slave node1
slave node2
slave node3
slave node4
Local PC
Remote PROOF Cluster
proof
proof = master server
root
stdout/obj
proof
proof = slave server
ana.C
proof
proof
proof
*.root
TFile
node1
ana.C
TNetFile
*.root
$ root
root [0] tree->Process(“ana.C”)
root [1] gROOT->Proof(“remote”)
root [2] chain->Process(“ana.C”)
$ root
root [0] .x ana.C
root [1] gROOT->Proof(“remote”)
$ root
root [0] .x ana.C
$ root
node2
TFile
*.root
node3
TFile
*.root
node4
March, 2003
CHEP'03

5. PROOF - Architecture Data Access Strategies Transparency
Local data first, also rootd, rfio, SAN/NAS
Transparency
Input objects copied from client
Output objects merged, returned to client
Scalability and Adaptability
Vary packet size (specific workload, slave performance, dynamic load)
Heterogeneous Servers
Migrate to multi site configurations
March, 2003
CHEP'03

6. Workflow For Tree Analysis – Pull Architecture
Slave 1
Master
Slave N
Process(“ana.C”)
Process(“ana.C”)
Initialization
Packet generator
Initialization
GetNextPacket()
GetNextPacket()
Process
0,100
100,100
Process
GetNextPacket()
GetNextPacket()
Process
200,100
300,40
GetNextPacket()
Process
GetNextPacket()
Process
340,100
440,50
Process
GetNextPacket()
GetNextPacket()
Process
490,100
590,60
Process
SendObject(histo)
SendObject(histo)
Wait for next
command
Add
histograms
Wait for next
command
Display
histograms
March, 2003
CHEP'03

7. Additional Issues Error handling Authentication
Death of master and/or slaves
Ctrl-C interrupt
Authentication
Globus, ssh, kerb5, SRP, clear passwd, uid/gid matching
March, 2003
CHEP'03

8. Running a PROOF Job // Analyze TChains in parallel
gROOT->Proof(“proof.cern.ch”);
TChain *chain = new TChain(“AOD");
chain->Add("lfn://alien.cern.ch/alice/prod2002/file1");
. . .
chain->Process(“myselector.C”);
// Analyze generic data sets in parallel
gROOT->Proof(“proof.cern.ch”);
TDSet *objset = new TDSet("MyEvent", "*", "/events");
objset->Add("lfn://alien.cern.ch/alice/prod2002/file1");
. . .
objset->Add(set2003);
objset->Process(“myselector.C++”);
March, 2003
CHEP'03

9. PROOF Scalability 8.8GB, 128 files 1 node: 325 s
32 nodes in parallel: 12 s
32 nodes: dual Itanium II 1 GHz CPU’s,
2 GB RAM, 2x75 GB 15K SCSI disk,
1 Fast Eth, 1 GB Eth nic (not used)
Each node has one copy of the data set
(4 files, total of 277 MB), 32 nodes:
8.8 Gbyte in 128 files, 9 million events
March, 2003
CHEP'03

10. PROOF and Data Grids Many services are a good fit
Authentication
File Catalog, replication services
Resource brokers
Monitoring
 Use abstract interfaces
Phased integration
Static configuration
Use of one or multiple Grid services
Driven by Grid infrastructure
March, 2003
CHEP'03

11. Different PROOF–GRID Scenarios
Static stand-alone
Current version, static config file, pre-installed
Dynamic, PROOF in control
Using grid file catalog and resource broker, pre-installed
Dynamic, ALiEn in control
Idem, but installed and started on the fly by AliEn
Dynamic, Condor in control
Idem, but allowing in addition slave migration in a Condor pool
March, 2003
CHEP'03

12. Running PROOF Using AliEn
TGrid *alien = TGrid::Connect(“alien”);
TGridResult *res;
res = alien->Query(“lfn:///alice/simulation/ /V0.6*.root“);
TDSet *treeset = new TDSet("TTree", "AOD");
treeset->Add(res);
gROOT->Proof(res); // use files in result set to find remote nodes
treeset->Process(“myselector.C”);
// plot/save objects produced in myselector.C
. . .
March, 2003
CHEP'03

13. Near Future Working with some early users
Ironing out of several remaining issues
Writing install and user guide
After CHEP general release
Part of standard ROOT distribution
Still to do for analysis:
Support for: event lists, friend trees
Still to do for grid:
Interfacing to file catalogs, resource broker
Multi site PROOF sessions
March, 2003
CHEP'03