1. Initial job submission and monitoring efforts with JClarens

2. Motivation for the work
Work basically started to make a demo for SOFTEC ( a software competition and exhibition in Pakistan
Aims at demonstrating Clarens as a framework for carrying out interactive Grid-based analysis
Also aims at demonstrating how Clarens facilitates analysis even from resource-limited devices such as PDAs

3. Interfacing with EMA
EMA (End-host monitoring agent): Developed at NUST under supervision of Iosif Legrand
Reports 1-minute load average, CPU clock rate, CPU Usage, Mem Usage, Disk IO etc. statistics
Statistics are stored in a properties file in user’s home directory
JClarens server running on the host periodically reads the properties file and calculates a rough estimate of the load
Server sends the statistics and load estimate to a master JClarens server, where the information is stored in a database

4. Interfacing with EMA Properties JClarens To master EMA file Write
periodically
To master
Read
periodically

5. Interfacing different from Mike Thomas’ proposal
No one JVM to rule them all – two separate JVMs run EMA and JClarens
MonALISA not integrated with JClarens – EMA is just a small module of MonALISA
Tomcat not embedded in EMA - the two run separately

6. Job submission Users select files on JClarens on which to execute jobs
Users submit jobs to master JClarens server
Jobs consist of
The executable’s name
The binary executable
The submit file’s name
The submit file
The names of the input files
Master JClarens decides where to submit the job based on monitoring information
The server with the least load estimate is selected
Master JClarens creates a client of the selected server and uses it to forward the job submission request to the server
The selected server submits the job to Condor and sends back a success / failure notification
If the master itself is the least loaded server, then input files are read from the local file system
Else, the input files are downloaded from the master using file.read
The server creates a temporary staging directory for Condor where it stores the executable, submit file and input files
A job ID is created for the job and sent back to the user

7. Checking Job status User sends the job ID in a status checking request
The master JClarens forwards the request to the server where the job of that job ID was submitted
The server responds with the result of “condor_q”
The result is sent back to the client

8. Retrieving outputs
User sends the job ID in a request to retrieve the output
User also sends the file for which the output is to be retrieved
The master JClarens forwards the request to the server where the job of that job ID was submitted
The server reads sends back the output file to the master, from where it is forwarded back to the client

9. Job submission and execution scenario
JClarens
Authentication +
File transfer
service
Client 6 1 2 3 7
JClarens+Condor
execution
service
Execution
Nodes
JClarens
(Job submission
module) 7 7 5 7 4
Monitoring service
EMA
Monitoring

10. Description of Steps
0. EMA periodically sends monitoring info to JClarens monitoring interface
User authenticates, asks for list of files, browses files
User gets back list of files, selects files, downloads files if required
User submits jobs
Monitoring info is read by JClarens to select best place to submit job
Job is forwarded to best available site
Input Files are downloaded to remote execution site, if necessary
User checks status of jobs repeatedly, gets back outputs when desired etc.

11. TODO:
Remove master JClarens server; instead replicate monitoring info to all servers or invent some other decentralized mechanism
Improve the mechanism of decision-making using monitoring information
Improve and speed up clients, especially GUIs and stuff!