1. Deploying Research in the Real World: The OurGrid Experience
Walfredo Cirne
Universidade Federal de Campina Grande

2. OurGrid Goals and Scope
Provide a real solution to real people
Focus on Bag-of-Tasks (BoT) applications
Serve as basis for our research

3. OurGrid Results
A way to automatically join a grid, sharing resources in a fair manner
Human intervention is not needed
Schedulers that achieve good performance with no information about the grid nor the application
A grid solution for BoT applications that you can download and use

4. Bag-of-Tasks Applications
Data mining
Massive search (as search for crypto keys)
Parameter sweeps
Monte Carlo simulations
Fractals (such as Mandelbrot)
Image manipulation (such as tomography)
And many others…

5. OurGrid Architecture ... Site Manager User Interface SWAN SWAN n1
... n
User Interface
SWAN
SWAN
MyGrid
Sandboxing

6. An Example: Factoring with MyGrid
init mg-services put $PROC ./Fat.class $PLAYPEN
grid1 java Fat output-$TASK
collect mg-services get $PROC $PLAYPEN output-$TASK
grid2 java Fat output-$TASK

7. MyGrid GUI

8. Network of Favors
OurGrid forms peer-to-peer community in which peers are free to join
It’s important to encourage collaboration within OurGrid (i.e., resource sharing)
In file-sharing, most users freeride
OurGrid uses the Network of Favor
All peers maintain a local balance for all known peers
Peers with greater balances have priority
The emergent behavior of the system is that by donating more, you get more resources
No additional infrastructure is needed

9. NoF at Work [1] B 60 D 45 * * A D C E B ConsumerQuery
broker A B C D E
ConsumerQuery
(broadcast)
ConsumerFavor
ProviderFavorReport
ProviderWorkRequest * *
* = no idle resources now

10. NoF at Work [2] B 60 D 45 E * * A D C E B ConsumerQueryA B C D E *
ConsumerQuery
ProviderWorkRequest *
broker
broker
* = no idle resources now

11. Free-rider Consumption
Epsilon is the fraction of resources consumed by free-riders

12. Equity Among Collaborators

13. Scheduling with No Information
Grid scheduling typically depends on information about the grid (e.g. machine speed and load) and the application (e.g. task size)
However, getting good information is hard
Can we schedule without information and deploy the system now?
Work-queue with Replication
Tasks are sent to idle processors
When there are no more tasks, running tasks are replicated on idle processors
The first replica to finish is the official execution
Other replicas are cancelled

14. Work-queue with Replication
8000 experiments
Experiments varied in
grid heterogeneity
application heterogeneity
application granularity
Performance summary:

15. WQR Overhead
Obviously, the drawback in WQR is cycles wasted by the cancelled replicas
Wasted cycles:

16. Data Aware Scheduling
WQR achieves good performance for CPU-intensive BoT applications
However, many important BoT applications are data-intensive
These applications frequently reuse data
During the same execution
Between two successive executions
Storage Affinity uses replication and just a bit of static information to achieve good scheduling for data intensive applications

17. Storage Affinity Results
3000 experiments
Experiments varied in
grid heterogeneity
application heterogeneity
application granularity
Performance summary:
Storage Affinity
X-Suffrage
WQR
Average (seconds)
57.046
59.523
Standard Deviation
39.605
30.213

18. Making it Work for Real...

19. OurGrid Status
OurGrid is open source (GPL) and is available at
We’ve had external contributions
OurGrid latest version is 3.0.2
It contains the 9th version of MyGrid
The Network of Favors is available since version 3.0
We’ve had around 150 downloads
SWAN is in final test phase
To be released with version 3.1, in the end of April 2005

20. PAUÁ: Deploying OurGrid
OurGrid has been deployed in the PAUÁ community by HP Brazil
PAUÁ requires members to identify themselves and submit an adhesion proposal
This makes people all the more comfortable (especially without SWAN)
PAUÁ currently encompasses 7 sites (and growing fast)

21. The Community Status: http://pauastatus.lsd.ufcg.edu.br

22. HIV research with OurGrid
prevalent in Africa O
prevalent in Europe and Americas ?
HIV-1 N
majority in the world M
HIV-2 A B C D F G H J K
18% in Brazil
B,c,F

23. HIV protease + Ritonavir
RMSD
Subtype F
Subtype B

24. The HIV Research Grid
55 machines in 6 administrative domains in the US and Brazil
Task = 3.3 MB input, 1 MB output, 4 to 33 minutes of dedicated execution
Ran 60 tasks in 38 minutes
Speed-up is 29.2 for 55 machines
Considering an 18.5-minute average machine

25. Conclusions
We have an easy-to-join grid solution for Bag-of-Tasks applications working today
Real users provide invaluable feedback for systems research
Delivering results to real users is really cool! :-)

26. Questions?

27. Thank you! Merci! Danke! Grazie! Gracias! Obrigado!
More at