ShareGrid: architettura e middleware Cosimo Anglano, Massimo Canonico, Marco Guazzone Distributed Computing Systems Research Laboratory Dipartimento di Informatica Universita’ del Piemonte Orientale Alessandria http://dcs.di.unipmn.it

Incontro su ShareGrid, 5/12/2007, Torino Outline Grid Computing panorama ShareGrid architecture The OurGrid middleware ShareGrid applications Conclusions and future works Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Grid computing E-Science: massive use of computers to perform scientific research investigation tools (simulation, data mining, etc.)‏ remote collaboration tools Grid computing conceived as the answer to these computation needs: use of a set of geographically dispersed resources as a single computing platform Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Classical Grids Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Classical Grids Works for you if: you know other people/institutions with similar needs you have “enough” resources to contribute you have enough skilled sysadms to configure and maintain your resources Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Volountary-Computing Grids e.g. Seti@home, you-name-it@home BOINC most prominent middleware Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Voluntary Grid Computing Works only if you have a very good support team to run “the server” invest a good deal of effort in “advertising” have a very high visibility project are in a prestigious institution Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Is a peer-to-peer grid participants join spontaneously, without prior agreements/negotiations, and may leave without prior notice focuses on cooperative resource sharing: “I will let you use my resources (when I don’t need them) if I can use yours” based on the OurGrid middleware carried on at the Universidad Federal de Campina Grande (http://www.ourgrid.org)‏ sponsored by HP Brazil Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) architecture Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) architecture OurGrid P2P network Peer: Site Manager Grid-wide Resource Sharing Resource Broker for submitting users Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) architecture MyGrid: User Interface & Application Scheduling Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) architecture Worker node (gum agent) Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) architecture SWAN: Sandboxing Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino OurGrid Properties Easy to join just download the peer and the worker agent code Easy to install, configure and program No need for specialized support team Easy to use MyGrid is simple to install and use Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino OurGrid Properties Clear incentive to join the system One can’t be worse off by joining the system Noticeable increased response time Free-riding resistant Basic dependability properties Some level of security (SWAN) Automatic resubmission of failed tasks Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid (OurGrid) application model Restricted to Bag-of-Tasks (BoT) applications no communication among tasks Many applications fall in the BoT class: data mining massive search bio computing parameter sweep Monte Carlo simulations fractal calculations image processing ... Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

OurGrid: security mechanisms SWAN: Xen virtual machine, with no network access, and disk access only to a designated partition leverage the fact that BoT applications only communicate to receive input and return the output input/output is done by OurGrid itself that runs in a Xen virtual machine SWAN is optional: a few minutes to required to boot and shutdown the virtual machine requires specific hw/sw configuration (Linux and hw support for virtualization) Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

OurGrid: fostering cooperation To avoid selfish behaviours (a.k.a. free riding) OurGrid uses the Network of Favours mechanism All peers maintain a local balance for all known peers Peers with greater balances have priority Newcomers and peers with negative balance are treated equally The emergent behaviour of the system is that by donating more, one gets more resources back No additional infrastructure is needed Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions Software dependencies Applications require the availability of specific library Current solution: installation of dynamic version of the libraries OR static compilation of applications (requires source code) Next solution: Encapsulation of all execution environment required by each application into a virtual machine Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions Input and output data all the data required and produced by an application must be staged in and out for each task current solution: OurGrid transparently copies the data to and from the worker node Storage affinity scheduling policy Next solution: Integration of a file transfer system Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions Data privacy some application process confidential or sensitive data Current solution: none Next solution: Use of encrypted file system to store data Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions The system does not manage itself One of the most annoying bug consists in unpredictable disappearance of a peer Current solution: Restarting the peer Next solution: Software monitor able to check the availability of each peer and automatically restart those are down Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions Outbound network access peers must be able to go through a firewall (if present) sometimes a laboratory may not be allowed to directly configure the corresponding firewall solution: if connected to the same network of another entity that has outbound access, create a Virtual Private Network Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

ShareGrid: issues and solutions The world is not Linux-centric OurGrid has a strong Linux orientation: peer and client processes run only on *nix machines Next solution none for the peer web portal providing access to ShareGrid (under development) Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Conclusion ShareGrid is a peer-to-peer desktop grid aiming at aggregating and sharing, in a very simple and transparent way, computing resources Good usability Rapid turnaround time Good satisfaction of the users Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino

Incontro su ShareGrid, 5/12/2007, Torino Future work Integration with general-purpose virtualization (go beyond SWAN) Integration of a better storage support Better support for data security and privacy Cosimo Anglano Incontro su ShareGrid, 5/12/2007, Torino