LEGO – Rennes, 3 Juillet 2007 Deploying Gfarm and JXTA-based applications using the ADAGE deployment tool Landry Breuil, Loïc Cudennec and Christian Perez IRISA / INRIA, PARIS project-team

LEGO - Rennes, 3 Juillet Data sharing for grid-based applications  Numerical simulations, collaborative design, distributed databases, code- coupled applications  Desirable features: transparency, consistency, uniform access, persistency Grid data sharing service (JuxMem, INRIA/IRISA)  Transparent access to replicated data  Fault tolerance and consistency mechanisms  Data stored in physical memory Global distributed file system (Gfarm, AIST/Univ.Tsukuba)  File fragmentation and replication  Smart replica selection  Data stored on disk: secondary persistent storage Motivations (1/2) Offering persistent storage to a grid data sharing service

LEGO - Rennes, 3 Juillet Motivations (2/2) Proposition of common architecture (#1 out of 3 propositions) Cluster #1Cluster #2 JuxMem Global Data Group (GDG) JuxMem Provider GDG Leader JuxMem Provider JuxMem Provider GFarm GFSD One particular JuxMem provider (GDG leader) flushes data to Gfarm Then, other Gfarm copies can be created using Gfarm’s gfrep command

LEGO - Rennes, 3 Juillet Deploying the Gfarm file system  4 roles: metadata server, metadata cache server, file system node and client node  Dependencies between roles: Tree-based a role should start after its father Deploying the JuxMem service (JXTA-based application)  2 types of peers: rendezvous and edges  Dependencies between peers: Tree-based a peer should start after its rendezvous peer Deployment constraints  Gfarm should be launched before JuxMem  The Gfarm client should share the same node than the JuxMem provider Generic deployment tool ADAGE (INRIA/IRISA) Towards a mixed deployment Using the ADAGE deployment tool

LEGO - Rennes, 3 Juillet Gfarm file system (1/2) Tree-based dependencies server cache client storage client storage Server IP, tcp port Cache IP, tcp port Server IP, tcp port Cache IP, tcpport Configuration file

LEGO - Rennes, 3 Juillet Gfarm file system (2/2) Describing the application <server name="metadata_server" port_master="10602" port_gfmd="10601" port_gfsd="10600" /> <proc name="metadata_cache_server" role="agent" cardinality="1" /> <proc name="fs_node" role="gfsd" cardinality="1" agent="metadata_cache_server" /> <proc name="gf_client" role="client" binary="true" cardinality="1" agent="metadata_cache_server" /> Defining tcp ports for Gfarm roles Naming this metadata cache server Defining which cache server to connect with Binary to execute on the client node

LEGO - Rennes, 3 Juillet JuxMem (1/3) is a JXTA-based application edge rdv edge rdv edge PlatformConfig file: tcp port, IP and tcp port of seed

LEGO - Rennes, 3 Juillet JuxMem (2/3) Describing the JXTA application <edge id="provider" profile_name="p_provider » cardinality="1" rdv="manager" /> <edge id="writer" profile_name="p_writer » cardinality="1" rdv="manager" /> <edge id="reader" profile_name="p_reader » cardinality="1" rdv="manager" /> Defining a profile for the manager Rendezvous instanciation Edge instanciation Seed name Langage: JDL (Mathieu Jan)

LEGO - Rennes, 3 Juillet JuxMem (3/3) Deploying step-by-step edge1 edge2 node1 node2 rdv ADAGE Topology edge3 Rdv: 9701 rdv Writing PlatformConfig Waiting peer up Callback=9701

LEGO - Rennes, 3 Juillet JuxMem (3/3) Deploying step-by-step, dynamic tcp ports assignation edge1 edge2 node1 node2 rdv ADAGE Topology edge3 Rdv: 9701 Edge2: 9703 Edge2(seed=node1:9701) Callback=9703 Edge1(seed=node1:9701) Edge3(seed=node1:9701) Callback=9701 Callback=9705 Edge1: 9701 Edge3: 9705

LEGO - Rennes, 3 Juillet Meta plugin (1/2) Mixed Architecture cluster group GFarm client GFarm file system node JuxMem Client JuxMem manager JuxMem Provider GFarm server GFarm agent

LEGO - Rennes, 3 Juillet Meta plugin (2/2) Describing the mixed architecture <specific id="gfarm-desc » path="./tests/gfarm-appl.xml"/> <specific id="juxmem-desc » path="./tests/jxta-appl.xml"/> gf_client_0 Pointing to the Gfarm description Pointing to the JuxMem description Expressing ack dependency Expressing colocation <plugin name="JXTA » file="tests/meta-gfarm-jxta-ctrl-params-spec.xml"/>

LEGO - Rennes, 3 Juillet Some ADAGE features used by these plugins  File transfer  Wait for file creation  Callback mechanism Some experimentations  Deploying up to 40 Gfarm roles  Adapting the JuxMem basic tutorial to ADAGE Future work  Using large scale configurations Towards a mixed deployment Using the ADAGE deployment tool