1. other servers Java client, ROOT (analysis tool), IGUANA (CMS viz. tool), ROOT-CAVES client (analysis sharing tool), … any app that can make XML-RPC/SOAP calls LHC Data Grid Hierarchy: developed at Caltech Tier 1 Tier2 Center Online System CERN Center PBs of Disk; Tape Robot FNAL Center IN2P3 Center INFN Center RAL Center Institute Workstations ~100-1500 MBytes/sec 2.5-10 Gbps 0.1 to 10 Gbps Tens of Petabytes by 2007-8. An Exabyte ~5-7 Years later. Physics data cache ~PByte/sec ~10-40 Gbps Tier2 Center ~2.5-10 Gbps Tier 0 +1 Tier 3 Tier 4 Tier2 Center Tier 2 Experiment CERN/Outside Resource Ratio ~1:2 Tier0/(  Tier1)/(  Tier2) ~1:1:1 Emerging Vision: A Richly Structured, Global Dynamic System GRID Analysis Environment for LHC Particle PhysicsClient Web server Web server Service 3 rd party applications Client Client Client http/https GAE development (services) MCPS. Policy based Job submission and workflow management portal, developed in collaboration with FNAL and UCSD JobStatus. Access to Job Status information through Clarens and MonALISA, developed in collaboration with NUST JobMon. implements a secure and authenticated method for users to access running Grid jobs, developed in collaboration with FNAL BOSS. Uniform job submission layer developed in collaboration with INFN SPHINX. Grid scheduler developed at UFL CAVES. Analysis code sharing environment developed at UFL Core services (Clarens): Discovery, Authentication, Proxy, Remote file access, Access control management, Virtual Organization management VO Management AuthenticationAuthorizationLogging (remote) File Access ShellKey Escrow MonaLisa ( monitoring) ROOT (analysis)Clarens portalIGUANA (viz. app.) Clarens Grid Portal: Secure cert-based access to services through browser GRID Enabled Analysis: User view of a collaborative desktop This work is partly supported by the Department of Energy as part of the Particle Physics DataGrid project (DOE/DHEP and MICS) and be the National Science Foundation (NFS/MPS and CISE). Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Department of Energy or the National Science Foundation More information: GAE web page: GAE web page: http://ultralight.caltech.edu/web-site/gae Clarens web page: http://clarens.sourceforge.net MonaLisa : http://monalisa.cacr.caltech.edu/ SPHINX: http://sphinx.phys.ufl.edu/ Scientific Exploration at the High Energy Physics Frontier Physics experiments consist of large collaborations: CMS and ATLAS each encompass 2000 physicists from approximately 150 institutes (300-400 physicists in 30 institutes in the US) HEP Challenges: Frontiers of Information Technology Rapid access to PetaByte/ExaByte data stores Secure, efficient, transparent access to heterogeneous worldwide distributed computing and data A collaborative scalable distributed environment for thousands of physicists to enable physics analysis Tracking the state and usage patterns of computing and data resources, to make possible rapid turnaround and efficient utilization of resources Clarens provides a ROOT Plug-In that allows the ROOT user to gain access to Grid services via the portal, for example to access ROOT files at remote locations The Clarens Web Service Framework A portal system providing a common infrastructure for deploying Grid enabled web services Features: Access control to services Session management Service discovery and invocation Virtual Organization management PKI based security Good performance (over 1400 calls per second) Role in GAE: Connects clients to Grid or analysis applications Acts in concert with other Clarens servers to form a P2P network of service providers Two implementations: Python/C using Apache web server Java using Tomcat servlets Monitoring SC04 BWC, 101 GBs A distributed monitoring service system using JINI/JAVA and WSDL/SOAP technologies. A Acts as a dynamic service system and provides the functionality to be discovered and used by any other services or clients that require such information. Can integrate existing monitoring tools and procedures to collect parameters describing computational nodes, applications and network performance. Provides the monitoring information from large and distributed systems to a set of loosely coupled "higher level services" in a flexible, self describing way. This is part of a loosely coupled service architectural model to perform effective resource utilization in large, heterogeneous distributed centers. Policy based access to workflows Tier2 Site Workflow Execution Network Compute Site Scheduler Catalogs Grid Services Web Server Execution Priority Manager Grid Wide Execution Service Data Management Fully- Concrete Planner Fully- Abstract Planner Virtual Data Replica Applications Monitoring Partially- Abstract Planner Metadata HTTP, SOAP, Sphinx MonALISA Clarens BOSS ORCA ROOT FAMOS Discovery,Discovery, Acl management,Acl management, Certificate based accessCertificate based access The GAE Architecture Implementations, developed within Physics and CS community associated with GAE components BOSS Workflow Definitions XML-RPC, JSON, RMI Runjob MCPS Storage JobMon DCache Reservation PlanningMonitoring MonALISA Global Command & Control MonALISA Monitoring Clients MonALISA Clients JobStatus Global view of the system Proactive in minimizing Grid traffic jams Analysis clients talk standard protocols to the Clarens Grid Service Portal Enabling Selection of Workflows (e.g. Monte Carlo simulation, data transfer, analysis) Jobs generated submitted to scheduler, which creates a plan based on monitor information Submission of jobs and feedback on job status MonALISA based monitoring services provide global views of the system MonALISA based components proactively manage sites and networks based on Monitoring information The Clarens portal and MonALISA clients hides the complexity of the Grid services from the client, but can expose it in as much detail as required for e.g. monitoring. Other Clients Web browser ROOT (analysis tool) Python Cojac (detector viz.)/ IGUANA (cms viz tool) “Analysis Flight Deck” JobMon Client JobStatus Client MCPS Client Grid Analysis Environment (GAE) The “Acid Test” for Grids; crucial for LHC experiments Large, diverse, distributed community of users Support for 100s to 1000s of analysis tasks, shared among dozen of sites Widely varying task requirements and priorities Need for priority schemes, robust authentication and security Operates in a severely resource limited and policy constrained global system Dominated by collaboration policy and strategy Requires real-time monitoring; task and workflow tracking; decisions often based on a global system view Where physicists learn to collaborate on analysis across the country, and across world regions Focus is on the LHC CMS experiment but architecture and services can potentially be used in other (physics) analysis environments