The GriPhyN Planning Process All-Hands Meeting ISI 15 October 2001

Meeting Goals Clarify our vision and direction –Know how we can make a difference! Map that vision to each experiment –Develop concrete examples of what we want to build Coordinate our research programs Shape toolkit to challenge-problem needs Coordinate overlapping technologies Organize as coordinated subteams with specific missions and defined points of interaction

Meeting Plan Meeting Goals –Discuss our needs; detail our year 2 plan –Create visible, demonstrable, useful results this (project) year Review of Architecture An in-depth look at virtual data and request mgmt Breakouts to focus on technologies Breakouts to focus on experiments Determine toolkit and testbed needs Shape concrete challenge problems –ask: when we’re done building what we propose, what value will we bring to the experiments? –create deliverables that will capture the imagination (and commitment) of the experiments

Refining the Project Plan Current draft is only a framework –team members need to create the real plan Approach: –Vision > VDT > Experiment Plans > CS Proposal: –Vision, framework: needs your input –Miron

Project Plan Responsibilities –VDT: Miron –ATLAS: Jenny –CMS: Rick –LIGO: Albert –SDSS: Jim –Testbed: tbd –CS Research: tbd GOAL: An accepted plan by Fri Nov 2

Project Flow

Project Activities Vision Focusing Experiment Analysis –Use cases, statistics, distributions, data flow patterns, tools, data types, HIPO Architectural Refinement Research –Attach both specific and general problems –Focus and coordinate Toolkit design and development Testbed construction –Two types: GR and GD Challenge Problems Result Integration – the goal!

Architecture Identify new functional modules that need to be created for GriPhyN –Monitoring service, –Request Planner –Virtual Data Catalog service Define how we will connect and integrate our solutions, e.g.: –Virtual data language –DAGman graphs

Technology Focus Talks Breakouts –Monitoring –Virtual data –Replica location –Job execution

Technology: Virtual Data Catalog Structures Catalog manipulation language (VDL) Complex Semantic issues –Transformation signatures –Derivation generation –Fuzzy dependencies and data equivalence

Technology: Replica Location Explore schema Explore distributed replica location service Explore interfaces to metadata databases

Technology: Job Planning and Execution Connections between Condor and Globus Roles of RSL and ClassAds Role of DAGman job desc. language Where planner fits in – key issue Requirements for data staging and storage management – tied to execution plan

Monitoring Drives research Drives planning modules One of the keys to scalability

Experiments Analysis goals –Identify existing data sources –Identify gaps and how to fill them –Identify one person responsible Challenge Problems –Identify a sequence for Y2 and items for Y3-5 Planning goals –Identify a project coordinator responsible for the coordination and plan production: one person

Experiment Specifics ATLAS: Using data challenge progressions; focus on analysis phase rather than simulation-production CMS: Virtual data for MOP, then high throughput and automated planning LIGO: Need to move into more complex LDAS operations and start tracking virtual data –Explore issues of tracking data dependencies that are derived from extensive searches SDSS: starting with cluster finding application –Goldmine of use-case info waiting for analysis of processing to date – need some CS detectives!

Experiment Plan Template Analysis of Experiment IT process Testbed / infrastructure construction VDT deployment Challenge Problem Cycles –Identify and analyze the problem –Communicate and negotiate VDT needs –Design and construct the solution –Negotiate and perform integration

Experiment plan details Analysis documents Infrastructure plans VDT feature requests VDT deployments Challenge problem analysis Challenge solution designs Challenge result (research publications) Challenge solution integration, usage, and support plans

Research VD Representation, provenance, transformation signatures, data generation paradigms Request Planning and Execution –Striving for increasingly greater resource leverage with increasing both ower AND transparency –Monitoring – tieing it all together –Fault tolerance – keeping it all running! User interfaces –How to access the power of the datagrid Scalability: more data, faster

Infrastructure / testbeds Need to create a testbed team within GriPhyN Hope to leverage resources from the GRIDS center Need to create a separate project plan for this activity In this meeting, we should identify requirements for this effort

Meeting Tips Designate both a leader and note taker for all sessions Share the bandwidth / airtime! Call out items that should be noted in the minutes Identify action items Track issues to take offline or devote later sessions to; identify a path to resolution We *can* adjust the schedule as makes sense – suggest this if you see the need your slides to