La partecipazione del Gruppo Informatica di Lecce al Progetto EU-US GRID Earth Observation Systems High Energy Physics ASI ESA

Intervento trasversale Sezione INFN-Lecce High Energy Physics

ATLAS ALICECMS VIRGO Cookbook Requirements GRID Middleware development

Giovanni Aloisio Massimo Cafaro UNIV. OF LECCE-Italy Roy Williams CACR/CALTECH Carl Kesselman ISI/USC SARA/Digital Puglia A grid enabled remote sensing digital library

An NPACI International Collaboration Advancing Digital Library Technology NPACI Digital Puglia ASI ASI

Five Emerging Models of Networked Parallelism From The Grid Distributed Computing –|| synchronous processing High-Throughput Computing –|| asynchronous processing On-Demand Computing –|| dynamic resources Data-Intensive Computing –|| databases Collaborative Computing –|| scientists

EU/US Workshop on Large Scientific Databases Annapolis-Maryland 8-11 Sept US Paul Messina (DOE/CACR-Caltech) Roy Williams Maria Zemankova EU Giovanni Aloisio (Univ. Lecce) John Darlington (IPC-UK) Fabrizio Gagliardi (CERN) Organized by CACR-Caltech and CERN Supported in part by the National Science Foundation (Grant IIS ) and the European Commission (EU Information Society Technology Programme) Organizing committee

GRID ISSUES  Scalability  Information Modeling  Interoperability  Information flow  Preservation of databases  Education and outreach

Data Base Scalability the quantity of bulk data in the database the geographical separation of the DB components size of the user community the defining limits of applicability of the DB the duration of the DB project complexity and heterogeneity of DBs to be federated Scalability issues must be considered with respect to:

Data Base Size Hierarchical storage systems Distributed storage systems Parallel data delivery Interoperability of “big data” systems Research on:

For data spread around the system, research on: Clustering which data objects should be stored “near” similar objects? Caching which data objects should be on fast storage? Redundancy which datasets should be stored redundantly in different organizational patterns? Indexing how efficient ways to search scientific data can be created? Summarization when should summary data be computed on-demand, and when pre-computed ?

Networking  A crucial requirement for effective GRID EU-US collaboration is trans-Atlantic data communication that provides: - high bandwidth - high availability - low latency  Regional data centers communicate with each other differently fromthe way they communicate with users The most important metric is throughput

 Scheduled streaming as a new paradigm for the analysis of large amount of data DataStreaming Data Streaming  Data architectures oriented to data movement rather than data storage  Shifting from file-oriented to stream-oriented processing  Constructing new kinds of data management components  Alternative structures for data  New roles for metadata

Distributed Databases The data movement generated by queries to the globally-distributed database must be optimized how queries and processing requests can be formulated to streamline this optimization process? how such a query can be split in separate, locally-executed queries, with machine-specific data access? how the cost, in terms of computation, communication, and time, can be estimated before and during execution

Distributed Databases  Load-balancing how computational work and data are spread around GRID?  Replication what should be replicated among the regional centers?  Protocols for - high-speed - parallel I/O - synchronous and asynchronous delivery - real-time steering and control of running jobs

Information modeling What is the nature of the contents of the database and its catalog? How the DBs interoperability can be achieved? Standardization of scientific data objects

Database Interoperability How can information from multiple collections be fused to extract new knowledge? A common infrastructure providing interoperability between European and US scientific databases common interfaces common information model semantic interoperability

Database Interoperability  Federation of collections - wrappers in front of existing collections that transform the information content into a standard representation - wrappers or servers are installed in front of the storage systems that support access through a common API - wrappers tend to be limited to the manipulation of relatively small data sets - wrappers provide an interoperability capability Large scale data manipulation requires the tight integration of data and compute resources

Security and Authentication  Log-in once to access multiple, heterogeneous services  Clear and unambiguous Access and control policies

Information flow How does information move in a complex system? How do users discover the database and its capabilities? How do users initiate and control a complex processing pipeline?

Preservation of databases  How to ensure that digital scientific data is still available, when necessary, many years in the future?  Preservation description information should be associated with digital objects so that: - the chain of custody and processing history available - quality of the data specified - relationships to other digital objects recognized - digital objects unambiguously identified - information content not altered in an undocumented manner

EGrid - The European Grid Forum Redondo Beach- Agosto 1999

...and many more Now, it is time to put things together