1. Evolving a Legacy System Evolution of the Earth Observing Data and Information System M. Esfandiari 1, H. Ramapriyan 1, J. Behnke 1, E. Sofinowski 2 1 NASA Goddard Space Flight Center, 2 SGT, Inc Jeanne.Behnke@nasa.gov

2. Science Archive Workshop 20072 EOSDIS NASA’s Earth Observing System Data and Information System (EOSDIS) is a petabyte-scale archive of environmental data that supports global climate change research –EOS Mission is to collect Earth remote sensing data for a 15 year global change research program; system designed in 1990’s EOS Data and Information System (EOSDIS) Overview –Designed to receive, process, distribute and archive several terabytes of science data on a daily basis –Provides a distributed information framework (data centers, system elements, interfaces and data model) supporting EOS investigators and other users –Interoperates with data archives of other agencies and countries –Distributed Active Archive Centers (DAACs) –Science Investigator-led Processing Systems (SIPS)

3. Science Archive Workshop 20073 EOSDIS Manages Data for All 24 EOS Measurements

4. Science Archive Workshop 20074 EOSDIS Context Flight Operations, Data Capture, Initial Processing, Backup Archive Data Transport to DAACS/SIPS Science Data Processing, Data Archive, & Distribution Distribution, Access, Interoperability,Reuse EOS Spacecraft Internet Value-Added Providers Interagency Data Centers Int’l Partners & DataCenters Data Acquisition White Sands Complex (WSC) Tracking & Data Relay Satellite (TDRS) Research Users Education Users Distributed Active Archive Centers Instrument Teams and SIPSs Data Processing & Mission Control EOS Polar Ground Stations Media Public ESIP 2/3’s RESACs RACs (Search,order, distribution) (Distribution)

5. Science Archive Workshop 20075 EOSDIS Distributed Data Centers

6. Science Archive Workshop 20076 EOSDIS Performance Metrics for Fiscal Year 2006: Archive Volume: 4.9 PB Granules in Archive: 56 million files Browse Granules in Archive: 14 million files Daily Ingest Rate: 3 TB Daily Distribution Rate: 3 TB Number of Accesses: 37Million Number of Distinct User Accesses: 2 Million

7. Science Archive Workshop 20077 How to Evolve the Legacy System In early 2005, NASA embarked on an EOSDIS Evolution Study to address multi-faceted goals: – Manage archive volume growth – Improve response to science needs – Reduce recurring costs of operations and sustaining engineering – Upgrade aging systems and components – Facilitate a more distributed environment A vision for the 2015 timeframe was developed to guide conduct of the study

8. Science Archive Workshop 20078 Create a Community Vision

9. Science Archive Workshop 20079 Develop a Community Approach Developed a baseline for current operations through workshops and questionnaires Identified cost drivers Explored range of ideas through brainstorming sessions with the EOSDIS elements (SIPS and DAACs) Developed scenarios and predicted outcomes Finally, synthesized an approach based on responses from all participating elements – Analyzed risks and costs of each response – Selected parts of several responses to mitigate overall risk –Discussed key aspects with lead scientists in affected areas –Created a preliminary plan

10. Science Archive Workshop 200710 Assess the Approach Maximize Science Value –Data access is easier and data products are quickly available to science community –Re-organizing our holdings to be more efficient Ex: MODIS data more closely integrated with science community –Potential pathfinder for migration of other data into science communities Substantial Cost Savings –Addresses operational and sustaining engineering –Takes advantage of current IT advances –Investments provide return on value within 3 years Manageable Risk –Minimizes software development efforts –Builds upon existing systems –Utilizes steps within plan as proof of value before proceeding NASA Headquarters approved the EOSDIS Evolution Plan in late 2005; Implementation began in early 2006

11. Science Archive Workshop 200711 Implement the Changes ChangeBenefits Migration to commodity-based hardware Improves scalability and reduces maintenance costs Selectively move to processing products on demand Reduces standing archive Begin transition from tape-based to disk-based archive Allows data to be online and available Increase relationship/control by science community More responsive to science needs, products, tools Re-architect the EOSDIS Core System Removes 750K SLOC (from 1.2M SLOC) and 15 S/W Components Consolidate data holdings into a single system at selected DAACs Reduces operations costs by eliminating multiple systems Facilitate middleware; focuse development of new clients using ECHO Improves user access to inventory and data

12. Science Archive Workshop 200712 Lessons Learned Having a program requirement for continuous technology assessment establishes a culture of innovation An on-going, small-scale, prototyping effort within operational elements is useful Involvement by all affected parties is essential to evolution planning Continuous collection and analysis of system usage metrics (e.g., request patterns for various data products) supported planning for the future Flexible management processes accommodating innovation, speed and efficiency are essential for increasing agility in development despite the higher perceived risks A risk management strategy must be included in planning for evolution Transitions affecting the user community need to be carefully managed with sufficient advance notification