Evolving a Legacy System Evolution of the Earth Observing Data and Information System M. Esfandiari 1, H. Ramapriyan 1, J. Behnke 1, E. Sofinowski 2 1 NASA Goddard Space Flight Center, 2 SGT, Inc

Science Archive Workshop EOSDIS NASA’s Earth Observing System Data and Information System (EOSDIS) is a petabyte-scale archive of environmental data that supports global climate change research –EOS Mission is to collect Earth remote sensing data for a 15 year global change research program; system designed in 1990’s EOS Data and Information System (EOSDIS) Overview –Designed to receive, process, distribute and archive several terabytes of science data on a daily basis –Provides a distributed information framework (data centers, system elements, interfaces and data model) supporting EOS investigators and other users –Interoperates with data archives of other agencies and countries –Distributed Active Archive Centers (DAACs) –Science Investigator-led Processing Systems (SIPS)

Science Archive Workshop EOSDIS Manages Data for All 24 EOS Measurements

Science Archive Workshop EOSDIS Context Flight Operations, Data Capture, Initial Processing, Backup Archive Data Transport to DAACS/SIPS Science Data Processing, Data Archive, & Distribution Distribution, Access, Interoperability,Reuse EOS Spacecraft Internet Value-Added Providers Interagency Data Centers Int’l Partners & DataCenters Data Acquisition White Sands Complex (WSC) Tracking & Data Relay Satellite (TDRS) Research Users Education Users Distributed Active Archive Centers Instrument Teams and SIPSs Data Processing & Mission Control EOS Polar Ground Stations Media Public ESIP 2/3’s RESACs RACs (Search,order, distribution) (Distribution)

Science Archive Workshop EOSDIS Distributed Data Centers

Science Archive Workshop EOSDIS Performance Metrics for Fiscal Year 2006: Archive Volume: 4.9 PB Granules in Archive: 56 million files Browse Granules in Archive: 14 million files Daily Ingest Rate: 3 TB Daily Distribution Rate: 3 TB Number of Accesses: 37Million Number of Distinct User Accesses: 2 Million

Science Archive Workshop How to Evolve the Legacy System In early 2005, NASA embarked on an EOSDIS Evolution Study to address multi-faceted goals: – Manage archive volume growth – Improve response to science needs – Reduce recurring costs of operations and sustaining engineering – Upgrade aging systems and components – Facilitate a more distributed environment A vision for the 2015 timeframe was developed to guide conduct of the study

Science Archive Workshop Create a Community Vision

Science Archive Workshop Develop a Community Approach Developed a baseline for current operations through workshops and questionnaires Identified cost drivers Explored range of ideas through brainstorming sessions with the EOSDIS elements (SIPS and DAACs) Developed scenarios and predicted outcomes Finally, synthesized an approach based on responses from all participating elements – Analyzed risks and costs of each response – Selected parts of several responses to mitigate overall risk –Discussed key aspects with lead scientists in affected areas –Created a preliminary plan

Science Archive Workshop Assess the Approach Maximize Science Value –Data access is easier and data products are quickly available to science community –Re-organizing our holdings to be more efficient Ex: MODIS data more closely integrated with science community –Potential pathfinder for migration of other data into science communities Substantial Cost Savings –Addresses operational and sustaining engineering –Takes advantage of current IT advances –Investments provide return on value within 3 years Manageable Risk –Minimizes software development efforts –Builds upon existing systems –Utilizes steps within plan as proof of value before proceeding NASA Headquarters approved the EOSDIS Evolution Plan in late 2005; Implementation began in early 2006

Science Archive Workshop Implement the Changes ChangeBenefits Migration to commodity-based hardware Improves scalability and reduces maintenance costs Selectively move to processing products on demand Reduces standing archive Begin transition from tape-based to disk-based archive Allows data to be online and available Increase relationship/control by science community More responsive to science needs, products, tools Re-architect the EOSDIS Core System Removes 750K SLOC (from 1.2M SLOC) and 15 S/W Components Consolidate data holdings into a single system at selected DAACs Reduces operations costs by eliminating multiple systems Facilitate middleware; focuse development of new clients using ECHO Improves user access to inventory and data

Science Archive Workshop Lessons Learned Having a program requirement for continuous technology assessment establishes a culture of innovation An on-going, small-scale, prototyping effort within operational elements is useful Involvement by all affected parties is essential to evolution planning Continuous collection and analysis of system usage metrics (e.g., request patterns for various data products) supported planning for the future Flexible management processes accommodating innovation, speed and efficiency are essential for increasing agility in development despite the higher perceived risks A risk management strategy must be included in planning for evolution Transitions affecting the user community need to be carefully managed with sufficient advance notification