1. 1 DAAC Technology Snapshot Tom Kalvelage Land Processes DAAC Manager For The DAACs ESISS, February 17, 2004

2. 2 Objective of the presentation: Present lessons learned from the collective experience of the DAACs that may be relevant with respect to the Data and Information Management Plan. Present experience with technology and its impacts on the DAACs.

3. 3 ESE Data Centers SEDAC Human Interactions in Global Change GES DAAC-GSFC Upper Atmosphere Atmospheric Dynamics, Ocean Color, Hydrology Global Biosphere, Radiance Data ASDC-LaRC Radiation Budget,Clouds Aerosols, Tropospheric Chemistry ORNL Biogeochemical Dynamics EOS Land Validation ASF SAR Products Sea Ice Polar Processes NSIDC Cryosphere Polar Processes LP DAAC-EDC Land Processes & Features PODAAC-JPL Ocean Circulation Air-Sea Interactions GHRC Hydrologic Cycle & Severe Weather

4. 4 Background The DAACs are located at host institutions that care about the user communities their DAAC serves. Each DAAC is unique, but together we are fully operational on a large scale serving all users. The DAACs use a variety of systems to help users. Cross-DAAC systems like the EOSDIS Core System (ECS) and EOS Data Gateway. Local DAAC systems like LaTIS, TRMM Support System, “Version 0” systems, ASTER Browse Tool, Mercury, etc. Many ‘add-ons’ to existing systems (ECS and non-ECS). Tools and scripts both online and delivered to users. In addition to systems; we also work with the science and applications communities to ensure their needs are met: Users via User Working Groups, Science Investigator-led Processing Systems (SIPS), REASoN CAN members, field campaigns, universities, applications community, education, national and international research projects and so on.

5. 5 Lessons Learned While the DAACs are very diverse and each has its own particular work, experience, and viewpoint, collectively there are some lessons learned across the DAACs that we believe may be relevant.

6. 6 Developer Interaction with Operations Having DAAC Operations work closely with developers has contributed to data system success. User Services made sure new capability would satisfy users. Operators provided realistic ops concepts and requirements. Systems were more likely to be useable and/or efficiently operable as delivered. Sustaining engineers provided installation and maintenance advice and requirements. Integration and test has taken less time and requires less rework (i.e. “measure twice, cut once”) to get from integration, through test, to operations. Ensured that the architecture and interfaces are flexible enough to be maintained and upgraded in the future. On-site developers from off-site development contractor can help, but care must be taken to avoid impacting operations or development.

7. 7 DAAC Engineering/Development Capability A DAAC on-site engineering and development capability has significantly contributed to mission success. It has: enabled the DAACs to evaluate and/or implement new information technology, helping keep data systems current and improving services for users. provided needed capability via scripts, tools, and subsystems when delivered systems (COTS or GFE) fall short. This includes tools for users (e.g., subsetting, reformatting) and for operations (e.g., data management). helped successfully integrate multiple delivered and developed systems (e.g., EOSDIS Core System, EOS Data Gateway, Product Distribution System, etc). provided improved capability to deploy advanced information technology into an operational or near-operational environment for more rapid feedback on operations readiness.

8. 8 User Services User Services (including order service and technical support) has significantly contributed to mission success. User Services has worked best when it is an integral part of the DAAC, able to: influence the engineering and operations of the DAAC on behalf of the users, and educate the users about the DAAC’s engineering and operations. User Services has been essential to explain to users the user search and order interfaces, data formats, data content, algorithms, applicability of the data, and potential ways of using the data. Informal outreach performed by User Services has been instrumental in making the wider user community aware that the data are even available.

9. 9 Big Systems vs. Little Systems Capability can be implemented in a wide spectrum of ways, from doing everything in one large system (one-size-fits-all) to doing each small thing in its own small system (one-size-fits-one). Experience shows that doing 100% in one system doesn’t work due to significant inertia and cost. Anything less than 100% (one-size-fits-most) will work, but collective experience does not tell us the best solution or equilibrium point – its different for each situation. Examples: After evaluation, half of the DAACs did not use the big ECS system; that turned out to be a good choice. A user interface providing some functionality to all data sets allowed unique work to be done by unique user interfaces. Moving unique processing out of ECS to smaller systems, reduced the effort required to address unique requirements. Supporting multiple missions with a ‘one-size-fits-most’ big system (e.g., ECS) allowed us to have more mature systems and processes and less risk when new missions started up.

10. 10 Systems In General Having worked with a number of systems, in general we have some lessons learned about them. Systems designed with the maximum amount of COTS software (e.g., ECS) do have a lot of functionality. However, complex COTS and hardware interdependencies combined with planned obsolescence cause significant integration work. Systems that are not designed to be automated are very difficult to automate later. Systems are more operable, and less costly to operate, if operated at or below specified performance levels (margin). Regarding standards and formats, in general: It’s best to use existing standards and formats. –Data in existing standards and formats were used, and did not require tool development. –Data in new standards and formats were not used, until tools were developed. Translators are generally preferable to new tools.

11. 11 Data Distribution on Hard Media Users continue to demand data distributed on hard media. While hard media distribution is expected to decline, at least partially due to improved networks, several types of users have resisted this trend: Users with poor or expensive network connections (such as international users). Users with limited local storage capacity. Users who just want the data on media.

12. 12 Communications Maintaining good communication and working closely together amongst the DAACs has been productive. The DAACs began meeting together in the early 1990s. The DAACs were advised to form an alliance by a 1998 National Research Council Review, and did so. The DAACs have balanced working independently and together: The DAACs have very different user communities, measurements, products, environments, and cultures. However, user service, data stewardship, and information technology are common to all DAACs. Agreement is by consensus; difficult sometimes, but rewarding and worthwhile.

13. 13 Information Technology Observations Hardware The price/performance trend for utilizing commodity CPUs shows favorable results. A real transitional jump will be when idle desktop PCs and disk can be used for science processing. With disk and tape price/performance improving, full archive on-line disk storage with necessary backup will become more feasible. WAN performance increases will further enable distributed EOSDIS data system architecture. Software Database usage continues to be limited by throughput performance Grid computing is certainly worth pursuing to determine its real applicability. Affordable mass storage will facilitate a proliferation of ‘personally’ customized datasets. Automated agents, already being utilized, will find additional areas for implementation (e.g., ordering data and performing data services). Translators versus standards Standard formats are necessary but not always enforceable. Translators between standards need to be implemented.

14. 14 Information Technology Observations Database Management Methods of managing large quantities of both metadata and data need to be examined with long term archiving in mind. Data Migration Data need to be migrated regularly to be preserved. Plans for migration need to be an integral part of all data management schemes, not an afterthought. Longer term observations We will need to better understand the relationship between data ownership, intellectual property rights, and really long- term preservation. The latter is likely to require much more geographically dispersed replication of data than we have now - and will require automated migration from technology to technology in order to deal with technological obsolescence

15. 15 Some Things That Keep Us Up At Night Examples: Technology Fads - reaching the right and affordable balance between innovation and experience. Additional infrastructure requirements (e.g., security, statistics) that strain fixed resources. Maintaining a highly competent IT staff. Timely receipt of data and documentation from field campaigns. Achieving metadata management and ‘harmony’ across datasets and user communities, including conversion between the many metadata standards used across the sciences. Long term data stewardship and orphaned data sets.

16. 16 Examples: The reuse/adaptability of new technologies including new web- based visualization tools that enhance the use and usefulness of data. Completing an implementation that the user community gets excited about (service, data access tools) because they find it useful. Developing and implementing concepts and technologies including open standards that enable users to extract information from data, and knowledge out of information. Collaborations to develop new products and services which employ maturing data management and advancing information technology. Some Things That Get Us Excited

17. 17 Backup Slides

18. 18 Staff Functions Definitions Operations: (data system operators, some on shift) Sustaining Engineering: (installs, tests software and in some cases hardware, maintains and tunes data system; e.g., data base administration, systems administration of hardware and systems software resources) Development Engineering: (designs, plans, and develops replacement systems and subsystems) User Services: (interface to users, support in filling orders, answering user questions, DAAC outreach to users) Mission (Science Data) Support: (interface to science data providers, DAAC-internal data set coordination, maintenance of data set documentation) Management: overall coordination and resource planning

19. 19 Functional Mix: FY2000 and FY2004 FY2004 OPERATIONS 24% SUS ENGR 24% DEV ENGR 8% USER SERVICES 13% MISSION (SCIENCE DATA) SUPPORT 23% MANAGEMENT 8% FY2000 OPERATIONS 30% SUS ENGR 19% DEV ENGR 7% USER SERVICES 12% MISSION (SCIENCE DATA) SUPPORT 25% MANAGEMENT 7%

20. 20 Functional Mix: Overall 2000 vs 2004 Each DAAC provided percentages for their FTE by functional group.

21. 21 Networks Observations DAACs have at least 100mb connectivity to their users with most in the gigabit range ASF currently has 10 mb connectivity Internal DAAC networks are 100mb or faster, with most at gigabit speeds

22. 22 COTS in FY2004 Examples of COTS packages used at DAACs. This list is not comprehensive.

23. 23 Acronyms ASDC Atmospheric Sciences Data Center ASFAlaska SAR Facility ASTERAdvanced Spaceborne Thermal Emission and Reflection Radiometer CANCooperative Agreement Notice COTSCommercial Off the Shelf CPUComputer Processing Unit DAACDistributed Active Archive Center DBMSData Base Management System ECSEOSDIS Core System EDCEROS Data Center EOSEarth Observing System EOSDISEOS Data and Information System ESEEarth Sciences Enterprise FDDIFiber Distributed Data Interconnect FTEFull Time Equivalent FYFiscal Year GES DAACGSFC Earth Sciences DAAC GFEGovernment Furnished Equipment

24. 24 Acronyms, continued GHRCGlobal Hydrology Resource Center GISGeographic Infromation System GSFCGoddard Space Flight Center HIPPIHigh-Performance Parallel Interface ITInformation Technology LaRCLangley Research Center LaTISLangley TRMM Information System LP DAACLand Processes DAAC NSIDC National Snow and Ice Data Center DAAC ORNL Oak Ridge National Laboratory DAAC PCPersonal Computer PO DAACPhysical Oceanography DAAC REASoNResearch, Education, and Applications Solutions Network SEDACSocioeconomic Data and Applications Center SIPSScience Investigator-led Processing System SWSoftware TRMM Tropical Rain Measurement Mission WANWide Area Network