1. CODATA Data Archiving Activities CODATA Data Archiving Activities Bill Anderson Co-chair CODATA Data Preservation Task Group ERPANET/CODATA International Workshop on the Selection, Appraisal, and Retention of Scientific Data 15-17 December 2003 Lisbon, Portugal

2. Scientific Data Issues in the News New View of Data Supports Human Link to Global Warming “A re-examination of 24 years of data from weather satellites has found that temperatures are rising in the lower layer of the atmosphere … at a rate that is consistent with what has been measured at the earth’s surface.” Tuesday, November 18, 2003 Lower atmosphere temperature may be rising Controversial satellite data analysis fuels global warming debate. 12 September 2003

3. Scientific Data Issues in the News Volunteers sift through mountains of data in search of medicine that works “A single systematic review of medical treatments requiring hand searching 2200 medical journals (since 1948) and cataloging 1 million published randomized trials.” 27 June 2003 Vol. 300 p. 2024-5

4. Scientific Data Issues in the News 31 October 2003 Vol. 302, pp. 787-788

5. “Digital resources will not survive or remain accessible by accident.” –Bernard Smith, European Commission ICSTI/ICSU/CODATA Digital Preservation Workshop 15 February 2002, Paris, France

6. CODATA: Who are we?  CODATA: Committee on Data for Science and Technology  interdisciplinary Committee of the International Council for Science (ICSU)  established in 1967  focus: organization, management, quality control and dissemination of scientific and technical data (all disciplines)

7. CODATA: Who are we, really?  A member organization collaborating with other organizations on common interests  23 Nations  15 International Scientific Unions  4 Affiliated Organizations  Numerous Supporting Organizations

8. CODATA Archiving Activities  CODATA Working Group formed in 2000  Preliminary web site established  Workshop in Pretoria, S. Africa, May 2002  Annotated list of primary references  Preliminary classification of issues  CODATA Task Group formed focusing on developing countries in 2002  Collaboration with ICSTI on internet portal  Workshop in Beijing, China, June 2004

9. Task Group Objectives  “Preservation and Archiving of Scientific and Technical Data in Developing Countries”  Improve understanding of S&T data management conditions in developing countries  Advance development and adoption of good archiving practices, policies, and tools  Provide interdisciplinary forums  Build a comprehensive directory of managers, experts, and archives

10. Data Archiving Issues  Four categories of issues  Science  Management  Policy  Technical

11. Data Archiving: Scientific Issues  Discipline specific needs and practices of communities  Interdisciplinary and pan-disciplinary values, methods

12. Data Archiving: Scientific Issues  What are scientific data?  OAIS model has reference definitions  Mandates of different archives differ  Data quality control and assurance  Selection and appraisal criteria  Value and relevance of data archived  Language differences  Not all data published in one language  Developing and developed country differences  Nomenclature / taxonomy  Differs inside and across communities  Names and concepts change over time (need to save historical contexts)  Barriers to preservation  original data in some fields on paper only  original data buried in spreadsheets, databases, documents  Interdisciplinary work can yield pan-disciplinary, unmanaged data

13. Data Archiving: Management Issues  Practices and procedures of individuals, archival institutions, and communities

14. Data Archiving: Management Issues  What is archiving?  Relation to other data management functions?  OAIS model distinguishes issues by  archive administration  external and community management  Advocacy needed to secure funding  Data management is not science  Business and organizational models  economic and cost, public and private  incentives and dis-incentives for populating and maintaining deposits  Selection and appraisal criteria and prioritization  Ownership and control  Planning and requirements issues  practices are changing  local practices differ  mandates and objectives differ  what is effective access?  Applications  diversity of customers: scientists, politicians, citizens  Some operational considerations  size  diversity: source, formats, documentation  time horizon for access  changes in data definitions, formats  hardware and software obsolescence

15. Data Archiving: Policy Issues  Rules, regulations, laws, external to the archive that inform, constrain, and assist management

16. Data Archiving: Policy Issues  National, regional & global perspectives  Cultural ownership of data & preference for use  Human data privacy & confidentiality  Environmental data privacy & security  Intellectual property: protection, limits & exceptions  Public vs. private data  Economic trends in public science: privatization and commercialization  National security  Incentives and dis-incentives for managing archive deposits  Enabling legislation & controlling authorities  Freedom of information  policies, regulations & practices  access authorization  Financing and cost recovery policies  economies of scale  unfunded mandates  Rationale for data archiving  pure research needs  cultural, economic & political needs  Policy enforcement mechanisms  Data rights  redistribution  transformation  derivative product rights

17. Data Archiving: Technical Issues  Standards, hardware and software that support data preservation, archiving, and access functions  Mostly discipline independent

18. Data Archiving: Technical Issues  Scientific data and databases are different from literature  size and volume differences  human readability vs. application access  Diversity of data types and formats, and media types, formats and standards  Nomenclature and taxonomy  issues apply to the technology itself  Search capabilities  Who need what and to what ends?  Metadata: difference between access and preservation (OAIS)  Preservation issues  Rapid evolution of technology  Information buried in software is hard to maintain and access  Information in proprietary formats and commercial databases  Directories  potential user authentication and authorization mechanism  potential archive and content discovery mechanism  Standards: OAIS, Open GIS  continuing work is needed  new standards are emerging  financial incentives required  Interoperability among archives

19. Summary  CODATA data archiving activities will pursue opportunities to  Promote and advance long-term management of, and access to, S&T data  Leverage common properties of digital data and information  Learn from previous and ongoing experiences with managing growing collections of digital data and information

20. Discussion