1. Data Archives: Migration and Maintenance Douglas J. Mink Telescope Data Center Smithsonian Astrophysical Observatory http://tdc-www.harvard.edu NSF 2004-03-23

2. Archiving Issues NSF 2004-03-23 What do we save? Reduced Data? Raw Data? Calibration data? Data Products? Publications? How do we access the data? Google Search? (ADS) Through discipline Portal(s)? (VO registries) Through Data Center? Where does the data live? One or few Data Center(s)? Many Data Centers reachable through few Portals Number of Centers is limited by long-term funding

3. Migration Issues NSF 2004-03-23 Why do we migrate archives? Better access, Cheaper storage, more compact storage What do we migrate from an archive? Everything? Reduced Data only? Data Products? What do we do with the old media? Paper and glass are more stable than digital media! Magnetic tapes may be more stable than optical disks! Who pays? Is migration maintenance? Is a new, more useful archive being created?

4. Maintenance Issues NSF 2004-03-23 What are the costs? Space, Staffing Equipment maintenance and repair Backup protection? What is the safest way to back up a multi-Terabyte archive? Cloning to other sites improves access as well as providing backup How do we maintain old media? Old media may be more stable over time than new media! Can we maintain older, less compact data?

5. Some US Astronomical Archives NSF 2004-03-23 Online (All NASA Funded) Hubble Space Telescope 17.6 Terabytes Two-Micron All-Sky Survey 5 Terabytes Sloane Digital Sky Survey 1 Terabyte online (50 more offline) Palomar-QUEST 6 Terabytes (1/month since 9/2003) Off-Line NOAO Save-the-Bits 44.4 Terabytes (7.6 Terabytes/year) HPSSP (Harvard Plate Stack Scanning Project) 200 Terabytes Future LSST (Large Scale Synoptic Array) 7 Terabytes per night!

6. Growing Astronomical Catalogs ● 1989HST Guide Star Catalog 25,541,952 sources ● 1996USNO-A1.0 Catalog 488,006,860 sources ● 1998USNO-A2.0 Catalog 526,280,881 sources ● 2001GSC II Catalog (2.2.01) 998,402,801 sources ● 2002USNO-B1.0 Catalog 1,036,366,767 sources NSF 2004-03-23

7. Virtual Observatory Portals NSF 2004-03-23 US: ADS (links from publications) Goddard (Skyview vizualization) JHU,NCSA,Caltech (VO Registry modelling) IPAC (IRSA, etc.) SAO WCSTools (desktop catalog access) England: The Grid France: CDS (Aladin/Vizier/Simbad)

8. International Virtual Observatory Alliance (IVOA) NSF 2004-03-23 Registries: Searchable databases containing descriptions of data available in the Virtual Observatory Data Model: Standards for data format and content VOTable: XML transfer format for metadata UCD: Uniform Column Descriptors (so everyone doesn't make up their own names for the same things) Data Access Layer (DAL): User interface Protocols: Open interfaces to large archives ease multi-level links (NSF funds US participation in IVOA)

9. IVOA Registries NSF 2004-03-23 Full Searchable Registry Full Searchable Registry Replicate Local Publisher (harvestable registry) Local Publisher (harvestable registry) Local Searchable Registry Client Data ReplicateReplicate DAL

10. ● 500,000 glass plates covering the entire sky from 1885-1989 ● Basis for fundamental discoveries in astronomy, such as using Cepheid variable stars as cosmic yardsticks ● A legacy of long-term commitment to astronomical photography and research ● Astronomy will not have an equivalent time frame from digital observations until 2080. Migrating Harvard's Astronomical Plate Collection from Paper and Glass to Bits CfA/PSSG, 2002-11-18

11. International Astronomical Union Resolution B3, 2000 Safeguarding the Information in Photographic Observations The International Astronomical Union, Recognising that unless urgent action is taken, this unique historical record of astronomical phenomena will be lost to future generations of astronomers, Recommends the transfer of the historic observations onto modern media by digital techniques, which will provide worldwide access to the data so as to benefit astronomical research in a way that is well matched to the tools of the researcher in the future. CfA/PSSG, 2002-11-18

12. Step 0: List what is in the archive (on the web) NSF 2004-0323

13. Typical large glass plate NSF 2004-03-23

14. First: Digitize Metadata From hand-written cards and logbooks NSF 2004-03-23

15. Digital access to plate metadata (interactive web page) NSF 2004-02-23

16. Results of metadata search NSF 2004-02-23

17. Next: Digital access to image data Move the plates out of the 20th century NSF 2004-03-23

18. Proposed access to digital images User Stack Catalog search FITS or Tiff Image Archive (100 Terabyte) FITS Header Archive (WCS information) FITS extractor Object or coordinates and time Plate names and object (x,y) FITS images of plate portions NSF 2004-03-23