1. Observing with Modern Observatories (the data flow)
NEON School 2004
Observing with Modern Observatories
(the data flow)

2. Lecture Layout Proposal Submission Phase I Phase II Observation
Observing modes
The proposal submission
The OPC selection
Long term scheduling
“Web Letters”
Phase II
P2PP, OBs, etc.
Observation
Short-term scheduling
Data acquisition

3. Lecture Layout Archival Connecting meta data to pixels
Data shipment (or replication) to Garching
Quality Control
Preparation of master calibration files
Preparation of reduced products for PIs of service mode programs
Archive use
Data distribution to PIs
Data distribution to archive users

4. The data flow: Another View

5. The Data Flow
User Support Group
The ESO Data Flow Model and its Supporting Organizational Structure
Paranal/La Silla Observatories
Phase II
proposal submission
observing blocks
medium term schedule
Operations Technical Support
Data Archive &
Supporting Databases
storage and re-distribution
PI packages distribution
Phase I
proposal submission
program selection
time allocation
Observation
short term schedule
archival of data
Quality Control
Master Cal files prod.
Instrument trending
PI products preparation
Data Flow Operations Group

6. Phase I: Proposal Submission
Provide information:
Identification of PI/CoIs
Facilities to be used
Split program in “runs” per facility/group of targets
Observing preferences
Scientific Justification
OPC selection:
Peer review of proposal
Below or above threshold?
Long term scheduling
“Web Letters”
Did I get time or not?
Message sent to all PI with link to explanations
program
run

7. Phase I: Observing Modes
Two observing modes:
“Visitor” mode
Keep contact with telescope and instrumentation (observing should not become a black box/magic activity)
Need to use a specific observing mode
Need to be able to assess quality and react immediately
“Service” mode
Program is to use normal, supported instrument modes
Program requires multiple visits of the targets
Program has lots of targets spread in r.a. space

8. Phase II: Send Details Provide detailed information:
About each RUN in the PROGRAM
About each target in the proposal
About instrument settings to be used
Define observing sequence
Define exposure times, filters, constraints (e.g., on the seeing)
==> Observing block (OBs)
program
run OB

9. Observation Steps in the observation sequence:
OBs sent to La Silla/Paranal
Short term scheduling to match required target visibility, moon brightness, seeing, … with prevailing conditions
OB execution
==> exposures
Two types of exposures: calibration and science
program
run OB
exposure

10. Archival
Connecting meta-data to pixels
Collecting telescope, environment, instrument and user provided details
Building a header with this information according to a pre-defined dictionary.
Assembling header and pixels into a FITS file.
Sending file to the archive system
Storing the file onto the archive medium (DVD, Mag disk)
Decomposing header and storing info into database
Replicating database to Garching
Sending archive media to Garching

11. Data Quality Control
Building calibration database
Assembling master flats, darks, biases
Sending those to the archive for future
Preparing calibrated science products
For each run, build a structure with raw, calibration and calibrated data for the PIs of service mode runs.
Send a “pack & send” signal to the archive

12. The Archive
Collects data:
Information about PIs, programs, runs, OBs, exposures
Pixel data (raw science and calibration files)
Distributes data:
Proprietary data to PIs
Public data to anyone

13. The Data Flow
User Support Group
The ESO Data Flow Model and its Supporting Organizational Structure
Paranal/La Silla Observatories
Phase II
proposal submission
observing blocks
medium term schedule
Operations Technical Support
Data Archive &
Supporting Databases
storage and re-distribution
PI packages distribution
Phase I
proposal submission
program selection
time allocation
Observation
short term schedule
archival of data
Quality Control
Master Cal files prod.
Instrument trending
PI products preparation
Data Flow Operations Group

14. The Archive: Content
Contains ESO and HST raw data
Started operations in 1991
Growth never stopped
Need to constantly adapt to new storage technology to keep costs low
Need to frequently adapt to current software user interfaces to provide users with better search tools (more and more data)
Need to look for “advanced” products to empower users with doing science

15. The Archive: Content

16. The Archive: Output
Data retrieval model is that of the Web Shop:
Browse catalog
Mark articles of interests
Go to “cashier”
Select delivery options
Confirm submission

17. Request Submission: 1) Browse

18. Archive: Selection of Frames of interest

19. The Archive: Selecting Request Options

20. The Archive: Output

21. Technical vs. Science Archive
Two types of Archives
Technical vs. Science Archive
Current system more tuned to serve the operational needs of the Observatory (data collection and distribution to internal/external users)
Science users still mostly left out (emphasis on raw data)
Discrepancy understood early by the VO “founding fathers”
Need to provide access to well-described, instrumental signature-free data
Need to provide hassle-free access to different archives without affecting their operation critical mission
Science users will get better products and access through the VO

22. Archives for the people!
The three layers
Separate the mission critical support role from the delivery of products
Leave the science service provision to the VO
Provide interfaces to the top layers (e.g., image and spectra access protocols, high-level data processing grids)
Virtual Observatory
VO tools and facilities
Instrumental signature-free data
“Real” Observatory
Data In/Out

23. Technical vs. Science Archive
Two type of archives
Technical vs. Science Archive
Archive downloads continue to increase, but usage increasingly technical:
Multiple data deliveries to PIs/CoIs
Easy retrieval of own data
Streamlining of internal data channels
Science usage still limited:
ESO Archive still has too few instrumental signature-free products to offer
Astronomers will produce archive-based science if they are paid and/or supported! (e.g., Hubble’s archive proposals, ASTROVIRTEL)

24. On-going efforts in the ESO archive
News from the “technical” archive:
Efforts on-going to put more reduced products available
Better web-based search screens under development
Data available world-wide probably in 2005 (ESO Council to decide)
News from the “VO” ESO:
On-going efforts to provide a multi-archive query system (Querator)
Image and spectral VO access implemented as soon as standards approved
Extra ESO resources arriving soon to help with all of the above