1. NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 PACS page 1 Workshop Slides NHSC-SPIRE Team NHSC Data Processing Workshop Pasadena 28-30 June 2010

2. PACS page 2 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 General Product Structure Products contain: –Metadata, –Datasets –Processing history Types of datasets are: –Array dataset –Table dataset –Composite dataset –Spectrum1d –Spectrum2d Generic Product Types are: –SimpleImage –SimpleCube –SpectralSimpleCube –Context Metadata Dataset1 (table dataset) Dataset2 (image dataset) Product Products are containers for datasets that can be stored within the HCSS system. A product can be exported to other software using a FITS representation.

3. PACS page 3 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Contexts are products that point to other products. Contexts provide structure to the data. Metadata Dataset1 (table dataset) Dataset2 (image dataset) Data Product Metadata Dataset1 (table dataset) Dataset2 (image dataset) Data Product Metadata Dataset1 (table dataset) Dataset2 (image dataset) Data Product Metadata Dataset1 (table dataset) Dataset2 (image dataset) Data Product Metadata Dataset1 Context (prod.) Metadata Dataset1 Context (prod.) Metadata Dataset1 Context (prod.) Contexts Metadata Dataset1 (table dataset) Dataset2 (image dataset) Data Product

4. PACS page 4 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 The Product Access Layer PAL Create Storage Save Product Delete ProductQuery Storage Local Store http Store HSA Store Internet DB Store Storage Load Product Delete Storage

5. PACS page 5 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Herschel Archive Access PAL Herschel Science Archive Internet HSA Store Local Store Internet FTP server Local harddisk Export structure Export package HIPE Import View Herschel Science Centre My Computer unzip untar FTP access procedure HSA Store access

6. PACS page 6 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 SPIRE in the Herschel Focal Plane

7. PACS page 7 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Photometer AOT 7 point jiggle (point source) 7-point jiggle for point source photometry, to compensate pointing error and under- sampling. Chopping and nodding at each jiggle position. 126” chop + nod single step ~ 6” small mapscan (large) map Scan map at speeds of 30 and 60 ”/sec is most efficient mode for large-area surveys. Parameters are optimized for full spatial sampling and uniform distribution of integration time. Cross scan capability (84.8 o ) Overlap region 348” 42.4 o Scan Single cross scan at 84.8 o replaces Jiggle map. Scan map at speeds of 30 and 60 ”/sec. Full spatial sampling in center of scans. 42.4 o Scan Fully sampled region ~4’

8. PACS page 8 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Parallel Mode SPIRE and PACS SPIRE Geometry Considerable redundancy for SPIRE due to smaller scan distance needed by PACS arrays. Sample rate lowered from 16 to 10Hz. Overlap region 155” orth. 168” nom. 42.4 o Scan SPIRE PACS Geometry Even field coverage for PACS. Additional frame averaging needed to keep data rate down. Blue array frames are averaged by additional factor two compared to PACS only mode. Overlap region 155” orth. 168” nom. 42.4 o Scan PACS Scan maps at speeds of 20 and 60”/sec with PACS and SPIRE active in parallel are useful for large-area surveys. The distance between PACS and SPIRE apertures is 21 arcmin. Two almost orthogonal (84.8 o ) directions for cross scanning are available.

9. PACS page 9 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Far Field Beam Hot Spot “I” verified with Jupiter in all 3 SPIRE wavelengths and by PACS with 2x2deg map in parallel mode. PACS found attenuation 5*10 -4 Good and Bad news: Measurement consistent with telescope model Low probability for Moon to enter, but observers should be wary of other bright sources close to telescope hot spots. A B D E C F H G I J Simulated sky map over ~66degx66deg around boresight (log scale) of hot spots/stray paths Amplitude: ~0.74 (+/-10%) Extent, as fitted FWHM of the long dimension distribution: ~34.9arcmin (+/-10%) Width: ~6.5arcmin (+/-10%) PSW OD136 Ghost as expected

10. PACS page 10 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Parallel Mode SPIRE and PACS SPIRE PACS SPIREPACS ~21’

11. PACS page 11 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 SPIRE Observation Sequence A SPIRE observation is divided into Building Blocks: Init Move BSM Move BSM FTS Scans EndCalib. FTS scans PCAL flash Initialization of instrument Calibrator flashes to track detector responsivity Reconfigure instrument, SMEC back to HOME etc. Science data Each basic instrument operation is contained in a separate building block (with a unique BBID#). Pipeline processing is per building block (to Level 1).

12. PACS page 12 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 SPIRE Pipelines – Common Part Level-0 Product (Raw Re-formatted digital telemetry) RAW Telemetry Calibration Files: Conversion to Engineering Units (ADU → Volts, Kelvins,...) Level-0.5 Product [Detector timeline in V, house keeping (HK) time line, BSM timeline, etc.] Photometer or Spectrometer Pipeline – AOT dependent

13. PACS page 13 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 SPIRE Pipelines – Photometer Level-0.5 Product Calibration 1 (AOT independent)

14. PACS page 14 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Scan Map Pipeline Flow Chart Baseline Removal

15. PACS page 15 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Simple Deglitching: Basic Idea time signal Determine baseline through tuned lowpass filtering Find glitches outside of  range from zero of filtered signal. Subtract baseline Determine standard deviation   Identified glitches Celestial scan map source Glitch event Flagged tail if large glitch

16. PACS page 16 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Non-linearity Correction & Flux Conversion The conversion from voltage to Jy is not linear (otherwise the inverse response is a constant). The inverse response function is approximated by: The flux conversion is: The K parameters and reference voltage V 0 are calibrated empirically.

17. PACS page 17 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Temperature Drift Correction (V) PLW, ~ 4 hours δT ~1mK δS ~20Jy dark field  ≤1% for δT ≤1mK Use thermistors (2 for PSW, 1 for PMW, 2 for PLW) to monitor detector array temperature drifts. Detector signal drifts due to the temperature drifts are then estimated using thermistor signal timelines. Thermistor signals are smoothed (Δt =5 sec) to filter out the (high frequency) measurement noise. The formula: Parameters A and B (both are constant for a given detector) and reference voltage V T0 are taken from a calibration file. themistor voltage (smoothed) reference voltage (in Jy)

18. PACS page 18 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Mapmaking Using Time Ordered Data (TOD) Basic Equation: d(t) = A(t,x) u(x) + n(t) TOD: obs data in a timeline (of a single detector, or of an array) signal map noise pointing matrix Mapmaking --- the Least Squares (MML) Solution of u(x) : ũ = (A T N -1 A) -1 A T N -1 d It has the “least” value of χ 2 = (d-Aũ) T N -1 (d-Aũ) Inverse noise covariance matrix of TOD (time domain) Noise covariance matrix of map (space domain)

19. PACS page 19 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Inverse noise matrix for a TOD of uncorrelated samplings (white noise only): Naïve Mapper: Inverse-noise weighted mean: SPIRE Naïve Mapper Assuming σ j =σ=constant: General Solution: ũ = (A T N -1 A) -1 A T N -1 d Adding all hits Number of hits Special Case 1 --- Naïve Mapper

20. PACS page 20 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Special Case 2 --- MADmap sub-matrix for a single detector correlation length N = a solution for detectors having uncorrelated 1/f noise Noise characteristics: Sub-matrix of each detector is not diagonal. Covariance items between diff. detectors are all 0. (No correlation between detectors.)

21. PACS page 21 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Assumptions in MADmap For each detector, the noise is stationary and circulant (the covariance between any two samplings depends only on the time difference |Δt|). Hence, in the noise matrix, the sub-matrix of a given detector (j) is Toeplitz: (note N j is also symmetric: it is a covariance matrix.) The covariance goes to 0 when |Δt| ≥ correlation length λ. (When λ=1, MADmap=naïve map.)

22. PACS page 22 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Major steps in MADmap For a given detector j, derive N -1 j from noise power spectra PS(ω): General Solution: ũ = (A T N -1 A) -1 A T N -1 d Calculate N -1 d for each detector: N -1 d = FFT -1 {FFT(d)/PS} --- when low freq (1/f) noise is strong, this discounts the low freq. variation of d (“whitening”). Projection to spatial pixels: A T N -1 d (A T does the TOD  space projection). Using the Conjugate Gradient Method (iterations~100) to solve the equation: This is to avoid the time consuming matrix inversion A T N -1 A  (A T N -1 A) -1. Drawback: no estimate for the map noise matrix N pp =(A T N -1 A) -1.

23. PACS page 23 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Special Case 3 --- SANEPIC N -1 = a solution for detectors having uncorrelated 1/f noise and a common mode (i.e. correlated) 1/f noise. Noise characteristics: Sub-matrix of each detector is not diagonal. Noises of different detectors are correlated. (Off-diagonal sub-matrixes are non-zero.)

24. PACS page 24 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Noise with a single common mode (the case for SANEPIC): uncorrelated noise of detector i at time t common mode scaling factor for detector i (use Principle Component Analysis algorithm for the noise decomposition.) Noise power spectra matrix (m = number of detectors): General Solution: ũ = (A T N -1 A) -1 A T N -1 d Inverse noise matrix: (The rest steps in SANEPIC is similar to those in MADmap.), PS of uncorr. noise PS of common mode

25. PACS page 25 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Effect of temp. drift correction on map-making white noise only with temp drift, no corr. after temp drift correction 1 2 3 3 - 1 Residual (1/f noise) = map(temp. corr.) – map(white noise only) (simulated data; naïve maps)

26. PACS page 26 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Spectrometer AOT Image Sampling Spectral Resolution Pointing Mode High 0.04 cm -1 Medium 0.25 cm -1 Low 0.83 cm -1 High & Low 0.04/0.83 cm -1 any combination allowed Overlapping spectrometer arrays projected on the sky Single Pointing Raster Pointing Full Intermediate Sparse example 3 x 3 map Each color shows the unvignetted beams of the same array for all sampling positions (jiggles) at one raster position. unvignetted beam

27. PACS page 27 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 SMEC Scans & Spectral Map Coverage Low Medium High Optical Path Difference 0 Spectral Resolution Differences in SMEC scan pattern for different spectral resolutions. Sparse Intermediate Full SLW FWHM SSW FWHM 4 point jiggle16 point jiggleno jiggling BSM used to increase filling factor (no chopping)

28. PACS page 28 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Spectral Line Shape in FTS The finite optical path difference (OPD <= L) introduces a truncated interferogram: I(  ) = FT -1 [ B(  )  sin(π  /Δσ)/(π  /Δσ) ] The SINC function, sin(π  /Δσ)/(π  /Δσ), is the FT of the top-hat function in the interferogram domain. This line shape is characteristic of FTS. For unresolved lines:  A sin[π(σ-σ 0 )/Δσ]/(π(σ-σ 0 )/Δσ);  Flux = A Δσ;  FWHM = 1.207 Δσ. Where Δσ is the spectrometer resolution element in wavenumber. Wavenumber σ (cm -1 ) Flux density B(σ) (Jy)

29. PACS page 29 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Extended Norton-Beer Functions Instrument Line Shapes of Extended Norton-Beer Functions: NB(1.1) NB(1.2) … NB(2.0) SINC function (Naylor & Tahic 2007)

30. PACS page 30 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Extended Norton-Beer Functions Effects in the spectral domain: No apodization NB 1.2 (FWHM 20% wider) NB 1.6 (FWHM 60% wider) NB 2.0 (FWHM 100% wider) (P. Davis 2008)

31. PACS page 31 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Level 1 products: unmodified interfergrams average spectrum (apodized) average spectrum (unapodized) Interferograms (stored in Level 1) Level 0.5 products: detector time lines scan mirror time line house keeping time lines Spectrometer Pipeline Data Flow SPIRE Common Pipeline 1. Modify Detector Timelines 2. Create Interferogram 3. Modify Interferogram 4. Fourier Transform 5. Modify Spectra ( V → Jy) 6. Spectral Mapping Level 2 product: Spectral Cubes (NOT yet available)

32. PACS page 32 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 1 st Level Deglitching Remove Electrical Crosstalk Clipping Correction Time-domain Phase Correction Bath Temperature Correction Cross talk matrix V(t) Step 1: Modify Timelines V(t) Level 0.5 Timelines Modified Level 0.5 Timelines Non-linearity Correction V(t) Bolometer Nonlinearity Table Bath temp. corr. product Time constants

33. PACS page 33 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Create Interferograms Once time domain processing is complete, the detector signals and SMEC positions can be merged to create interferograms. The created “unmodified” interferograms are also stored in Level 1 in case users want to do their own interferogram-to- spectrum process. V(t) Step 2: Create Interferograms Level 0.5 Timelines V(t) Unmodified Interferograms V(x) (Stored in Level 1) SMEC Positions x(t') Pointing P(t'')

34. PACS page 34 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Telescope/SCAL/Beamsplitter Correction Baseline Removal 2 nd Level Deglitching Phase Correction (Default apodization) V(x) Step 3: Modify Interferograms V(x) (Level 1) Interferograms Modified Interferogram Products (both unapodized and apodized) Reference background interferogram Nonlinear phase calibration table Norton Beer Order-1.5 function

35. PACS page 35 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Fourier Transform Apply the Fourier Transform to each interferogram to create a set of spectra for each spectrometer detector. Step 4: Transform Interferograms Spectra V(σ) Modified Interferograms V(x)

36. PACS page 36 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Spectral Averaging Flux Conversion: V->Jy Remove Optical Crosstalk V(σ) Step 5: Modify Spectra I(σ) Spectra Level 1 Spectrum Products Point-source case volt-to-Jy factors (both unapodized and apodized) Detector optical crosstalk matrix

37. PACS page 37 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010 Spatial Regridding Merge the Level 1 spectra from each FTS Scan Building Block into a single spectral cube. V(t) Step 6: Spatial Regridding Level 1 Spectra I(σ) Level 2 Spectral Cube I(σ) (Not yet available) Level 1 Spectra I(σ) Level 1 Spectra I(σ)

38. PACS page 38 NHSC Data Processing Workshop – Pasadena 25 8h 30 th June 2010