1. PACS Photometer Standard Pipeline Level 0 to Level 1 processing From raw to calibrated data cubes B. Altieri (ESA/HSC), N.Billot (IPAC/NHSC) ESAC March 2011

2. B. Altieri | ESAC | March 2011 | Slide 2 Introduction Level 0: Raw cubes Level 0: Raw cubes Level 1: Data cubes calibrated in Jy/pixel With associated pointing information and masks Level 1: Data cubes calibrated in Jy/pixel With associated pointing information and masks Level 1 products from the archive can in principle be used to build maps However reprocessing the raw data to Level 1 might be necessary to exploit updated calibration files or pipeline scripts, or to interactively improve the deglitching process to suit the need of your data set. Nevertheless the Level 0 to 1 standard processing is already close to optimum, and one should keep to a minimum modifications to the pipeline. Level 1 products from the archive can in principle be used to build maps However reprocessing the raw data to Level 1 might be necessary to exploit updated calibration files or pipeline scripts, or to interactively improve the deglitching process to suit the need of your data set. Nevertheless the Level 0 to 1 standard processing is already close to optimum, and one should keep to a minimum modifications to the pipeline.

3. B. Altieri | ESAC | March 2011 | Slide 3 Level 0 to 1 : overview steps 1.load the script 2.Review the observation context 3.Extract the frames and relevant information 4.From level 0 to 0.5 5.From level 0.5 to 1

4. B. Altieri | ESAC | March 2011 | Slide 4 Step 1 : What script to use ? ipipe scripts: ipipe (interative pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observations (need for concatenating scan/cross ‐ scan observations, or modify deglitching parameters). ipipe scripts: ipipe (interative pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observations (need for concatenating scan/cross ‐ scan observations, or modify deglitching parameters).

5. B. Altieri | ESAC | March 2011 | Slide 5 Observation context Observation Context: HIPE variable holding an entire Herschel observations. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary information such as the telescope pointing, and all the calibration files necessary to reprocess the data. Observation Context: HIPE variable holding an entire Herschel observations. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary information such as the telescope pointing, and all the calibration files necessary to reprocess the data. Definition of Observation Content: Herschel Data Analysis Guide Section 1.2.3 Definition of Observation Content: Herschel Data Analysis Guide Section 1.2.3 The Observation Context can be loaded in HIPE from the Herschel Science Archive (HSA)

6. B. Altieri | ESAC | March 2011 | Slide 6 Step 2: Load observation context The Observation Context can be loaded in HIPE from the Herschel Science Archive (HSA) HIPE> obsid = 1342189191 # public observation of Delta Draconis HIPE> obs = getObservation(obsid, useHsa=True) HIPE> obsid = 1342189191 # public observation of Delta Draconis HIPE> obs = getObservation(obsid, useHsa=True) … or from a local pool on your hard drive disk assuming it was previously saved to a local store … or from a local pool on your hard drive disk assuming it was previously saved to a local store HIPE> obs = getObservation(obsid)

7. B. Altieri | ESAC | March 2011 | Slide 7 Inspect the Observation Context HIPE> print obs metadata Housekeeping Pointing etc.. Calibration files are Included in obsContext Different levels of data processing generated by the pipeline and served by the archive

8. B. Altieri | ESAC | March 2011 | Slide 8 Inspect the Observation Context Double click obs in the variables window to open the Observation Viewer in the Editor window (creation of a new tab)

9. B. Altieri | ESAC | March 2011 | Slide 9 Inspect the Observation Context Click Meta Data and scroll to explore some information about your observation Click Meta Data and scroll to explore some information about your observation

10. B. Altieri | ESAC | March 2011 | Slide 10 Inspect the Observation Context Single Click obslevel2/ HPPPMAPB/0/image to view the image of the level2 product MAP Blue Single Click obslevel2/ HPPPMAPB/0/image to view the image of the level2 product MAP Blue

11. B. Altieri | ESAC | March 2011 | Slide 11 Inspect the Observation Context Single Click obs/level2/ HPPPMAPB/0/image to open the image with Image Viewer (creation of a new tab) Single Click obs/level2/ HPPPMAPB/0/image to open the image with Image Viewer (creation of a new tab) Click the zoom button to fit the image to the screen

12. B. Altieri | ESAC | March 2011 | Slide 12 Step 3: Extract the frames and related information from the Observation Context in HIPE Level 0: raw data cube HIPE> frames = obs.level0.refs["HPPAVGB"].product.refs[0].product HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecraft after on ‐ board averaging HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecraft after on ‐ board averaging HIPE> frames = obs.level0.refs["HPPAVGR"].product.refs[0].product Synthax for Blue/Red array

13. B. Altieri | ESAC | March 2011 | Slide 13 Inspect the frames Actual datasets stored in the frames: Signal, Status and Mask HIPE> print frames.class herschel.pacs.signal.Frames HIPE> print frames.class herschel.pacs.signal.Frames metadata

14. B. Altieri | ESAC | March 2011 | Slide 14 Inspect the frames Access individual parameter values, eg. number of scan legs HIPE> print frames.meta HIPE> print frames.meta["mapScanNumLegs"].value

15. B. Altieri | ESAC | March 2011 | Slide 15 Inspect frames signal frames.signal contains the raw data cube The cube dimensions depends on the channel and the dura>on of the observation frames.signal contains the raw data cube The cube dimensions depends on the channel and the dura>on of the observation HIPE> print frames.signal.class herschel.ia.numeric.Double3d HIPE> print frames.signal.dimensions array([16, 32, 2890], int) HIPE> print frames.dimensions[2] 2890 HIPE> print frames.signal.class herschel.ia.numeric.Double3d HIPE> print frames.signal.dimensions array([16, 32, 2890], int) HIPE> print frames.dimensions[2] 2890

16. B. Altieri | ESAC | March 2011 | Slide 16 Inspect frames signal Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ 30000 ADU Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ 30000 ADU Readout sequence 10 Hz − PACS Prime 5 Hz – Pacs PMode Readout sequence 10 Hz − PACS Prime 5 Hz – Pacs PMode 64 (32) pixels 32 (16) pixels

17. B. Altieri | ESAC | March 2011 | Slide 17 Inspect frames signal Indicates that the third dimension is the readout sequence Complete documentation on Data Display: Herschel Data Analysis Guide Chapter 2 Complete documentation on Data Display: Herschel Data Analysis Guide Chapter 2 HIPE> Display(frames.signal, depthAxis=2) Scroll to explore the data cube Scroll to explore the data cube

18. B. Altieri | ESAC | March 2011 | Slide 18 Inspect frames signal Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Calibration Block: prior to any observations, 2 internal sources are observed with the internal chopper for calibration purposes Calibration Block: prior to any observations, 2 internal sources are observed with the internal chopper for calibration purposes Science block

19. B. Altieri | ESAC | March 2011 | Slide 19 Inspect frames signal Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Zoom onto the Calibration Block Chopping pattern on internal sources 1 & 2 Chopping pattern on internal sources 1 & 2

20. B. Altieri | ESAC | March 2011 | Slide 20 Inspect frames signal Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Zoom onto the Science Block The spike is likely a glitch

21. B. Altieri | ESAC | March 2011 | Slide 21 Inspect frames signal Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Zoom onto the spike The spike is asymmetric and affects only 2 readouts: It is a glitch The spike is asymmetric and affects only 2 readouts: It is a glitch

22. B. Altieri | ESAC | March 2011 | Slide 22 Inspect frames status & mask HIPE> print frames.status HIPE> print frames.mask

23. B. Altieri | ESAC | March 2011 | Slide 23 Extract the auxiliary data from the ObservationContext HIPE> pp = obs.auxiliary.pointing HIPE> photHK = obs.level0.refs["HPPHK"].product.refs[0].product["HPPHKS"] HIPE> oep = obs.auxiliary.orbitEphemeris HIPE> pp = obs.auxiliary.pointing HIPE> photHK = obs.level0.refs["HPPHK"].product.refs[0].product["HPPHKS"] HIPE> oep = obs.auxiliary.orbitEphemeris pp: pointing product photHK: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain critical information required in subsequent processing modules. pp: pointing product photHK: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain critical information required in subsequent processing modules.

24. B. Altieri | ESAC | March 2011 | Slide 24 Extract the calibration files (calTree) from the ObservationContext HIPE> calTree = getCalTree(time = frames.startDate ) Some calibration files changed with time, e.g. the SIAM or pointing calibration file, so it is necessary to provide the date of observation for retrieving the appropriate calFiles Some calibration files changed with time, e.g. the SIAM or pointing calibration file, so it is necessary to provide the date of observation for retrieving the appropriate calFiles Complete documentation on the Calibration Framework: http://www.herschel.be/twiki/pub/Public/PacsDocumentation/ The_PACS_Calibration_Framework_ ‐ _issue_0.10.pdf Complete documentation on the Calibration Framework: http://www.herschel.be/twiki/pub/Public/PacsDocumentation/ The_PACS_Calibration_Framework_ ‐ _issue_0.10.pdf The Calibration Tree calTree contains all the files necessary to process your data

25. B. Altieri | ESAC | March 2011 | Slide 25 Inspect the calTree This is the flat-field in the blue filter Explore the Calibration Tree with the GUI Explore the Calibration Tree with the GUI

26. B. Altieri | ESAC | March 2011 | Slide 26 Inspect the calTree HIPE> print calTree Product specific calTree Flight Model 13 th version of the calTree

27. B. Altieri | ESAC | March 2011 | Slide 27 Inspect the calTree HIPE> print calTree.photometer Photometer specific calibration products

28. B. Altieri | ESAC | March 2011 | Slide 28 Inspect the calTree HIPE> trans = calTree.photometer.filterTransmission.blue["transmission"].data HIPE> wav = calTree.photometer.filterTransmission.blue["wavelength"].data HIPE> PlotXY(wav, trans, xtitle = "Wavelength [micron]", ytitle=“Transmission") HIPE> trans = calTree.photometer.filterTransmission.blue["transmission"].data HIPE> wav = calTree.photometer.filterTransmission.blue["wavelength"].data HIPE> PlotXY(wav, trans, xtitle = "Wavelength [micron]", ytitle=“Transmission") Zoom on a feature

29. B. Altieri | ESAC | March 2011 | Slide 29 Step 4: Pipeline from level 0 to 0.5 HIPE> frames = findBlocks(frames, calTree=calTree) HIPE> frames = detectCalibrationBlock(frames) HIPE> frames = removeCalBlocks(frames) HIPE> frames = findBlocks(frames, calTree=calTree) HIPE> frames = detectCalibrationBlock(frames) HIPE> frames = removeCalBlocks(frames) The chopping pattern from the calibration block is gone. The chopping pattern from the calibration block is gone. 1. Identify Blocks in the observation and remove the Calibration Blocks before after

30. B. Altieri | ESAC | March 2011 | Slide 30 Step 4: Pipeline from level 0 to 0.5 HIPE> frames = photFlagBadPixels(frames, calTree=calTree) Provide the photometer HouseKeeping to derive soft saturation limits that depend on the bolometer bias setting Provide the photometer HouseKeeping to derive soft saturation limits that depend on the bolometer bias setting 2. Flag Bad Pixels and populate frames.mask 3. Flag saturated pixels and populate frames.mask HIPE> frames = photFlagSaturation(frames, calTree=calTree, hkdata=photHK) CL and ADC saturation checking HIPE> frames = photFlagSaturation(frames, calTree=calTree, hkdata=photHK) CL and ADC saturation checking

31. B. Altieri | ESAC | March 2011 | Slide 31 Step 4: Pipeline from level 0 to 0.5 BADPIXELS and SATURATION masks were created in frames.mask BADPIXELS and SATURATION masks were created in frames.mask HIPE> print frames.mask

32. B. Altieri | ESAC | March 2011 | Slide 32 Step 4: Pipeline from level 0 to 0.5 HIPE> frames = photConvDigit2Volts(frames, calTree=calTree) 4. Convert the signal from digital units (ADU) into physical units (Volts) 5. Convert the chopper angle from digital units (ADU) into physical units (degrees) HIPE> frames = convertChopper2Angle(frames, calTree=calTree)

33. B. Altieri | ESAC | March 2011 | Slide 33 Step 5: Pipeline from level 0.5 to 1 6. Deglitch the data and create a glitch mask in frames.mask HIPE> frames = photMMTDeglitching(frames, incr_fact=2, mmt_mode='multiply’, scales=3, nsigma=5) TEMPORAL DEGLITCHING The standard pipeline uses a time ‐ domain deglitching algorithm: Multiresolution Median Transform of individual pixel timelines (Stark et al. 1998) TEMPORAL DEGLITCHING The standard pipeline uses a time ‐ domain deglitching algorithm: Multiresolution Median Transform of individual pixel timelines (Stark et al. 1998) The set of parameters given here works well with most observations Bright sources might however be flagged as glitches by this module The set of parameters given here works well with most observations Bright sources might however be flagged as glitches by this module  See talk about the deglitching disabling the GlitchMask on source  See dedicated deglitching talk  See talk about the deglitching disabling the GlitchMask on source  See dedicated deglitching talk

34. B. Altieri | ESAC | March 2011 | Slide 34 Step 5: Pipeline from level 0.5 to 1 Alternative : spatial deglitching # spatial deglitching with slim index from herschel.pacs.signal import MapIndex from herschel.pacs.spg.phot import MapIndexTask from herschel.pacs.spg.phot import IIndLevelDeglitchTask deg = IIndLevelDeglitchTask() s = Sigclip(30, 30) s.mode = Sigclip.MEDIAN s.behavior = Sigclip.CLIP s.outliers = Sigclip.BOTH_OUTLIE mi = mapindex(inframes = frames, slimindex = True) deg(mi, frames, algo = s, map = False, mask = True) # spatial deglitching with slim index from herschel.pacs.signal import MapIndex from herschel.pacs.spg.phot import MapIndexTask from herschel.pacs.spg.phot import IIndLevelDeglitchTask deg = IIndLevelDeglitchTask() s = Sigclip(30, 30) s.mode = Sigclip.MEDIAN s.behavior = Sigclip.CLIP s.outliers = Sigclip.BOTH_OUTLIE mi = mapindex(inframes = frames, slimindex = True) deg(mi, frames, algo = s, map = False, mask = True) Also called second order deglitching  See talk about the deglitching disabling the GlitchMask on source  See dedicated deglitching talk  See talk about the deglitching disabling the GlitchMask on source  See dedicated deglitching talk

35. B. Altieri | ESAC | March 2011 | Slide 35 Step 5: Pipeline from level 0.5 to 1 HIPE> frames = photAddInstantPointing(frames, pp, calTree=calTree, orbitEphem=oep) 7. Add the pointing information to the frames This module adds the spacecraft pointing to the frames.status as coordinates and additional pointing information Provide the telescope pointing product and the orbit ephemeris that were extracted from the ObservationContext Provide the telescope pointing product and the orbit ephemeris that were extracted from the ObservationContext HIPE> print frames.status photAddInstantPoinAng has created all these variables in frames.status

36. B. Altieri | ESAC | March 2011 | Slide 36 Step 5: Pipeline from level 0.5 to 1 HIPE> execfile("/path/to/the/script/plotscanpath.py") HIPE> plotscanpath(frames) HIPE> execfile("/path/to/the/script/plotscanpath.py") HIPE> plotscanpath(frames) plotscanpath.py exploits the pointing information added by the module photAddInstantPointing, namely RaArray and DecArray, to plot the trajectory of the center of the focal plane. plotscanpath.py exploits the pointing information added by the module photAddInstantPointing, namely RaArray and DecArray, to plot the trajectory of the center of the focal plane. This command allows to compile a function at the command line or within a script

37. B. Altieri | ESAC | March 2011 | Slide 37 Step 5: Pipeline from level 0.5 to 1 HIPE> frames = photAssignRaDec(frames, calTree=calTree) 8. Add the pointing information to the frames This modules assigns coordinates to individual pixels using distortion information from the calTree. This modules assigns coordinates to individual pixels using distortion information from the calTree. Note : This task is necessary only if one want to proceed with MADmap, Scanamorphos or certain type of processing to level 2

38. B. Altieri | ESAC | March 2011 | Slide 38 Step 5: Pipeline from level 0.5 to 1 HIPE> frames = photRespFlatfieldCorrection(frames, calTree = calTree) 9. Apply the flat ‐ field correc5on and convert the signal from Volt/pixel into Jy/pixel 9. Apply the flat ‐ field correc5on and convert the signal from Volt/pixel into Jy/pixel You have reach Level 1 The frames are now calibrated and ready for the map ‐ making process You have reach Level 1 The frames are now calibrated and ready for the map ‐ making process

39. B. Altieri | ESAC | March 2011 | Slide 39 Step 5: Pipeline from level 0.5 to 1 HIPE> save("/my/directory/my_frames.save","frames") 10. Save the frames before further processing The frames are saved in the “save” format of HIPE The saved file can only be read back into HIPE The frames are saved in the “save” format of HIPE The saved file can only be read back into HIPE HIPE> restore("/my/directory/my_frames.save”)

40. B. Altieri | ESAC | March 2011 | Slide 40 Step 5: Pipeline from level 0.5 to 1 HIPE> FitsArchive().save("/my/directory/my_frames.fits", frames) 10. Save the frames before further processing The frames are saved in standard fits format The saved file can be read back into HIPE or IDL The frames are saved in standard fits format The saved file can be read back into HIPE or IDL HIPE> frames = FitsArchive().load("/my/directory/my_frames.fits”)

41. B. Altieri | ESAC | March 2011 | Slide 41 Pipeline summary to level 1 frames = findBlocks(frames, calTree=calTree) frames = detectCalibrationBlock(frames) frames = removeCalBlocks(frames) frames = photFlagBadPixels(frames, calTree=calTree) frames = photFlagSaturation(frames, calTree=calTree, hkdata=photHK) frames = photConvDigit2Volts(frames, calTree=calTree) frames = convertChopper2Angle(frames, calTree=calTree) frames = photRespFlatfieldCorrection(frames, calTree = calTree) frames = photMMTDeglitching(frames, incr_fact=2, mmt_mode='multiply’,\ scales=3, nsigma=5) frames = photAddInstantPointing(frames,pp,orbitEphem = oep) #frames = photAssignRaDec(frames, calTree=calTree) save(savfile,"frames") frames = findBlocks(frames, calTree=calTree) frames = detectCalibrationBlock(frames) frames = removeCalBlocks(frames) frames = photFlagBadPixels(frames, calTree=calTree) frames = photFlagSaturation(frames, calTree=calTree, hkdata=photHK) frames = photConvDigit2Volts(frames, calTree=calTree) frames = convertChopper2Angle(frames, calTree=calTree) frames = photRespFlatfieldCorrection(frames, calTree = calTree) frames = photMMTDeglitching(frames, incr_fact=2, mmt_mode='multiply’,\ scales=3, nsigma=5) frames = photAddInstantPointing(frames,pp,orbitEphem = oep) #frames = photAssignRaDec(frames, calTree=calTree) save(savfile,"frames") Only tunable parameters: deglitching

42. B. Altieri | ESAC | March 2011 | Slide 42 Documentation  Most relevant documentation for this tutorial  PACS Data Reduction Guide  PACS User’s Reference Manual  HCSS User’s Reference Manual  General documentation necessary to manipulate the data:  HIPE Owner’s Guide  Herschel Data Analysis Guide  Scripting and Data Mining