1. - page 1 NHSC – DP workshop – Feb. 2011 – N. Billot PACS PACS Photometer Standard Pipeline Level 0 to Level 1 processing: From raw to calibrated data cubes

2. - page 2 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Introduction This presentation is largely inspired by the online tutorial PACS-201. Level 1 products from the archive can in principle be used to build maps. However reprocessing the raw data to Level 1 might be necessary to exploit updated calibration files or pipeline scripts, or to interactively improve the deglitching process to suit the need of your data set. Nevertheless the Level 0 to 1 standard processing is already close to optimum, and one should keep to a minimum modifications to the pipeline. Level 1 products from the archive can in principle be used to build maps. However reprocessing the raw data to Level 1 might be necessary to exploit updated calibration files or pipeline scripts, or to interactively improve the deglitching process to suit the need of your data set. Nevertheless the Level 0 to 1 standard processing is already close to optimum, and one should keep to a minimum modifications to the pipeline. Level 0: Raw data cubes Level 1: Data cubes calibrated in Jy/pixel with associated pointing information and masks

3. - page 3 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Level 0 to 1: Overview Step 1 Load the script Step 2 Retrieve the ObservationContext Step 3 Extract the frames and relevant information Step 4 From Level 0 to 0.5 Step 5 From Level 0.5 to 1

4. - page 4 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Step 1 Load the relevant data reduction script

5. - page 5 NHSC – DP workshop – Feb. 2011 – N. Billot PACS ipipe scripts: ipipe (interactive pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observations (need for concatenating scan/cross-scan observations, or modify deglitching parameters). Custom scripts: The NHSC provides examples of customized scripts based on ipipe scripts. They can be downloaded from our webpage https://nhscsci.ipac.caltech.edu/sc/index.php/Pacs/DataProcessing We encourage you to write your own custom scripts starting from the ipipe scripts and incorporating processing steps specific to your own data set. Load your script following tutorials PACS-102 ipipe scripts: ipipe (interactive pipeline) scripts are part of HIPE. They are delivered and updated with new material with each release of HIPE. They are to be used as templates to reduce your data. You might want to SAVE AS and modify these scripts to suit the specifics of your own set of observations (need for concatenating scan/cross-scan observations, or modify deglitching parameters). Custom scripts: The NHSC provides examples of customized scripts based on ipipe scripts. They can be downloaded from our webpage https://nhscsci.ipac.caltech.edu/sc/index.php/Pacs/DataProcessing We encourage you to write your own custom scripts starting from the ipipe scripts and incorporating processing steps specific to your own data set. Load your script following tutorials PACS-102 What script should I use?

6. - page 6 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Step 2 Retrieve the ObservationContext

7. - page 7 NHSC – DP workshop – Feb. 2011 – N. Billot PACS ObservationContext: HIPE variable holding entire Herschel observations. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary information such as the telescope pointing, and all the calibration files necessary to reprocess the data. ObservationContext: HIPE variable holding entire Herschel observations. It contains the raw data cube as well as the data processed with the standard pipeline up to Level 0.5, 1 and 2. It also contains auxiliary information such as the telescope pointing, and all the calibration files necessary to reprocess the data. Definition of Observation Content: Herschel Data Analysis Guide Section 1.2.3 Definition of Observation Content: Herschel Data Analysis Guide Section 1.2.3

8. - page 8 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> obsid = 1342189191 # public observation of Delta Draconis HIPE> obs = getObservation(obsid, useHsa = True) The ObservationContext can be loaded in HIPE from the Herschel Science Archive (HSA) HIPE> obs = getObservation(obsid, poolLocation = “/where/you/saved/it/”) … or from a local pool on your hard drive disk assuming it was previously saved to a local store (see the usage of the getPACSdata.py script in tutorial PACS-103) … or from a local pool on your hard drive disk assuming it was previously saved to a local store (see the usage of the getPACSdata.py script in tutorial PACS-103)

9. - page 9 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print obs metadata Different levels of data processing generated by the pipeline and served by the archive Calibration files are included in the ObservationContext Pointing, House Keeping, etc. Inspect the ObservationContext

10. - page 10 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Double click obs in the variables window to open the Observation Viewer in the Editor window (creation of a new tab) Inspect the ObservationContext

11. - page 11 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Click Meta Data and scroll to explore some information about your observation Inspect the ObservationContext

12. - page 12 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Single Click obs/level2/HPPPMAPB/0/i mage to view the image of the level2 product MAP Blue Inspect the ObservationContext

13. - page 13 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Double Click obs/level2/HPPPMAPB/0/ image to open the image with the Image Viewer (creation of a new tab) Double Click obs/level2/HPPPMAPB/0/ image to open the image with the Image Viewer (creation of a new tab) Click the zoom button to fit the image to the screen Inspect the ObservationContext

14. - page 14 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Step 3 Extract the frames and related information from the ObservationContext and HIPE

15. - page 15 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> frames = obs.level0.refs["HPPAVGB"].product.refs[0].product HIPE> frames = obs.level0.refs["HPPAVGR"].product.refs[0].product Extract the Level 0 data cube (frames) from the ObservationContext HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecraft after on-board averaging HPPAVGB/R: Herschel PACS Photometer AVGerage Blue/Red This is the signal downlinked from the spacecraft after on-board averaging Level 0: raw data cube Syntax for Blue/Red array

16. - page 16 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.class herschel.pacs.signal.Frames HIPE> print frames metadata Actual datasets stored in the frames: Signal, Status and Mask Inspect the frames

17. - page 17 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.meta HIPE> print frames.meta["mapScanNumLegs"].value 8 Access individual meta data parameter values Number of scan legs Inspect the frames metadata

18. - page 18 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.signal.class herschel.ia.numeric.Double3d HIPE> print frames.signal.dimensions array([16, 32, 2890], int) frames.signal contains the raw data cube The cube dimensions depends on the channel and the duration of the observation frames.signal contains the raw data cube The cube dimensions depends on the channel and the duration of the observation Inspect the frames signal

19. - page 19 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ 30000 ADU Most sources will not be visible on single frames due to intrinsic offset dispersion. Signal dispersion ~ 30000 ADU Readout sequence 10 Hz − PACS Prime 5 Hz – Pacs PMode 64 (32) pixels 32 (16) pixels

20. - page 20 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> Display(frames.signal, depthAxis=2) Indicates that the third dimension is the readout sequence Scroll to explore the data cube Complete documentation on Data Display: Herschel Data Analysis Guide Chapter 2 Complete documentation on Data Display: Herschel Data Analysis Guide Chapter 2 Inspect the frames signal

21. - page 21 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Science Block Calibration Block: prior to any observations, 2 internal sources are observed with the internal chopper for calibration purposes Calibration Block: prior to any observations, 2 internal sources are observed with the internal chopper for calibration purposes Inspect the frames signal

22. - page 22 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Chopping pattern on internal sources 1 & 2 Zoom onto the Calibration Block Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Inspect the frames signal

23. - page 23 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 The spike is likely a glitch Zoom onto the Science Block Inspect the frames signal

24. - page 24 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> PlotXY(frames.signal[8,8,:]) Temporal evolution of pixel (8,8) Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 Complete documentation on Data Plotting: Herschel Data Analysis Guide Chapter 3 The spike is asymmetric and affects only 2 readouts: It is a glitch The spike is asymmetric and affects only 2 readouts: It is a glitch Zoom onto the spike Inspect the frames signal

25. - page 25 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.mask HIPE> print frames.status Inspect the frames status & mask

26. - page 26 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Extract the auxiliary data from the ObservationContext HIPE> pp = obs.auxiliary.pointing HIPE> photHK = obs.level0.refs["HPPHK"].product.refs[0].product["HPPHKS"] HIPE> oep = obs.auxiliary.orbitEphemeris pp: pointing product photHK: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain critical information required in subsequent processing modules. pp: pointing product photHK: photometer HouseKeeping oep: orbitephemeris product We recommend not to modify these variables, they contain critical information required in subsequent processing modules.

27. - page 27 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Extract the calibration files (calTree) from the ObservationContext HIPE> calTree = getCalTree( time = frames.startDate ) The Calibration Tree calTree contains all the files necessary to process your data Some calibration files changed with time, e.g. the SIAM or pointing calibration file, so it is necessary to provide the date of observation for retrieving the appropriate calFiles Complete documentation on the Calibration Framework: http://www.herschel.be/twiki/pub/Public/PacsDocumentation/The_PACS_C alibration_Framework_-_issue_0.10.pdf Complete documentation on the Calibration Framework: http://www.herschel.be/twiki/pub/Public/PacsDocumentation/The_PACS_C alibration_Framework_-_issue_0.10.pdf

28. - page 28 NHSC – DP workshop – Feb. 2011 – N. Billot PACS This is the flatfield in the blue filter Explore the Calibration Tree with the GUI Inspect the calTree

29. - page 29 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print calTree Flight Model 13 th version of the calTree Product specific calTree Inspect the calTree

30. - page 30 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print calTree.photometer Photometer specific calibration products Inspect the calTree

31. - page 31 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> trans = calTree.photometer.filterTransmission.blue["transmission"].data HIPE> wav = calTree.photometer.filterTransmission.blue["wavelength"].data HIPE> PlotXY(wav, trans, xtitle = "Wavelength [micron]", ytitle=“Transmission") Zoom on the feature Inspect the calTree

32. - page 32 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Step 4 From Level-0 to Level-0.5

33. - page 33 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> frames = findBlocks(frames, calTree=calTree) HIPE> frames = detectCalibrationBlock(frames) HIPE> frames = removeCalBlocks(frames) Identify Blocks in the observation and remove the Calibration Blocks Before After The chopping pattern from the Calibration Block is gone.

34. - page 34 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Flag Bad Pixels and populate frames.mask HIPE> frames = photFlagBadPixels(frames, calTree=calTree) Flag saturated pixels and populate frames.mask HIPE> frames = photFlagSaturation(frames, calTree=calTree, hkdata=photHK) CL and ADC saturation checking Provide the photometer HouseKeeping to derive soft saturation limits that depend on the bolometer bias setting More on saturation in Friday talk Photometer: Calibration and other science aspect (9.30 am) More on saturation in Friday talk Photometer: Calibration and other science aspect (9.30 am)

35. - page 35 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.mask BADPIXELS and SATURATION masks were created in frames.mask

36. - page 36 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Convert the signal from digital units (ADU) into physical units (Volts) HIPE> frames = photConvDigit2Volts(frames, calTree=calTree) Convert the chopper angle from digital units (ADU) into physical units (degrees) HIPE> frames = convertChopper2Angle(frames, calTree=calTree)

37. - page 37 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Add an initial estimate of the bolometers noise level to the frames HIPE> frames = photAddNoisePerPixel(frames, method = "median", calTree=calTree) The NOISE dataset is created in the frames object The noise estimate is used later in the deglitching process The NOISE dataset is created in the frames object The noise estimate is used later in the deglitching process

38. - page 38 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Step 5 From Level-0.5 to Level-1

39. - page 39 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Deglitch the data and create a glitch mask in frames.mask HIPE> frames = photMMTDeglitching(frames, incr_fact=2, mmt_mode='multiply', scales=3, nsigma=5, imagenoisearray = frames.noise[:,:,0]) The standard pipeline uses a time-domain deglitching algorithm: Multi-resolution Median Transform of individual pixel timelines (Stark et al. 1998) The standard pipeline uses a time-domain deglitching algorithm: Multi-resolution Median Transform of individual pixel timelines (Stark et al. 1998) The set of parameters given here works well with most observations Bright sources might however be flagged as glitches by this module The set of parameters given here works well with most observations Bright sources might however be flagged as glitches by this module See the next talk, and tutorial PACS-202, for guidance on disabling the GlitchMask on-source. See also the Deglitching talk later today at 4.40 pm.

40. - page 40 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Add the pointing information to the frames HIPE> frames = photAddInstantPointing(frames, pp, calTree=calTree, orbitEphem=oep) Provide the telescope pointing product and the orbit ephemeris that were extracted from the ObservationContext This module adds the spacecraft pointing to the frames.status as coordinates and additional pointing information

41. - page 41 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> print frames.status photAddInstantPointing has created all these variables in frames.status

42. - page 42 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> execfile("/path/to/the/script/plotscanpath.py") HIPE> plotscanpath(frames) plotscanpath.py exploits the pointing information added by the module photAddInstantPointing, namely RaArray and DecArray, to plot the trajectory of the center of the focal plane. This command allows to compile a function at the command line or within a script

43. - page 43 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Add the pointing information to the frames HIPE> frames = photAssignRaDec(frames, calTree=calTree) This modules assigns coordinates to individual pixels using distortion information from the calTree

44. - page 44 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Apply the flat-field correction and convert the signal from Volt/pixel into Jy/pixel HIPE> frames = photRespFlatfieldCorrection(frames, calTree = calTree) You have reach Level 1 The frames are now calibrated and ready for the map-making process (see tutorial PACS-202 & 401) You have reach Level 1 The frames are now calibrated and ready for the map-making process (see tutorial PACS-202 & 401)

45. - page 45 NHSC – DP workshop – Feb. 2011 – N. Billot PACS Save the frames before further processing HIPE> save("/my/directory/my_frames.save","frames") The frames are saved in the “save” format of HIPE The saved file can only be read back into HIPE The frames are saved in the “save” format of HIPE The saved file can only be read back into HIPE HIPE> restore("/my/directory/my_frames.save”)

46. - page 46 NHSC – DP workshop – Feb. 2011 – N. Billot PACS HIPE> FitsArchive().save("/my/directory/my_frames.fits", frames) HIPE> frames = FitsArchive().load("/my/directory/my_frames.fits”) The frames are saved in standard fits format The saved file can be read back into HIPE or IDL The frames are saved in standard fits format The saved file can be read back into HIPE or IDL This is the extension of the fits file Save the frames before further processing

47. - page 47 NHSC – DP workshop – Feb. 2011 – N. Billot PACS http://herschel.esac.esa.int/hcss-doc-5.0/ http://www.herschel.be/twiki/bin/view/Public/ Most relevant documentation for this tutorial: PACS Data Reduction Guide PACS User’s Reference Manual HCSS User’s Reference Manual General documentation necessary to manipulate the data: HIPE Owner’s Guide Herschel Data Analysis Guide Scripting and Data Mining Documentation