1. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 1 The Data-reduction Pipeline for the INT Wide Field Camera Jim Lewis and Guy Rixon Cambridge Astronomy Survey Unit

2. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 2 The pipeline is:  The software package wfcred.  Designed for CCD data from the INT WFC.  Adaptable to other ING imagers; probably to imagers of other observatories.  Runnable on general-purpose computers.  Minimally dependent on specialised s/w infrastructure.

3. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 3 The process Steps shown in yellow are already supported by wfcred. Steps shown in red will be supported soon. Most steps need operator intervention.

4. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 4 Introduction to wfcred  Developed by JRL with input from Peter Bunclark (IoA), Mike Irwin (CASU) and Robert Greimel (ING).  Designed to have minimal user interaction.  Work is done by IRAF: mscred plus several home grown applications in C, SPP and cl.  Command line as well as Perl/Tk user interfaces.

5. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 5 WFCRED Components  Two components  caltool & calcombine:  Does combinations of bias and flat frames  caltool allows for some user interaction  process:  Processes target frames with no user interaction

6. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 6 What wfcred does now (1)  Give each frame a rough WCS  Calculated from RA, Dec, aperture offsets, rotator angle and known camera geometry  Good to 5-10 arcsec (usually)  Linearise the data  WFC detectors are not linear.  Interpolate to remove bad rows/columns

7. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 7 What WFCRED does now (2)  Bias subtract  Either a constant or a mean bias frame  Flat field division  mscred does a "gain correction" at this stage  Image detection  Uses an algorithm similar to the APM image detection programs  This is not the final image-catalogue!

8. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 8 What WFCRED does now (3)  Proper WCS definition  X,Y positions taken from objects detected in the previous step  Equatorial coordinates of standards taken from: catalogues of APM scans of POSS1 or UKST plates (via the internet); locally held GSC FITS table; any other locally-held FITS table; a combination of these.  Uses IRAF's ZPX projection.  Internal accuracy generally 0.3'' or better. (Target 100mas)

9. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 9 What WFCRED does now (4)  Defringing  Mean fringe frames are either: Formed from current set of target frames Chosen from previously defined mean fringe frames  Fringes are removed by scaling the mean fringe frames to each individual target frame.  Defringing is automatic once fringe-frames are chosen or generated.

10. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 10 What WFCRED will do soon  Catalogue generation:  Generate image lists.  Image Classification.  Improved WCS definition.

11. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 11 Order of processing  Pipeline processes all frames found in the as one current working directory as one job: typically data from 7 nights of observations.  wfcred sorts frames into families by CCD number and filter.  Each family uses the same calibration frames.  Wfcred processes each family to completion before starting on the next.  This makes the most-efficient use of IRAF.

12. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 12 Parallelism  Runs on uni-processors, multi-processors, Beowulf clusters.  Parallelism in units of one “family” of frames: each call to IRAF can go to separate processor.

13. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 13 Areas of Difficulty  Poorly characterised detectors  Poorly defined meta-data  Poorly defined end product  Lack of observing protocol

14. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 14 Poorly Characterised Detectors  Non-linearity in WFC "detectors".  Dud pixels

15. Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 15 Poorly Defined Meta-data  Headers are incomplete, inconsistent or wrong a certain percentage of the time.  Pipelines and archives are header driven.  Means that a preprocessor is sometimes required.