1. Data Handling and Reduction Jamie Stevens May 13 2009

2. Introduction CABB will be different, but similar What should you expect? What should you look for during reduction? Communicating your experiences

3. CABB Data basics CABB data is still in RPFITS format MIRIAD is going to be the package of choice for CABB reduction alterations are being made to support CABB data Other packages won’t be supported

4. CSIRO. Data Handling and Reduction, April 8 2009 CABB File Sizes CABB files are BIG, really BIG You might think downloading videos from the internet takes a lot space, but that’s just peanuts to CABB! Current system of 2 IFs, 2048 channels per pol, 2 pols, 15 cross- correlations + 6 auto-correlations gives a rate of ~24 GB/day Full system with all zoom bands will generate over 400GB every day Each file is limited to 4GB by the correlator Files will close automatically when it reaches this size, and a new file opened Each file represents about 4 hours of observing Files are limited to this size so they can fit on DVD, and are below the maximum file size supported by FAT32 filesystems

5. Taking data away A new approach to archiving is needed Users should expect to either take a lot of DVDs away with them, or bring an external USB hard drive Computers in the observers area can be used to transfer data from the correlator to your hard disk Only support USB interface Only support FAT32 or ext2/3 disks Of course you can just plug in your laptop and use scp If you’re in ATNF, then you can copy it from the correlator over the network If you’re outside, check that your institution doesn’t charge large amounts for big data transfers! Or you can wait for the data to get copied to the archive (a couple of weeks at the moment, should decrease in the future to hours)

6. Reduction There is a new MIRIAD version that you will need to use to reduce CABB data New ATLOD needed for new file format, including yet-to-be- supported zoom modes New uvsplit allows the wideband to be split up into smaller band chunks Array sizes changed behind the scenes to allow plotting etc. of big datasets New MIRIAD is installed on kaputar Can get new binaries/sources from usual web address

7. CSIRO. Data Handling and Reduction, April 8 2009 CABB File Format Currently, a CABB file has 2 IFs with 2048 channels (IFs 1 & 2) Using ATLOD Either use ifsel to choose which IF to extract from the file Can’t select more than 1 at a time OR extract all IFs and use select=window(n) to select IF n My experience is that this doesn't always work

8. CSIRO. Data Handling and Reduction, April 8 2009 Reduction Philosophies Two main philosophies: All-at-once Divide and conquer All-at-once Have to be careful about large fractional bandwidth effects Increased sensitivity Increased complexity with current routines Straightforward process Divide and conquer Can use current routines with little change More effort required to run for each sub-band More complicated to produce a wideband image

9. Experiences Reduction process has not changed a great deal Calibration still uses the same routines mfcal, gpcal, gpboot, gpcopy all work on the increased bandwidth Imaging should be considered carefully Recommend using mfs for all observations, and using mfclean as well There are some things that you should look out for

10. Calibration Mfcal Works as expected Uses models of source to perform calibration Be careful with choice of interval May be more prudent to calibrate on chunks of time with select than to increase interval Gpcal Works as expected Doesn't require as much buffer space as mfcal, so shouldn't have problems in this way gpcopy & gpboot Use them as usual mfboot != better gpboot!

11. Inspecting Data Quality uvplt, uvspec, gpplt all necessary tools for inspecting quality of your data By default, uvplt will average over frequency This can hide some problems from you that are more easily observed with options=nofqav

12. Interference Self-generated interference can cause some problems when the source is transiting ie when the fringe rotation rate is low restricted to known channels, easily flaggable

13. Interference

14. Flagging Can achieve good results with only uvflag Flag based on channels and time ranges Flag out 50 channels from each edge Other flagging options Blflag Suffers because of default frequency averaging Tvflag Still doesn't run with decent colour depth Tvclip Can be run in batch mode without output, useful for flagging out obviously wrong amplitudes/phases Pieflag Struggles with such large datasets

15. Calibration Quality Can achieve good quality calibration using entire bandwidth

16. Imaging When making broadband image, should strongly consider using mfclean If your source strength varies with frequency, clean will distort your image Use of mfclean particularly important for high dynamic range imaging Need to set options=mfs,sdb in invert Also need to image out to three times your primary beam size (setting imsize values) When running mfclean, need to set region to be just the area inside the primary beam Can be useful even with smaller bandwidths, as dynamic range is improveddynamic range is improved

17. Broadband imaging CenA at 12mm

18. Divide and Conquer Using uvsplit, you can break up your dataset into chunks of smaller bandwidth Use the new maxwidth option As calibration works quite well over 2 GHz, suggest splitting after calibration, just before imaging Imaging smaller bandwidth chunks has its advantages Less problems with fractional bandwidth effects Can measure spectral variation of source flux density

19. Calibrators We are mindful of the need for wide-band characterisation of calibrators, and new mm calibrators Two projects are going to be doing this C007 Edwards C2050 Newton-Mcgee First C007 data from immediately after science operations start has been reduced and is available through calibrator search tool 7mm fluxes for most mm sources across the sky

20. Experiment! Data coming from the CABB system is looking good, and we have some exciting results coming up in later talks Good time now, before your observations, to take some existing data and practice your reduction Credit to Baerbel Koribalski

21. Summary CABB is on-track and looking good Getting experience in reducing the new data is vital Please communicate successes, failures to us so we can improve our understanding of the processes, and improve MIRIAD A big thanks to Warwick and all the people who made CABB a reality! Contact: Jamie Stevens re: data reduction Mark Wieringa re: MIRIAD CABB issues Anybody who has done CABB observing/reduction!

22. An Aside

23. Contact Us Phone: 1300 363 400 or +61 3 9545 2176 Email: enquiries@csiro.au Web: www.csiro.au Thank you CSIRO ATNF Narrabri Jamie Stevens ATCA Senior System Scientist Phone: 02 6790 4064 Email: Jamie.Stevens@csiro.au

24. CSIRO. Data Handling and Reduction, April 8 2009 Reduction 1934-638

25. Reduction Using normal cleanUsing mfclean

26. Interval Choice When running mfcal on large dataset, it may stop and complain about running out of buffer space Will suggest that you increase interval Be very cautious about doing this, especially at higher frequencies interval=1interval=0.1