1. FGM report 5 th Operations Review of the CAA Elizabeth Lucek, Patrick Brown Chris Carr, Tim Oddy, André Balogh I mperial College London June 9 th 2010

2. Introduction The FGM team generates the following data products (1 file per orbit) –Primary data products Full resolution magnetic field vectors (FULL) Magnetic field averaged to 5 vectors/second (5VPS) Spin averaged data (SPIN) –Each product is calibrated using the same calibration file valid over 1 orbit –Data are averaged using a boxcar window, implemented in the data processing software –Supporting data products Files listing data gaps introduced as the processing stage (GAPF) –These gaps normally indicate that no data are available Files listing data gaps introduced at the validation stage (VALF) –These gaps are introduced when bad or corrupted data are removed Caveat files (CAVF) –These are generated if a problem is found with the data or the calibration Calibration accuracy files (CALA) –Files contain diagnostics of the quality of calibration achieved for that orbit 2

3. Management Funding level for FGM CAA activities –Until March 2010: 1 FTE/year (STFC) + 0.5 FTE/year (ESA) –April – Dec 2010: 0.5 FTE/year (STFC) + 0.5 FTE/year (ESA) –STFC funding will cease in December 2010. Negotiations underway with ESA for continued support of critical activities in 2011-2012 Personnel –Paul French (Archive Scientist) left the team in October 2009 –Since then his duties have been divided among the team, and we expect this situation to continue for the near future –We have benefited from some invaluable support from Leah-Nani Alconcel, who previously worked on CAA tasks –Patrick Brown took over the role of FGM Operations and Data Processing Manager when Chris Carr took up a staff position at Imperial College in October 2008 3

4. Status of software development – I – completed tasks Changing Cluster orbit  FGM making measurements in range 6 (not designed for science use) and range 7 (designed for ground testing only) –First R6 data in Sept and Oct 2008 –First R7 data in Nov 2009 –Changes to data processing and calibration software were therefore required –Initial calibration parameters were required for R6, since no ground testing was done FGM data processing software has been updated to handle R6 and R7 by Karl-Heinz Glassmeier (TUB), Edita Georgescu (MPS, Lindau) and Karl-Heinz Fornacon (TUB) –We thank them for making the changes FGM CAA calibration software has been updated to calibrate data from R6 Initial R6 calibration parameters have been calculated 4

5. Status of software development – II – outstanding tasks Modifications to FGM CAA calibration software to calibrate data from R7 are in progress –Aim to complete changes in order to deliver data in R7 from 2009 before Oct 2010 Changes are required to the software to generate auxiliary files, especially the CALA files –Modifications to CALA software to allow processing of R6 and R7 outstanding –We are in the process of considering whether the CALA files would be better replaced with more detailed CAVF files supplemented by the FGM User Guide Automated generation of CAVF files –This is incomplete due to lack of resources Preparation of calibration refinement files to allow tracking of changes in calibration parameters in a single orbit –These files would be particularly useful for recent periods, where eclipses and payload switch-off has led to calibration parameters often changing within a single orbit –BUT we have had insufficient resources to complete the development 5

6. Status of experiment calibration/cross calibration The FGM calibration procedure is described in: –FGM Calibration report Issue 2 –J. Gloag J. M., E. A. Lucek, L.–N. Alconcel, A. Balogh, P. Brown, C. M. Carr, C. N. Dunford, T. Oddy, J. Soucek, ‘FGM data products in the CAA’, accepted for publication in Astrophysics and Space Science Proceedings Determination of spin axis offset –Using appropriate solar wind data, inter-spacecraft accuracy of ~0.1 nT in R2 is possible –When solar wind coverage is poor (e.g. Dec 2008) or absent (tail season) inter- spacecraft accuracy can be less good (e.g. a total spread of 0.25-0.5 nT) This is not a significant problem in high fields Largest effect in short intervals of solar wind data (e.g. May) at change of season Most significant cases covered by CAVF files –We would like to investigate this problem further, but have not had the opportunity in the last year 6

7. Status of dataset delivery DatasetsDelivered intervalComments FGM magnetic field data: FULL, 5VPS, SPIN – V01 Feb 2001 – Feb 2009Preparation of these core datasets was our highest priority. FGM supporting data: GAPF and VALF files Feb 2001 – Feb 2009These files are prepared routinely along with the core data sets. FGM calibration files: CALFFeb 2001 – Feb 2009Detached headers required in order for these non-CEF data files to be ingested has been submitted to the CAA. FGM caveat files: CAVFFeb 2001 – Dec 2006Files are prepared for affected orbits only. Lack of resources has led to a delay in the preparation of CAFV files, now underway FGM calibration accuracy files: CALA Feb 2001 – Dec 2006, files prepared for all orbits Additional software modifications are required in order to generate CALA files for R6 and R7 (from Sept 2008) 7 Staffing problems led to some delay in data delivery, but we are now catching up with the schedule With limited resources over the last year, we have concentrated on preparation and delivery of the core data

8. Data delivery plan for the remaining years FGM data delivery dates ProductsDS 07NS 07DS 08NS 08DS 09NS 09 Cx_CP_FGM_FULL30/05/200831/07/200830/09/200802/12/200930/06/201030/09/2010 Cx_CP_FGM_5VPS"""""" Cx_CP_FGM_SPIN"""""" Cx_CC_FGM_CALF"""""" Cx_CQ_FGM_GAPF"""""" Cx_CQ_FGM_VALF"""""" Cx_CQ_FGM_CAVF31/10/2010 Cx_CQ_FGM_CALA"""""" 8 We expect to deliver data up to October 2009 by late summer 2010 Data for Nov – Dec 2009 Contain R7 data We expect to deliver these data by October 2010 CAVF files: We hope to deliver these ahead of the schedule above CALA files: These require additional study and work before preparation and delivery

9. Documentation Documentation for FGM and FGM data –Interface Control Document (ICD) New ICD delivered to the CAA (CAA_FGM_ICD_0002_V0_5) May 2010 –FGM User Guide V 2.0 delivered to the CAA in May 2010 Includes examples showing signatures in the data that arise from imperfections in calibration –Cross-Calibration report V 2.0 delivered to the CAA in May 2010 Description of calibration procedure –J. Gloag J. M., E. A. Lucek, L.–N. Alconcel, A. Balogh, P. Brown, C. M. Carr, C. N. Dunford, T. Oddy, J. Soucek, ‘FGM data products in the CAA’, accepted for publication in Astrophysics and Space Science Proceedings. 9

10. Replies to the recommendations of the 4 th OR - I OR04-R1: The review board would like to commend the teams on their efforts to provide data to the CAA in a timely manner. The delivery schedule requires that data to the end of 2009 be ingested into the CAA by May 2010, and the board expects that teams will adhere to this schedule. Teams should recognize that the current mission ends soon and should be preparing for this eventuality. The delivery schedule for 2009 was updated and we anticipate that all data will be delivered before October 2010. The 2008 files were delivered well before the OR-5 review in May 2010. OR04-R2: The board would like to reiterate that the CAA represents the final and full repository of Cluster data, and that every effort should be made by instrument teams to include all, best-calibrated data possible. This is our principal aim. OR04-R3: The board notes that the peer review process undertaken in 2009 was valuable. The board recommends that the CAA and instrument teams attempt to address and implement recommendations from the peer review to the level that is feasible within the existing resources. We have addressed the recommendations of the peer review board as far as we are able. 10

11. Replies to the recommendations of the 4 th OR - II OR04-R12: The instrument user guides are an essential tool for the CAA end user. The board is pleased to note the development of these guides by the instrument teams, as requested at the last review. However, the board notes that some of the guides could be improved by making them more complete, self-contained, readable, and usable. For instance, full descriptions of quality flags should be included, with diagrammatic examples of caveats where appropriate; plots of specific examples taken from the CAA graphical output would be most useful for the user. The board requests that all user guides are brought up to a consistent standard, as exemplified by the RAPID user guide. We have prioritised updating the FGM UG and have added graphical examples of instrumental or calibration artefacts in the data as listed above. OR04-R14: The board also commends the inter-instrument cross-calibration efforts that have been undertaken, believing that this is the best route to producing high-quality, consistent data. The board recommends that these cross-calibration activities continue. The board also recommends that inter-spacecraft cross-calibration is undertaken as well and the results are presented in the calibration report. FGM will continue to implement inter-spacecraft cross calibration where is it appropriate and if we have sufficient resources to do so. 11

12. Status of Actions items: Closed items - I Closed actions from the 2008 CAA/FGM Progress Meeting FGM08-AI-2: provide a header file for the CALF dataset (calibration files) CLOSED Sample header files for the CALF files have been prepared for all four Cluster spacecraft and sent to the CAA. FGM08-AI-1: FGM to check inventory plots as there are some missing files of high resolution data. CLOSED It was very time consuming to check for missing files using the inventory plots, so staff at the CAA checked the FGM data product database and sent a list of missing files to FGM Closed actions from the 8 th Cross-Calibration meeting CC8-AI-12: to explain why axial offset error was up to 1 nT as one expect it to be ~0.1 nT and why it was limited to only certain intervals. As the axial offsets during the tail season are based on interpolation of the solar wind seasons, does this have any impact on other months? CLOSED This is an occasional problem affecting low |B| in the solar wind. It has a minor effect in high fields Three intervals of data were affected: –Jan 2003, Feb 2003 (weeks 1 and 2) : recalibrated and redelivered COMPLETE –Mar 2002: seems to result from poor data coverage: CAVF files have been submitted COMPLETE 12

13. Status of Actions items: Closed items - II Closed actions from the 2009 CAA/FGM Progress Meeting FGM09-AI-3: to redeliver the missing files reported by the CAA: CLOSED The list of files was received. Those files that were present in the Imperial Archive redelivered. Four files required reprocessing. Three have been completed and delivered but the fourth is a day where the data are corrupted. Closed actions from the 10 th Cross-Calibration meeting CC10-AI-5: to investigate the source of high-frequency noise in an event of 2002-Jul4, 12:50-12:52 UT (C1); for reference, see STAFF presentation (page 9): CLOSED This is a case where the FGM noise floor can be observed. 13

14. Status of Actions items: Open items Open actions from the 8 th Cross-Calibration meeting CC8-AI-13: Teams to investigate whether it is feasible to produce data for the commissioning phase. OPEN This is feasible in not straightforward. It is likely that a cut-down calibration procedure would be used. In the first instance we plan to generate data from Jan 2001 using the calibration files generated for the first half of February 2001 (see FGM09-AI-5 below). Open actions from the 2009 CAA/FGM Progress Meeting FGM09-AI-5: to consider producing the FGM files for January 2001 using the calibration files of Feb 2001: OPEN We have considered this. The task should be relatively straightforward and will be done when time allows. 14

15. 2009 CAA Peer Review Recommendations to all teams - I PR09-ALL01 – UG should illustrate instrument issues. –We have added figures showing the following to the UG residual spin tone in the data residual jump at a range change spectral signature of residual tone at the spin frequency and twice spin frequency, also showing that single calibration file cannot always fit the data from a full orbit PR09-ALL02 – Caveat information –Caveat information routinely produced covering interval affected and likely cause if known up to 2006. Remainder will be delivered as high priority PR09-ALL03 – ICD updates –FGM ICD updated to include CALA file description. Will consider the need for a further update before end of CAA PR09-ALL04 – Spacecraft-intercalibration –Some routine measures, e.g. when spacecraft are in solar wind –Results included in Calibration Accuracy (CALA) files –Formal inter-calibration rarely possible 15

16. 2009 CAA Peer Review Recommendations to all teams - II PR09-ALL05 – Discussion of data errors/standard deviations –Qualititative estimates of data quality and quantitative measures are included in the CALA files PR09-ALL06 - A mission-long assessment of inter-calibration fidelity is missing from the calibration reports. –Analysis of the CALA files will be included in the calibration reports if time allows PR09-ALL07 – Documentation of physical region – instrument mode mismatch issues –FGM does not suffer from these effects PR09-ALL08 – Differences between CAA ‘simple’ and ‘advanced’ interfaces –FGM data are straightforward enough that we anticipate that the two interfaces will be the same PR09-ALL09 – Inclusion in Calibration Report of CAA generated cross calibration plots –We are not currently aware of any proposals of this nature for FGM PR09-ALL10 – Discussion of use of PP/CAA data –PP data are included in the CAA for completeness. –The difference between PP and CAA data has been described in Issue 2 of the UG. –Advise that CAA data used where available 16

17. 2009 CAA Peer Review Recommendations to all teams - III PR09-ALL11 - Digital versions of calibration results presented in the calibration reports should also be archived at the CAA. –Not currently applicable to FGM since results derived results from calibration files are already archived in the CAA PR09-ALL12 – All reference documents referred to in the UG should be available at the CAA –Most FGM references are publically available. –The data processing software has been updated, and the guide to this software will be included once it has been updated PR09-ALL13 – The UG should contain a table of all datasets in the CAA –This table has been included in Issue 2 of the FGM User Guide. PR09-ALL14 – All teams should include some flags in their data to indicate usability –Changes to FGM data products (FULL, 5VPS, SPIN) not feasible at this stage –Calibration accuracy files contain parameters summarising the usability of the data for different types of science. 17

18. 2009 CAA Peer Review Recommendations to FGM - I PR09-FGM01 – UG should show how to identify data intervals with range changes, interference and spin modulation –These effects are already explained in the User Guide, but additional figures have been included in Issue 2 as listed in the response to PR09-ALL01 PR09-FGM02 – A description is needed of the Calibration Accuracy files –A description was included in the UG but some additional information on interpreting the ‘score value’ has been included in Issue 2 PR09-FGM03 – The FGM team should try to provide a plan for long-term calibration issues such as instrument drifts –Long term drifts in instrument calibration parameters are accounted for in the routine calibration, and no further action is required PR09-FGM04 – PP data should not be included in the CAA –PP/UP data have been widely used, so it is essential that they are archived. However, we recommend SPIN for new studies PR09-FGM05 – Variances should be included in 5VPS and SPIN –With hindsight inclusion of the variances might have been useful, changes to the CAA datasets (FULL, 5VPS and SPIN) are not feasible at this stage 18

19. 2009 CAA Peer Review Recommendations to FGM - II PR09-FGM06 – Quality flags and range information should be added to each record –We do not think that meaningful quality flags can be applied vector-by-vector to the FGM data –The instrument range and mode (normal or burst mode) are already supplied as part of every data record. PR09-FGM07 – FGM should be provided in ISR2 –If co-ordinate transformations are required for data products from more than one instrument we suggest that it would be better included as part of the CAA interface PR09-FGM08 – For SPIN and 5VPS the metadata should contain an entry describing how the averaging is done –This information has been included – averaging is done using a boxcar average. PR09-FGM09 – On the CAA Web page the dataset description for the CALA files is missing –This description has been supplied to the CAA 19