1. Status Report of EDI on the CAA

2. CAA Public website
All pages fully accessible by anyone

3. CAA Test Area Currently services accessible only by ESTEC/ESAC
Individual instrument team members will be allowed on a need basis
Some of the CAA Command-line services are available to all

4. CSA Public website

5. EDI Datasets for Downloading

6. CSA EDI Graphics

7. EDI Status Information
panel1: EDI general – NOT DONE YET
mode__C1_CQ_EDI_CLIST [value -1, 0, 1]; note that -1 needs to be generated
Status__C*_CP_EDI_AE, bit 0 [value 0 or 1]
Status__C*_CP_EDI_AE, bit 1 [value 0 or 1]
crf__C1_CP_EDI_CRF, float, values up to ~20 (kHz]
code_flag__C1_CP_EDI_CRF [value 0 or 1]
panel2: Status__C*_CP_EDI_MP
status[1]: 0, 1, 2        -> plot: status[1]                   values=0, 1 or 2
status[2]: 0, 100        -> plot: status[2]/100          values=0-1
status[3]: 0-100        -> plot: status[3]/100          values=0-1
status[5]: 0-100        -> plot: status[5]/100          values=0-1
status[6]: no idea of max value
status[7]:0-180         -> plot: status[7]/90             values=0-2
panel3: Status__C*_CP_EDI_MP
status[4]: integer value between  0-255        seven bits can be set -> resolve bits
plot seven bits: if bit is set, it has value = 1, else = 0

8. EDI Status Information

9. CAA EDI QL
Should one have one plot for each spacecraft? Either 3- or 24-hr plots
Combine observations from both AE and E-field modes
Add auxiliary information
This could be generated by the CAA using the calibrated data products

10. EDI inconsistencies Notes:
All QZC counts = 0 => not displayed
Notes:
If EGD exists, there is a chance for PP/SPIN/MP
If EGD does not exist, no chance for PP/SPIN/MP
QZC and CRF should exist always in EF-mode, so they should have the same coverage as CLIST/EF-mode
PP and SPIN should have identical coverage
MP should have a wider coverage than PP/SPIN (all qualities)

11. Dataset Inventory Analysis
There are several similar datasets where the main difference is that data vectors or matrices are converted into another system. For instance, scientific data are given in different scientific units or coordinate systems
Consistence of such datasets have never been investigated although there is some evidence that some errors can occur
Such similar datasets can often be expected to have the same number of records
Exception: if a dataset is given in raw units, the corresponding datasets in scientific units may have less records if poor records were deleted
However, if poor data have been replaced with FILLVAL, the same number of records should exist
In addition such datasets should have similar values for
Version numbers
Generation and ingestion dates
Note: these metadata can have significant differences, so there is no automatic
CAA has developed a tool that collects such metadata and gives them in a text file

12. Inventory Output
The beta version of the inventory tool is available (currently) at
Inventories are executed a few times a week, the inventory generation date is shown in the header of the inventory

13. Example: PP – SPIN datasets
Inventory analysis for C3 "Spin resolution" (C3_EDI_SPIN). Compared datasets:
1: C3_PP_EDI
2: C3_CP_EDI_SPIN
Generation date: :32:31
Analysis is based on the database content at :25:12
Data coverage: T00:00:00Z/ T00:00:00Z
Processing policy:
- DON'T CHECK FILES
- DON'T CHECK NUMBER OF RECORDS
- DON'T CHECK TIMESTAMPS
- DON'T CHECK FILE VERSIONS
- DON'T CHECK IF GENERATION DATES ARE WITHIN 90 DAYS
- CHECK IF DATASET 1 IS THE OLDEST ONE
- EXTEND FILE TIME SPAN
- IGNORE LAST FILE IF UNTIL AROUND 2020
- SPECIAL EDI 1:
- CHECK NUMBER OF RECORDS AND TIMESTAMPS ONLY IF THE NUMBER OF RECORDS IN
PP_SPIN IS LARGER THAN 0 AND IN CP_EDI_SPIN DIFFERENT THAN 1 (ERR R, T)
- REQUIRE THAT EDI_SPIN FILE DOESN'T EXIST DURING PROBE FAILURES:
- C1: from
- C2: from

14. Example: PP – SPIN datasets
Date OK? ERR R R2 V1 V G G I I2
ERR F T15:11:44Z T17:37:17Z
ERR F T14:05:28Z T15:30:41Z
ERR F T15:28:00Z T08:27:32Z …
ERR F T15:28:00Z T08:27:48Z
ERR F T09:47:18Z T17:23:00Z
ERR F T09:47:28Z T17:23:12Z
ERR F T15:45:35Z T08:35:08Z
ERR F T15:45:36Z T08:35:10Z
ERR F T15:45:37Z T08:35:12Z
ERR F T15:45:38Z T08:35:14Z
ERR F T15:45:39Z T08:35:16Z
ERR F T15:45:40Z T08:35:19Z
ERR F T15:45:41Z T08:35:22Z
ERR F T15:45:42Z T08:35:24Z
ERR F T15:45:43Z T08:35:27Z
ERR F T15:45:44Z T08:35:29Z
ERR F T15:45:45Z

15. Example: EDI – AE datasets
Inventory analysis for C1 "Ambient Electron Counts" (C1_EDI_AE). Compared datasets:
1: C1_CP_EDI_AE
2: C1_CP_EDI_AEDC
Generation date: :32:31
Analysis is based on the database content at :25:04
Data coverage: T00:00:00Z/ T23:59:56Z
Columns description:
Date: YYMMDD
OK?: OK: comparison OK, ERR: error
ERR: If ERR, the reason for error:
F: Not all files exist R: Number of records don't match T: Timestamps don't match
V: Versions don't match G: Generation dates error S: Special conditions not met (if applicable)
Rx: Number of records in file x Vx: Version of file x
Gx: Generation time Ix: Ingestion time
Processing policy:
- CHECK FILES
- CHECK NUMBER OF RECORDS
- CHECK TIMESTAMPS
- DON'T CHECK FILE VERSIONS
- CHECK IF GENERATION DATES ARE WITHIN 90 DAYS
- CHECK IF DATASET 1 IS THE OLDEST ONE

16. Example: EDI – AE datasets
Date OK? ERR R R V1 V2 G G I I2
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR F T17:10:37Z T09:37:55Z
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR MORE THAN ONE FILE WITHIN ONE DAY ***
ERR G T14:32:39Z T18:52:53Z T08:16:49Z T13:52:46Z
ERR F T15:35:31Z T18:38:32Z
ERR F T14:12:32Z T09:14:23Z
ERR F T14:13:10Z T09:14:52Z
ERR F T14:13:48Z T09:15:05Z

17. Example: EDI – AE datasetsOK
T15:08:51Z T16:10:06Z T16:25:56Z T10:35:30Z
ERR MORE THAN ONE FILE WITHIN ONE DAY *** OK
T15:09:23Z T16:10:40Z T16:26:06Z T10:35:34Z
C1_CP_EDI_AE_ _v27.cef:
T00:00:00.000Z / T23:59:57.048Z
C1_CP_EDI_AE_ _v27.cef.gz                   
T23:59:57.049Z / T23:59:59.850Z

18. EGD - Duplicate Time Stamps

19. EDI Delivery/Ingestion Activity
The plots are regenerated daily around mid-night
Monthly and 6-month plots
Top two panels are taken from database
Top: Number of files ingested into the database
2nd from top: average time used for one file to validate/add into the database
Bottom five shows an instantaneous situation at the time of plot production
3rd: Number of files failed validation: e.g. wrong version number
4th and 5th: number of CEF and nn-CEF files in the delivery area
6th and 7th: number of CEF and non-CEF files waiting for validation

20. Status of File Transfer to CSA
File transfer from CAA to CSA has been automated
The two database are now identical
New files on the CAA will be transferred to the CSA within a day