ASIC 3 Workshop, May 17, System Level Approach to Satellite Instrument Calibration Space Dynamics Laboratory at Utah State University: Joe Tansock, Alan Thurgood, Gail Bingham, Nikita Pougatchev, Randy Jost NIST: Raju Datla Ball Aerospace & Technologies Corp.: Edward Knight

ASIC 3 Workshop, May 17, Outline Calibration Philosophy “Specmanship” Workshop to Improve Calibration Calibration Planning Subsystem/Component measurements Ground Calibration On-Orbit Calibration –Internal and external calibration sources Satellite Instrument Validation and End-to-end Error Model Summary

ASIC 3 Workshop, May 17, Calibration Philosophy Calibration –Provides a thorough understanding of sensor operation and performance –Verifies a sensor’s readiness for flight –Verifies requirements and quantifies radiometric and goniometric performance –Provides the needed tools to convert the sensor output to engineering units that are compatible with measurement objectives –Provides traceability to appropriate standards –Estimates measurement uncertainties

ASIC 3 Workshop, May 17, Calibration Philosophy – Cal Domains A complete calibration will address five responsivity domains –Radiometric responsivity Radiance and irradiance traceable to NIST Response linearity and uniformity corrections Nominal/outlying pixel identification Transfer calibration to internal calibration sources –Spectral responsivity Sensor-level relative spectral response –Spatial responsivity Point response function, effective field of view, optical distortion, and scatter –Temporal Short, medium, and long-term repeatability, frequency response –Polarization Polarization sensitivity

ASIC 3 Workshop, May 17, Calibration Philosophy – Cal Domains The goal of calibration is to characterize each domain independently –Together, these individually characterized domains comprise a complete calibration of a radiometric sensor Domains cannot always be characterized independently –Complicates and increases calibration effort –Example: Spectral spatial dependence caused by Stierwalt effect Calibration parameters are grouped into two convenient categories –Calibration equation Converts sensor output (counts, volts, etc.) to engineering units –Radiometric model All parameters not included in calibration equation but required to meet calibration requirements

ASIC 3 Workshop, May 17, Calibration Philosophy – Phases of Cal A complete and methodical approach to sensor calibration should address the following phases: Calibration planning during sensor design Ground measurements Subsystem/component measurements Sensor-level engineering tests and calibration Sensor-level ground calibration Integration and test On-orbit measurementsOn-orbit calibration

ASIC 3 Workshop, May 17, Establishment of Good Specifications Improves Calibration Programs often start with a requirement such as –“The instrument shall be radiometrically calibrated to a 3% absolute error, 1.5% band to band error, and a 0.25% intra- band pixel to pixel error” The designers are then asked for cost, schedule, and risk to meet this requirement, which could vary dramatically –E.g., is “error” a 1-sigma or 3-sigma requirement? Furthermore, incomplete, changing, or impossible specifications are often the cause of cost and schedule overruns

ASIC 3 Workshop, May 17, So, What Makes a Good Specification? A good specification clearly communicates what must be accomplished –To an audience that is reading (vs. oral communication) No other “clues” to help understanding –To an audience that may not be able to ask questions easily Example: reading the specification at the end of the program after there’s been personnel turnover –To an audience that may have a different background, training, or understanding of the problem than the author “Good” Requirements Tests (examine every formal requirement with these tests) A.Is the requirement complete (domains, interactions, worst cases)? B.Is the requirement unambiguous (terminology, grammar)? C.Is the specification free of errors (for example, typos, math mistakes)? D.Is there at least one identifiable method to implement this requirement? E. Is there at least one identifiable method to verify this requirement? Also see E. Knight, “Lessons Learned in Calibration Specsmanship, CALCON 2005 proceedings.

ASIC 3 Workshop, May 17, Lessons Learned in Specifications Lessons –Cover all domains (spectral, spatial, temporal, radiometric, polarization) Including interactions and “worst case” for requirements –Scrub for ambiguity –Use mathematical equations whenever possible to define requirements –Have at least one idea for implementation in mind when writing the specification Or upon first round of review/questions –Have at least one idea for verification in mind as well Conclusion –The chance of an instrument Being “poorly calibrated” Overrunning cost and schedule targets c an be reduced with improved calibration specsmanship

ASIC 3 Workshop, May 17, EO/IR Calibration & Characterization Workshops held in Feb 2005 and March 2006 at SDL/USU –Envision self governing community based organization with goal of improving calibration for all participating organizations Workshop Objectives –Explore ways to improve the quality of IR/Visible/UV measurements, community-wide, based on an ISO standard, as pioneered by the RCS community Benefits based on experiences of RCS community –Measurably and quantifiably improve the quality of measurements made in the community –Facilitate data comparison between sensors, systems, facilities, programs and customers –Increase in customer confidence in measurement results due to improved: accuracy, uncertainty, repeatability, comparability, consistent documentation Workshop to Improve Quality of Calibration

ASIC 3 Workshop, May 17, Universal Agreement –There is an unmet need that can not be addressed by any one organization Intermediate results will continue to be presented at annual CALCON (Calibration Conference) For more information –CD available containing the presentations and recommendations of the 2005 and 2006 workshops. – Based on attendee feedback provided at the 2006 workshop, we have started the planning process for the next workshop, to be held Spring 2007, at NIST, in Gaithersburg, MD Workshop to Improve Quality of Calibration

ASIC 3 Workshop, May 17, Calibration Planning Calibration planning –Start as soon as possible (I.e. requirements definition, concept design, sensor design, etc.) –Influence sensor design to allow for efficient and complete calibration –Encourages optimum sensor design and calibration approach to achieve performance requirements Planning phase can help shake out problems –Schedule and cost risk can be minimized by understanding what is required to perform a successful calibration –Calibration equipment needs should be identified early to allow time to build and test any required new equipment

ASIC 3 Workshop, May 17, Calibration Planning Identify instrument requirements that drive calibration Identify calibration measurement parameters and group into: –Calibration equation –Radiometric model Flow calibration measurement parameters to trade study –Schedule –Sensor design feedback –GSE hardware & software –Measurement uncertainty –Risk Perform trade study to determine best calibration approach Mission Requirements Sensor Design Cost & Schedule Risk GSE Hardware & Software Calibration Planning Measurement Uncertainty Instrument Requirements Calibration Measurement Parameters Calibration Equation Radiometric Model

ASIC 3 Workshop, May 17, Subsystem/Component Measurements Subsystem and/or component level measurements –Help verify, understand, and predict performance –Collect Parameters for the Radiometric Model that can't be measured well at the system level –Minimize schedule risk during system assembly Identifies problems at lowest level of assembly Minimizes schedule impact by minimizing disassembly effort to fix a problem System/Sensor level model development and measurements –Allow for the development of Measurement Equation and Performance prediction –Allow for end-to-end measurements –Account for interactions between subsystems and components that are difficult to predict

ASIC 3 Workshop, May 17, Subsystem/Component Measurements Merging component-level measurements to predict sensor level calibration parameters may bring to light systematic system-level uncertainties A,B –Comparison of System-level estimate using component measurements with end-to-end measurement of SABER relative spectral responsivity (RSR) 9 of 10 channels < 5% difference 1 channel  24% difference (reason unknown) Helps to resolve and correct for component degradation and sensor performance after launch A.) Component Level Prediction versus System Level Measurement of SABER Relative Spectral response, Scott Hansen, et.al., Conference on Characterization and Radiometric Calibration for Remote Sensing, 1999 B.) System Level Vs. Piece Parts Calibration: NIST Traceability – When Do You Have It and What Does It Mean? Steven Lorentz, L-1 Standards and Technology, Inc, Joseph Rice, NIST, CALCON, 2003

ASIC 3 Workshop, May 17, Engineering Ground Calibration Engineering calibration –Performed before ground calibration –Perform abbreviated set of all calibration measurements –Verifies GSE operation, test configurations, and test procedures –Checks out the sensor –Produces preliminary data to evaluate sensor performance –Feedbacks info to flight unit, calibration equipment, procedures, etc. Engineering calibration data analysis –Evaluates sensor performance, test procedures, calibration hardware performance and test procedures Based on results of engineering calibration, appropriate updates can be made to prepare for ground calibration data collection

ASIC 3 Workshop, May 17, Ground Calibration Provides complete calibration needed to meet related requirements Is performed under conditions that simulate operational conditions for intended application/measurement Careful in-lab calibration minimizes problems that arise after launch –Minimizes risk of not discovering a problem prior to launch –Promotes mission success during on-orbit operations For many sensor applications –Detailed calibration is most efficiently performed during ground calibration –On-orbit calibration will not provide sufficient number of sources at needed flux levels –Operational time required for on-orbit calibration is minimized Best to perform ground calibration at highest level of assembly possible –Sensor-level at a minimum is recommended

ASIC 3 Workshop, May 17, Calibration continues after ground calibration Internal Calibration Source Response Trending – Trend sensor response to quantify relative response changes over time – Source types Blackbodies, glow bars, diffusers, lamps, etc. – Ensure source is stable and repeatable for sensor operational life Sensor Design/Fabrication Ground CalibrationOn-Orbit/Field Operations Internal Calibration Source Response Trending On-Orbit/Field Calibration/Verification Extending Calibration to Operational Environment

ASIC 3 Workshop, May 17, Internal Calibration Sources Challenges –Ensure calibration source is stable and repeatable for sensor operational life –Ideally, calibration source should use same optical path as external measurements Detailed trade to determine best approach is needed for each specific application Considerations => source type, flux level, configuration, power, space, and weight limitations, etc. –Sources of variability Temperature stability and/or temperature measurement Emissivity changes Thermal variations (external and internal) Separate drift in observed response between calibration source and sensor response Control and/or monitor electronics IR internal calibration source developments are required to achieve stringent stability requirements of many climate change measurements

ASIC 3 Workshop, May 17, Track, trend, and update calibration throughout a sensor’s operational life –In addition to internal calibration sources make use of external calibration sources External On-orbit sources –Standard IR stars Stars  Boo,  Lyra,  Tau,  CMa,  Gem,  Peg –Celestial objects Moon Planets provide bright variable sources Asteroids, etc. Sometimes you have to be creative: –Off-axis scatter characterization using the moon –Other techniques View large area source located on surface of earth (often termed vicarious calibration) Cross-calibration between sensors Use of atmospheric lines Etc. On-Orbit Calibration Verifies Cal and Quantifies Uncertainty

ASIC 3 Workshop, May 17, Satellite Instrument Validation The purpose of validation is to assess actual accuracy and precision of Satellite Instruments by comparison with validating measurements Apparent differences in results between validating and measurement system –Satellite and validation data are not co-located in time and space –Satellite and validating system have different vertical and horizontal resolution –Satellite and validating system have finite accuracy and repeatability –Physical measurement differences (I.e. spectral, sensor, platform, etc.) Validation Assessment Model makes comparisons more accurate by understanding and accounting for theses differences –Make results comparable Validation Assessment Model can be used as a tool to better understand the tradeoff between validation approaches

ASIC 3 Workshop, May 17, Validation Assessment Model Reconcile differences to make results comparable Performance Assessment y End-to-End Error Model Overall Concept True Profile x sat Radiance y sat Parameter Error - δb Noise – ε Instrument SDR Smoothing Parameter Noise Retrieval EDR True Profile x val y val Validation System Radiosondes, Aircraft Measurement Systems, Cross-Calibration, etc. ˆ ˆ x val ˆ y ˆ x

ASIC 3 Workshop, May 17, Summary Calibration Philosophy –What does calibration provide –Calibration domains –Phases of calibration Planning through operational environment Importance and benefit of good specsmanship –Facilitate clear communication and minimizes risk of failure Workshop to improve quality of calibration –Community wide participation working to improve calibration Calibration Planning –Address all phases of calibration as early as possible Specification and design phases

ASIC 3 Workshop, May 17, Summary (cont) Calibration Measurements –Subsystem/Component Measurements Minimizes schedule risk and facilitates development of instrument model and measurement equation –Engineering Calibration and Calibration Methodical and careful approach leads to efficient and thorough calibration –Extending Calibration to Operational Environment Internal calibration sources (I.e. in-flight internal sources) –Challenges and need for improvement External on-orbit sources –External sources and need for improvement Satellite Instrument Validation –Overall concept and the need for validation assessment model to account for differences in space, time, resolution, etc.