NCRs and Waivers GLAST Large Area Telescope LAT Pre-Shipment Review Gamma-ray Large Area Space Telescope GLAST Large Area Telescope LAT Pre-Shipment Review NCRs and Waivers Pat Hascall Systems Engineering Stanford Linear Accelerator Center NCRs and Waivers

NCR Introduction Presentation focus is on the main hardware related NCRs that remain open Impact assessment for those NCRs identified as “Can Not Duplicate” (CND) will be discussed. Several NCRs were left open for Environmental Testing at NRL and will be closed when final data review is completed after TV (Documentation of analysis or minor rework/inspection) NCRs are classified into categories for discussion purposes NCR Summary List of all open NCRs is presented for reference NCRs and Waivers

NCR Category Definitions Open NCR’s classified into categories for discussion purposes: Category Definition Count Hardware Discrepancy H/W Issue, half can be closed with minor rework or inspection after TV 6 FSW Discrepancy Identified FSW bug, FSW JIRA in work or completed Monitor for Verification Likely test issue, trending for repeat 2 Known Feature Specification not violated, but trending or changes required to accommodate behavior 11 EGSE/Data Processing NCR has been isolated to EGSE/Data Processing 12 CND Could not duplicate 7 NCRs and Waivers

NCR Resolution Definitions Open NCR’s classified into categories for discussion purposes: Closure Plan Definition Count Issue resolved, closure plan in place or no impact to LAT Closing Issue resolved, documentation required to close 13 Close after TV Monitored during TV with no issues seen (complete writeup and close) or minor rework scheduled after TV 5 B0.6.11 Resolved with FSW release B0.6.11 1 B0.7.0 Resolved with FSW release B0.7.0 4 B1.0.0 Resolved with FSW release B1.0.0 Closure deferred EGSE issues, low priority to fix or hold open to track repeats 9 Work continuing QAR CND or other issue expected to be transferred to a QAR 7 In process Investigation ongoing 3 Waiver Waiver in process NCRs and Waivers

NCR Resolution vs Category NCRs and Waivers

NCRs For Discussion NCR Date Opened Affected Hardware Problem Description NCR Category Resolution 855 3/23/06 DAQ LATC verify occcasionally reports errors in CRC, SPT, ARC CND QAR 880 4/12/06 SIU Spontaneous Reboot 881 EPU Spontaneous Reboot 902 5/9/06 EPU 0 unexpected reboot 922 6/5/06 Housekeeping telemetry stopped FSW Discrepancy in process 946 7/11/06 ACD ACD FREE board 5 not responsive after power up 948 7/19/06 EPU 1 reboot, EPU 2 reboot 949 7/21/06 Primary SIU reboot, redundant SIU reboot 957 8/4/06 EPU 0 CPU junction temperature higher than expected in vacuum Hardware Discrepancy Closure near 975 9/1/06 Datagram timetag error NCRs and Waivers

NCR 855 LATC Verify Errors CND, QAR Issue Calorimeter and Tracker front-end register (RC and FE) was not read successfully Analysis Issue in 16 of 3805 LATC configurations Affects about 10 bits of ~ 2 million bits written/read in each LATC execution Mostly at the start of commissioning: Fourteen happened in first 290 runs One in run 777 One in run 1426 None in last 2400 write/reads None using FSW B0.6.9 (2120 runs) Have not been able to replicate on the testbed Resolution Plan Mitigation plans in FSW JIRA FSW-653 (B0.6.10) adds a reset to the calorimeter at power up JIRA FSW-729 adds a retry by LATC when a verify error occurs, to be implemented if the problem reappears Impacts on On-orbit performance With no action, the LAT will not start the physics acquisition and one orbit worth of data would not be collected NCRs and Waivers

NCR 922 Housekeeping Telemetry Stopped QAR Issue: During a charge injection test, LAT housekeeping process (LHK) has twice emitted an error and subsequently did not create telemetry packets successfully Analysis: Review by FSW indicates a possible cause that the LHK and LCI (LAT Charge Injection) process may be in contention for memory Was reproduced on testbed, analysis in progress Resolution Plan: Find and correct FSW error Impacts on On-Orbit performance: Loss of housekeeping data until restart NCRs and Waivers

NCR 946 ACD Free 5 Power Up CND, QAR Issue: ACD Free Board 5 was not responsive to commands after power up Analysis: Other power up parameters (voltages and currents) reviewed with no abnormalities seen Occurred once in over 275 power ups (122 using redundant GASU) Two possible causes Look-at-me from redundant GASU did not reach or might have been decoded improperly on the Free board in which case it would only respond to commands from the primary (unpowered) GASU Custom power up sequence (compensates for an old bug in ASIC primary/redundant switching code) did not result in successful power up in redundant mode Resolution Plan: Cycle power to FREE card (interim solution) Add functionality to FSW to be able to send Look-at-me command to a FREE card without power cycling (JIRA FSW 718 adds capability, targeted for B1.0.0) Impacts on On-Orbit performance: Extend power up sequence timeline slightly NCRs and Waivers

CPU Reboots CND, QAR Issue EPU and SIU reboots seen on 7 occasions (next chart has details) Analysis Several of these have likely causes relating to EGSE Two recent reboots have occurred without the EGSE symptoms Rate is 7 in about 1800 hours, or once per 250 hours Existing data does not give sufficient information to determine the cause Resolution Plan FSW diagnostic software to be loaded that will help diagnose the cause of the reboots by identifying which tasks are executing Records most recent 1024 task switches Can be read out in primary or secondary boot Run LAT on a non-interference basis at observatory to accumulate run hours Impacts on On-orbit performance Loss of data and LAT housekeeping telemetry until SIU is rebooted, potential for SC directed loadshed Loss of ½ the science data until EPU is rebooted NCRs and Waivers

CPU Reboots (Continued) NCR Unit Type of reboot LAT Activity Potential Cause 880 SIU redundant VxWorks reboot TkrTotGain_SVC_500hz 881 EPU2 uncorrectable memory error  Transient memory error 902 CPU exception, PPC Vector 0x300 (DSI)  During LatReinit, concurrent with main feed on command EPU2 was in primary boot and transmitting boot telemetry when script rebooted the SIU. This is a nonstandard configuration that may have contributed to the reboot 948-1 EPU1 Watchdog LPA: tackscan-6_0.55hr LCB errors at the time of the reboot indicate that the VSC was falling behind. 949-1 SIU primary e2e_LAT_22xGammafilterNoPer_0.17hr 1 pps/timetone errors from all 3 processors, and timehack table entry errors from the SIU indicate reboot likely induced due to incorrect sequence 1 PPS and timetone messages from the VSC 948-2 Checkstop  LAT-22x_0.50hr muon run   949-2 intSeAppLrs_e2e_LAT-22xGammafilterNoPer_0.50hr No clear pattern has developed NCRs and Waivers

NCR 957 EPU 0 CPU Junction Temperature Issue: After applying vacuum, EPU 0 CPU junction temperature was higher than the other CPUs by up to 28 degrees and reached 104 degrees during bakeout. Telemetry returned to normal after repressurization. Analysis: Not observed at unit test or ambient level LAT testing Possible causes Degradation of thermal connection from heat sink to CPU CPU on-chip sensor defective (unlikely) Thermal analysis of board and tests on similar hardware support first hypothesis (disconnected heat sinkn test and analysis results consistent with observed temperature rise) Under this condition, thermal and reliability margin still exceeds requirements Expected junction temperature for this box at hot is 80 degrees No potential for electrical side effects Heatsink is brazed to the rail, and is adjacent to a empty slot Epoxy is not electrically conductive Board is conformal coated Resolution Plan: Use as is Impacts on On-Orbit performance: None NCRs and Waivers

NCR 975 Datagram Time Tags Issue: Some datagrams have a time tag that is 4.2 seconds “fast” Analysis: Rate is about 1 in 10000 datagrams Likely FSW bug Analysis in progress Will reproduce on testbed Could potentially be worked around on the ground Resolution Plan: Find bug and fix it Impacts on On-orbit performance None, if fixed May use GSW mitigation until FSW error is found and fixed NCRs and Waivers

Monitor for Verification Open NCRs NCR Date Opened Affected Hardware Problem Description NCR Category Resolution Could Not Duplicate 535 6/20/05 TKR Monitor trend data for TKR 4 thru LAT testing (spin-off NCR) Known Feature Close after TV   624 8/27/05 ACD Fluctuations in temperature reading (originally an ACD PR) Monitor for Verification 625 AcdVetoHitmapPha apparent retrigger Closure near 626 ACDMonitor script high counts during T/V (originally ACD PR) 684 10/3/05 Noise Occupancy failures due to intermittent hot strips. 718 10/27/05 VETO threshold for channel 1123 can't be set below minimum value of 0.45pC (originally an ACD PR) 806 1/17/06 Trigger CAL is retriggering during SVAC run Closing 840 3/8/06 DAQ RunControl software problems observed on spare PDU during T/V test (NCR #794); opened NCR to track FSW updates B0.7.0 851 3/16/06 One or more EPU resets unexpectedly due to large number of events and interaction with end of run activities FSW Discrepancy B1.0.0 852 LCB errors observed during muon run EGSE Closure deferred 855 3/23/06 FSW LATC verify occcasionally reports errors in CRC, SPT, ARC CND QAR NCRs and Waivers

Open NCRs NCRs and Waivers NCR Date Opened Affected Hardware Problem Description NCR Category Resolution Could Not Duplicate 859 3/23/06 FSW CAL LCI data compression less than expected, results in very large datagrams FSW Discrepancy B0.6.11   880 4/12/06 Integration SIU Spontaneous Reboot CND QAR 881 EPU Spontaneous Reboot 882 DAQ GLAT2525 power up software crash due to Result FIFO not empty. Known Feature B0.7.0 884 4/15/06 VSC errors observed during LAT power on EGSE Closure deferred 894 4/29/06 LATC dump GTFE mask error in process 902 5/9/06 EPU 0 unexpected reboot 909 5/16/06 LAT06X EEPROM writecount test results misinterpreted Closing 913 5/23/06 FSW error in packing ACD charge injection data 922 6/5/06 Housekeeping telemetry stopped 932 6/20/06 TKR RS103 susceptiblity in tracker Hardware Discrepancy Waiver 933 6/22/06 EGSE overwrites timetone messages NCRs and Waivers

Open NCRs NCRs and Waivers NCR Date Opened Affected Hardware Problem Description NCR Category Resolution Could Not Duplicate 936 6/24/06 EGSE EGSE timed out on the 1553 bus Closure deferred   938 6/26/06 Housekeeping packet sampled early, causing test error 939 6/28/06 Thermal CS102 sensitivity in RTD sensors Hardware Discrepancy B0.7.0 941 Mech Temperature sensors swapped Closing 942 6/29/06 ACT to LAT blanket interface not consistent Close after TV 945 7/6/06 DAQ EGSE map of FSW files (in FMX) not properly initialized 946 7/11/06 ACD ACD FREE board 5 not responsive after power up CND QAR 948 7/19/06 EPU 1 reboot, EPU 2 reboot 949 7/21/06 Primary SIU reboot, redundant SIU reboot 957 8/4/06 EPU 0 CPU junction temperature higher than expected in vacuum 958 8/7/06 GTRC phase error induced by FPGA bug Known Feature 959 8/8/06 EGSE VSC crashes 960 Script missed an LPA stop command, resulting in test error NCRs and Waivers

Open NCRs NCRs and Waivers NCR Date Opened Affected Hardware Problem Description NCR Category Resolution Could Not Duplicate 966 8/14/06 EGSE EGSE current measurement scale incorrect Closing   967 EGSE temperature measurements inaccurate over temp Closure deferred 969 8/16/06 ACD ACD FREE HV monitors read 100V at cold Known Feature 971 8/18/06 TKR GTFE (0,-x6,20) readback intermittent at cold 975 9/1/06 DAQ Datagram timetag error FSW Discrepancy in process 976 9/2/06 Watchdog reboot at hot induces correctable memory errors 977 9/5/06 Script sampled telemetry too soon, causing test error NCRs and Waivers

LAT Waivers (1/3) CCR # Title Description Status 433-0311 DC Voltage Tolerance LAT is required to tolerate 0-40V DC. Due to MOSFET switches at power feed inputs, LAT can tolerate minimum 15V, excluding transient events. Approved 433-0356 Test Point Short Circuit Isolation LAT is required to operate within spec if any test point is shorted to ground. A shorted external clock select pin would render the redundant GASU inoperable. 433-0357 DC Voltage Tolerance #2 LAT required to tolerate 0-40V DC. After a voltage drop analysis, it was found that the TEM MOSFET switches would receive too low a voltage with the DAQ feed voltage at 15V. To operate the TEM's safely, the input voltage needs to be 18.5V minimum. 433-0358 GTFE TID LAT is required to perform TID testing on all GTFE ASIC lots. The final two lots were not tested since previous lots exhibited such large margins. 433-0360 Tracker Environmental Test With Non-Flt or Missing Cables Several tracker towers went through environmental test with a subset of missing or non-flight flex cables. The replacement flight cables were not subjected to component-level vibe and will not see twelve tvac cycles. NCRs and Waivers

LAT Waivers (2/3) CCR # Title Description Status 433-0361 24AWG STD Strength Cu High strength Cu alloy is required for 24AWG wire. LAT uses standard strength Cu wire. As reported by the LAT PCB, standard strength 24AWG wire has been used on previous NASA projects with GSFC’s approval with no compromise to product reliability. Approved 433-0362 J-STD vs NASA STD LAT circuit card assemblies uses J-STD-001 as the workmanship standard instead of NASA-STD-8739.3. 433-0367 Tracker Flex Cable and MCM Coupon Failures Several flex cables and MCM’s are installed on the LAT although they have failed coupons. 433-0368 Radiator Sine Vibe The radiators will not be installed for LAT-level sine vibe test. Instead, the radiators were subjected to alternative tests, i.e. pull test, tap test, LAT-level acoustic test. 433-0369 EMI Skirt Stay Clear Center EMI skirt pieces near SC-LAT flexures exceed the LAT stay-clear by 0.015” max. 433-0374 VCHP CECM The VCHP feed violates the CECM requirement. The measured value is ~700mVp-p vs the requirement of 200mVp-p. NCRs and Waivers

LAT Waivers (3/3) CCR # Title Description Status 446-0402 Keyed Test Point Connectors Test point connectors are not keyed. Approved 446-0403 Segregate Signal Types on Test Connectors JL-39 contains signals of different signal classes. 446-0XX RS103 TKR Noise Occupancy Tracker noise occupancy exceeded its limit of 10-4 during RS 103 (vertical polarization) between 95-101MHz. Submitted to GSFC Cable Shield Termination Floating shields are not properly terminated per NASA-STD- 8739.4. NCRs and Waivers

SC-LAT ICD Waivers ICN # Title Description Status -095 LAT Grid Interface Hole Out-of-Tolerance Several grid interface hole locations are out of tolerance. Using the as-built LAT Grid and SC interface hole locations, the analysis shows the predicted forces to align the shear pins are small and a minimum of 0.007” exist between the bolts and holes in the flexures and mating should not be an issue. Approved -107 Recessed Grid Bushings The +Y and –Y LAT grid interface hole bushings are recessed by 0.022” worst case. Stress analysis at the SC mount interface shows the margins of safety for ultimate and yield bearing strength is 7% which is acceptable. The margin of safety for pin bending is >200%. NCRs and Waivers

NCR and Waiver Summary The LAT Team is confident that none of the NCRs or Waivers presented are significant enough to prevent the LAT from shipping. The LAT Team recommends proceeding with the LAT shipping as planned. NCRs and Waivers