1. 1 Fault Tolerant FPGA Co-processing Toolkit Oral defense in partial fulfillment of the requirements for the degree of Master of Science 2006 Oral defense in partial fulfillment of the requirements for the degree of Master of Science 2006 Douglas Michael DiSabello

2. 2 Overview Computation in radiation space environments is slow compared to terrestrial computation FPGAs can fill and exceed this gap for specific applications The Fault Tolerant FPGA Co-processing Toolkit facilitates a general capability of FPGA co- processing capability for space based applications Computation in radiation space environments is slow compared to terrestrial computation FPGAs can fill and exceed this gap for specific applications The Fault Tolerant FPGA Co-processing Toolkit facilitates a general capability of FPGA co- processing capability for space based applications

3. 3 Outline The Space Computation Performance Gap Ionizing Radiation and FPGA Background Development Board Fault Tolerant Co-Processing Toolkit Support System Fault Tolerant Support System End User Environment The Space Computation Performance Gap Ionizing Radiation and FPGA Background Development Board Fault Tolerant Co-Processing Toolkit Support System Fault Tolerant Support System End User Environment

4. 4 The Space Computation Performance Gap Radiation Hardened Microprocessor BAE Systems 133MHz PowerPC Special and proprietary design techniques Laptop running this presentation 1,500 MHz PowerPC Radiation Hardened Microprocessor BAE Systems 133MHz PowerPC Special and proprietary design techniques Laptop running this presentation 1,500 MHz PowerPC

5. 5 Computation Gap Problems Bare minimum of computations performed in space Large and slow data transfers to limited number of receiving stations Bare minimum of computations performed in space Large and slow data transfers to limited number of receiving stations

6. 6 Field Programmable Gate Arrays 200 – 1600 fold speed ups Exploits fine grain parallelism of algorithms Speed up computations in space beyond what can even be offered by normal terrestrial computers Fast development cycles In flight reprogramming to adapt to changing mission requirements 200 – 1600 fold speed ups Exploits fine grain parallelism of algorithms Speed up computations in space beyond what can even be offered by normal terrestrial computers Fast development cycles In flight reprogramming to adapt to changing mission requirements

7. 7 FPGAs Details SRAM Based Configuration memory stores and implements design Programmed using JTAG or SelectMap interfaces Configuration Logic Blocks Look Up Tables and supporting logic Input/Output Blocks Used for all general I/O package pins Block RAMs General Routing Matrix SRAM Based Configuration memory stores and implements design Programmed using JTAG or SelectMap interfaces Configuration Logic Blocks Look Up Tables and supporting logic Input/Output Blocks Used for all general I/O package pins Block RAMs General Routing Matrix

8. 8 Ionizing Radiation Low Earth Orbit contains ionizing particles trapped in the Van Allen Belts Particles cause direct and secondary nuclear reactions in silicon substrate Decreasing feature sizes and shrinking threshold voltages increase probability of these interactions causing errors in circuits Low Earth Orbit contains ionizing particles trapped in the Van Allen Belts Particles cause direct and secondary nuclear reactions in silicon substrate Decreasing feature sizes and shrinking threshold voltages increase probability of these interactions causing errors in circuits

9. 9 Ionizing Radiation Definitions Single Event Transient Temporary change in logic value Single Event Upset SET that is latched into a memory Single Event Functional Interrupt Component stops service Single Event Transient Temporary change in logic value Single Event Upset SET that is latched into a memory Single Event Functional Interrupt Component stops service

10. 10 Ionizing Radiation and FPGAs Architecture SEUs in configuration memory Instantiated design changes Usually results in a SEFI Data SEUs in Flip Flops, Latches, BRAMS, etc… Incorrect computation results and/or SEFIs Off Chip Communication Non-dedicated configurable input/outputs package pins SEUs to input/output blocks can disable a package pin Architecture SEUs in configuration memory Instantiated design changes Usually results in a SEFI Data SEUs in Flip Flops, Latches, BRAMS, etc… Incorrect computation results and/or SEFIs Off Chip Communication Non-dedicated configurable input/outputs package pins SEUs to input/output blocks can disable a package pin

11. 11 Typical FPGA Cross Section Configuration Memory accounts for 91% of a typical FPGA cross section 78% - 84.8% Routing Structure 20% Control bits and CLB LUT values Configuration Memory accounts for 91% of a typical FPGA cross section 78% - 84.8% Routing Structure 20% Control bits and CLB LUT values Michael Affrey, Paul Graham, Eric Johnson, Michael Wirthlin, Nathan Rollins, and Carl Carmichael, “Single-Event Upsets in SRAM FPGAs” MAPLD, Sep. 2002

12. 12 Fault Mitigation Techniques Scrubbing fixes architectural upsets Continuously rewrite static portions of configuration memory Active partial reconfiguration bitstream SEUs corrected at given reconfiguration rate (shorter than expected upsets rate for given orbit) Scrubbing fixes architectural upsets Continuously rewrite static portions of configuration memory Active partial reconfiguration bitstream SEUs corrected at given reconfiguration rate (shorter than expected upsets rate for given orbit)

13. 13 Fault Mitigation Techniques Triple Modular Redundancy Allows continuous service when architectural upsets occur Majority Voters determine final output Inherent data redundancy Triple Modular Redundancy Allows continuous service when architectural upsets occur Majority Voters determine final output Inherent data redundancy

14. 14 Hardware Development Board Design developed by Naval Post Graduate School Naval Research Laboratory modified with Virtex II FPGA Designed for Configurable Fault Tolerant Computing Design developed by Naval Post Graduate School Naval Research Laboratory modified with Virtex II FPGA Designed for Configurable Fault Tolerant Computing

15. 15 Hardware Development Board General I/O SelectMap PC104/ISA Bus JTAG FLASHFLASH EEPROMEEPROM SelectMap Xilinx Virtex FPGA Xilinx Virtex II FPGA Embedded X86 PC Support FPGA Co-processing FPGA Only off-chip memory

16. 16 Development Board Radiation Testing NPS and NRL conducted tests at the Crocker Nuclear Lab, U.C. Davis Protons were emitted from a cyclotron to interact with the FPGAs Both the Virtex and Virtex II were irradiated NPS and NRL conducted tests at the Crocker Nuclear Lab, U.C. Davis Protons were emitted from a cyclotron to interact with the FPGAs Both the Virtex and Virtex II were irradiated

17. 17 Development Board Radiation Testing Results: 1 upset for every 5 days for the Virtex in orbit Order of magnitude greater for Virtex II in orbit Scrubbing was proven to repair configuration memory upsets TMR was proven to allow continuous service between SEU scrubbing intervals Results: 1 upset for every 5 days for the Virtex in orbit Order of magnitude greater for Virtex II in orbit Scrubbing was proven to repair configuration memory upsets TMR was proven to allow continuous service between SEU scrubbing intervals James C. Coudeyras, “Radiation Testing of The Configurable Fault Tolerant Processor (CFTP) for Space-Based Applications,” Thesis, United States Naval Post Graduate School, 2005

18. 18 The Toolkit Objective Build a suite of VHDL designs, C++ software, and tools to give a general FPGA co-processing capability Modular design for easy integration into new hardware platforms and with new HDL modules Allow designers to concentrate mainly on the co- processing algorithm and design instead of Fault Tolerance Build a suite of VHDL designs, C++ software, and tools to give a general FPGA co-processing capability Modular design for easy integration into new hardware platforms and with new HDL modules Allow designers to concentrate mainly on the co- processing algorithm and design instead of Fault Tolerance

19. 19 How Objective was Achieved 1. Create a support system to interface between all components 2. Modify the support system into a Fault Tolerant version 3. Co-processing designer templates and interfaces 1. Create a support system to interface between all components 2. Modify the support system into a Fault Tolerant version 3. Co-processing designer templates and interfaces

20. 20 Support System Interpret and execute all commands from a host embedded computer Route data between all components on the board Program FPGAs Software for embedded x86 to interact with support system instantiated into an FPGA Interpret and execute all commands from a host embedded computer Route data between all components on the board Program FPGAs Software for embedded x86 to interact with support system instantiated into an FPGA

21. 21 FLASH INTERFACE Virtex SelectMap Interface Virtex II SelectMap Interface Flash Arbitrator PC104/ISA Bus Interfaces FLASH Control Interface Inter-FPGA Communication Interface Support System

22. 22 PC104 / ISA Bus Interface Responsible for all data transfers on and off the FPGA board Two addresses from the host PC are used: Data address 8 Bit data words Control address A write causes the interface to reset and send a reset to any other modules A read gives the status of the buffer FIFOs Each main support module use a copy of this interface Responsible for all data transfers on and off the FPGA board Two addresses from the host PC are used: Data address 8 Bit data words Control address A write causes the interface to reset and send a reset to any other modules A read gives the status of the buffer FIFOs Each main support module use a copy of this interface

23. 23 Input FIFO Output FIFO Bus Control Logic Bus Data EMPTY READ ENABLE FULL Write Enable Read Enable EMPTY FULL WRITE ENABLE ADDRESS Bus Data BUS DATA BUS WRITE BUS READ AEN Data Out Data In PC104 / ISA Bus Interface

24. 24 Flash Components Interface Translates commands and data into a series of signals to interact with a Flash chip Control Facilitates interaction between the flash interface and the Bus Interface Arbitrator Each module that requires Flash access is given a priority number A modules must relinquish control before another module can be given access Interface Translates commands and data into a series of signals to interact with a Flash chip Control Facilitates interaction between the flash interface and the Bus Interface Arbitrator Each module that requires Flash access is given a priority number A modules must relinquish control before another module can be given access

25. 25 SelectMap Interfaces Control configuration data flow to the FPGAs configuration interfaces Configuration Clock is used to allow non-uniform data loading Actual FPGA configuration commands are contained in the configuration data Two versions: Virtex and Virtex II (and Virtex 4) Control configuration data flow to the FPGAs configuration interfaces Configuration Clock is used to allow non-uniform data loading Actual FPGA configuration commands are contained in the configuration data Two versions: Virtex and Virtex II (and Virtex 4) SelectMap Interface Flash Address Flash Interface CMD Flash Data Flash Data Valid Flash Interface Busy Flash Control Request Current Flash Owner SelectMap Data Out Write Chip Select Configuration Clock

26. 26 SelectMap Interface Commands Load Start Address Load Stop Address Program Using Flash Load Bus Word Number Program Using Bus Scrub using Flash Abort (Virtex version only) Load Start Address Load Stop Address Program Using Flash Load Bus Word Number Program Using Bus Scrub using Flash Abort (Virtex version only)

27. 27 Inter-FPGA Communication Interface Modified ISA Bus interface for delays between physical FPGA chips Co-processing FPGA has direct access to BUS and own memory space Designed specifically to use resources of support FPGA Modified ISA Bus interface for delays between physical FPGA chips Co-processing FPGA has direct access to BUS and own memory space Designed specifically to use resources of support FPGA

28. 28 x86 Host PC Programs Flash program Flash verify SelectMap Configuration Scrub On and Scrub Off Co-processing echo check program Flash program Flash verify SelectMap Configuration Scrub On and Scrub Off Co-processing echo check program

29. 29 Fault Tolerant Support System Add fault tolerance to original support system Specialized fault mitigation techniques FPGA configuration, scrubbing, and BitStream manipulation Support System End User Environment Add fault tolerance to original support system Specialized fault mitigation techniques FPGA configuration, scrubbing, and BitStream manipulation Support System End User Environment

30. 30 Support System Main Fault Tolerance Methods Triple Modular Redundancy All modules are made in triplicate Majority voter determines correct output Between HDL modules three voters are used to keep redundancy of signal paths Place and Route to keep redundant modules separate Triple Modular Redundancy All modules are made in triplicate Majority voter determines correct output Between HDL modules three voters are used to keep redundancy of signal paths Place and Route to keep redundant modules separate

31. 31 Majority Voters Tri-State Buffers Not made from SRAM material Only interconnects are susceptible and correct operation still results Takes multiple SEUs for incorrect function Tri-State Buffers Not made from SRAM material Only interconnects are susceptible and correct operation still results Takes multiple SEUs for incorrect function Carl Carmichael, “Triple Modular Redundancy Design Techniques for Virtex FPGAs,” Xilinx Application Note 197, 2001

32. 32 FSMs and TMR Method keeps FSM synchronized if SEU occurs in state register Outputs are also majority voted Method keeps FSM synchronized if SEU occurs in state register Outputs are also majority voted Current State Register Next State Logic Next State Logic Next State Logic INPUTS V V V

33. 33 Block RAM Fault Mitigation TMR Necessary for routing structure Inherent data redundancy Method is fine for short term data storage SEUs are not corrected during long term data storage in individual BRAM TMR Necessary for routing structure Inherent data redundancy Method is fine for short term data storage SEUs are not corrected during long term data storage in individual BRAM

34. 34 BRAM TMR w/ Refresh All BRAMs are dual ported Second ports are used to constantly read data values from three copies, vote, and rewrite the values Data write collision avoidance Not needed for support system, but useful for co-processing applications All BRAMs are dual ported Second ports are used to constantly read data values from three copies, vote, and rewrite the values Data write collision avoidance Not needed for support system, but useful for co-processing applications Carl Carmichael, “Triple Modular Redundancy Design Techniques for Virtex FPGAs,” Xilinx Application Note 197, 2001

35. 35 Specialized Techniques Off FPGA transfers TMR of package pins to a single trace TMR of package pins not available on development board Off FPGA transfers TMR of package pins to a single trace TMR of package pins not available on development board Carl Carmichael, “Triple Modular Redundancy Design Techniques for Virtex FPGAs,” Xilinx Application Note 197, 2001

36. 36 Specialized Techniques Inter-FPGA transfers Not enough pins available to triplicate all signals 8 data bit and 4 redundant bit Hamming code used for data Double Error Detection and Single Error Correction Triplication of all other signals Inter-FPGA transfers Not enough pins available to triplicate all signals 8 data bit and 4 redundant bit Hamming code used for data Double Error Detection and Single Error Correction Triplication of all other signals

37. 37 Specialized Techniques PC104 / ISA Bus transfers Updated fault tolerant module allows for extended data transfer sizes All data sent is encoded in (8,4) hamming code No redundancy available for other signals PC104 / ISA Bus transfers Updated fault tolerant module allows for extended data transfer sizes All data sent is encoded in (8,4) hamming code No redundancy available for other signals

38. 38 Flash Memory Data Reads CRC values embedded every 512 16-bit flash words by Flash program Fault tolerant Flash interface uses BRAM buffer cache to hold every 512 blocks of data when data is requested If CRC value is incorrect data cache is flushed and the data is read again CRC values embedded every 512 16-bit flash words by Flash program Fault tolerant Flash interface uses BRAM buffer cache to hold every 512 blocks of data when data is requested If CRC value is incorrect data cache is flushed and the data is read again

39. 39 Configuration and Scrubbing Three modes of configuration determined by the bit file created using Xilinx Bitgen tool Initial Configuration Contains startup commands Reconfiguration Contains shutdown and startup commands Doesn’t require a powercycle, but FPGA is taken out of service Active Partial Reconfiguration Used for scrubbing Rewrites static portions of bitstream Removes initial BRAM contents Any portion of design could be masked out Three modes of configuration determined by the bit file created using Xilinx Bitgen tool Initial Configuration Contains startup commands Reconfiguration Contains shutdown and startup commands Doesn’t require a powercycle, but FPGA is taken out of service Active Partial Reconfiguration Used for scrubbing Rewrites static portions of bitstream Removes initial BRAM contents Any portion of design could be masked out

40. 40 Support System End User Environment Flash program and verify host programs Store initial co-processing, co-processing scrubbing, and support scrubbing bitstreams at designated flash addresses Scrub On and Off Selectmap interfaces begin scrubbing at given periods using flash data configuration can also still be done across the bus C++ examples programs and headers files for data transfers to the co-processing FPGA Flash program and verify host programs Store initial co-processing, co-processing scrubbing, and support scrubbing bitstreams at designated flash addresses Scrub On and Off Selectmap interfaces begin scrubbing at given periods using flash data configuration can also still be done across the bus C++ examples programs and headers files for data transfers to the co-processing FPGA

41. 41 Toolkit User Environment Fault Tolerant Support System Controls all data flow Modular design for quick integration to different physical systems with varying number of FPGAs C++ programs and header files Co-Processing Tools Templates for HDL co-processing components HDL “hooks” for data transfers C++ programs and header files Fault Tolerant Support System Controls all data flow Modular design for quick integration to different physical systems with varying number of FPGAs C++ programs and header files Co-Processing Tools Templates for HDL co-processing components HDL “hooks” for data transfers C++ programs and header files

42. 42 Future Toolkit Additions Different bus modules for interaction with radiation hardened microprocessors Co-processing library Use support system on single and multiple FPGA boards Different bus modules for interaction with radiation hardened microprocessors Co-processing library Use support system on single and multiple FPGA boards