1. FPGA-Based Radiation Tolerant Computing
Dr. Brock J. LaMeres
Associate Professor
Electrical & Computer Engineering
Montana State University

2. Outline Overview of Montana State University Research Statement
Vitals, Highlights of Interest
Research Statement
Enabling Reconfigurable Computing for Aerospace
Radiation Effects in Electronics
Sources & Types of Radiation
Effects (TID, SEE, Displacement Damage)
FPGA Specific Effects
Existing Mitigation Techniques
Physical (Shielding, RHBD, RHBP)
Architectural (TMR, Scrubbing, Error Correction Codes)
Our Approach
Redundant Tiles (TMR+Spares+Scubbing) with Sensor
Test Results (TAMU Cyclotron, Borealis Flights, HASP Flights)
Future Flights (Sounding Rocket)

3. Overview of MSU Public, Land Grant Institution
Located in Bozeman, MT (pop. ~40k)
14,600 students
70% in state, 87% undergrad
Ranked in top tier of Carnegie Foundation’s Research Universities with “Very High Research Activity” (~$115M)
Academically known for:
Engineering, Agriculture, Nursing, Architecture

4. Overview of MSU What we’re really known for: FCS Football
A Mountain Town
Fishing
Skiing
Hiking
Yellowstone Park
FCS Football
Currently ranked #3
Cat-Griz Rivalry 31st oldest in country (started in 1897)

5. Overview of ECE Electrical & Computer Engineering at MSU
13 full time tenure/tenure-track + 3 research faculty
~330 students (300 undergrad + 30 graduate)
BS Degrees in EE and CpE
MS in EE
Ph.D. in Eng with EE Option
Heavy Emphasis on Hands-On Education
Flying Experiments on Local High Altitude Balloon Platform
Participating in RockOn Sounding Rocket Workshop
AUVSI RoboSub Design Team
Participating in NASA Lunabotics Mining Competition

6. Micro/Nano Fabrication
ECE Research Areas
Digital Systems
Reconfigurable Computing
Fault Tolerant Architectures
Micro/Nano Fabrication
MEMS Structures
Deformable Mirrors
Nano Optics
Optics
Remote Sensing
Climate Monitoring
Power & Energy
Wind / PV
Fuel Cells
System Modeling
Signals & Controls
Acoustics
System Identification
Communications
Wireless Networks for Rural Regions
Smart Antenna Systems

7. MSU Facilities of Interest to My Work
Montana Microfabrication Facility
Class 1000 Clean Room
Low volume, high mix technology
Research, Education and Local Business
Space Science & Engineering Laboratory
Student Fabricated Small-Sat Program
Launched 1st CubeSat in October 2011
Measuring radiation levels in Van Allen Belts
Has been sending data to MSU for over a year
Over 5500 orbits as of this week
Subzero Science and Engineering Laboratory
2700 ft2 facility with 8 walk in cold rooms
Used to study effect of cold on projects across many scientific disciplines
Temperatures down to -60 oC

8. Space Launch System (SLS)
Research Statement
Support the Computing Needs of Space Exploration & Science
Computation
Power Efficiency
Mass
Space Launch System (SLS)

9. Research Statement cont…
Provide a Radiation Tolerant Platform for Reconfigurable Computing
Reconfigurable Computing as a means to provide:
Increased Computation of Flights Systems
Reduced Power of Flight Systems
Reduced Mass of Flight Hardware
Mission Flexibility through Real-Time Hardware Updates
Support FPGA-based Reconfigurable Computing through an underlying architecture with inherent radiation tolerance to Single Event Effects
The Future
The Problem

10. What is Reconfigurable Computing?
Let’s start with what is NOT Reconfigurable Computing
A CPU/GPU – while you have flexibility via programming, the hardware is still fixed
The instructions that can be executed are fixed in the sequence controller
The size of memory is pre-defined
The IO is pre-defined
An ASIC – the hardware is fixed during fabrication.
Are There Advantages to these Conventional Systems?
Yes, they are well understood and easy to program (particularly the single core model)
Yes, when the task maps well to the hardware, they have high performance (e.g., GPU)
Yes, they can handle a large array of tasks (albeit sometimes in a inefficient manner)
Are There Disadvantages to these Conventional Systems?
Yes, unless the task does not map directly to the hardware, they perform poorly.
Yes, much of the hardware that allows them to handle a variety of tasks sits idle most of the time.

11. What is Reconfigurable Computing?
A System That Alters Its Hardware as a Normal Operating Procedure
This can be done in real-time or at compile time.
This can be done on the full-chip, or just on certain portions.
Changing the hardware allows it to be optimized for the application at hand.

12. What Technology is used for RC?
Field Programmable Gate Arrays (FPGA)
Currently the most attractive option.
SRAM-based FPGAs give the most flexibility
Riding Moore’s Law feature shrinkage

13. What are the Advantages of RC?
Computational Performance
Optimizing the hardware for the task-at hand = architectural advantages
Eliminating unused circuitry (minimize place/route area, reduces wiring delay)
Reduced Power
Implement only the required circuitry
Shutdown or un-program unused circuitry when not in use
Reduced Mass
Reuse a common platform to conduct multiple sequential tasks in flight systems
This effect is compounded when considering each flight system has backup hardware
Mass is the dominant driver of cost for space applications
$10,000/lb to get into orbit.
NASA’s goal is $100/lb by 2025
Shuttle cost ~$300-$500M per launch with 50,000 lb capacity
A Sequence of Unique Tasks

14. Radiation Effects on Electronics
On Earth Our Computers are Protected
Our magnetic field deflects the majority of the radiation
Our atmosphere attenuates the radiation that gets through our magnetic field
Our Satellites Operate In Trapped Radiation in the Van Allen Belts
High flux of trapped electrons and protons
In Deep Space, Nothing is Protected
Radiation from our sun
Radiation from other stars
Radiation from the Big Bang
Particles & electromagnetic
You Are Here

15. Radiation Effects on Electronics
Radiation Categories
Ionizing Radiation
Sufficient energy to remove electrons from atomic orbit
Ex. High energy photons, charged particles
Non-Ionizing Radiation
Insufficient energy/charge to remove electrons from atomic orbit
Ex., microwaves, radio waves
Types of Ionizing Radiation
Gamma & X-Rays (photons)
Sufficient energy in the high end of the UV spectrum
Charged Particles
Electrons, positrons, protons, alpha, beta, heavy ions
Neutrons
No electrical charge but ionize indirectly through collisions
What Type are Electronics Sensitive To?
Ionization which causes electrons to be displaced
Particles which collide and displace silicon crystal

16. Radiation Effects on Electronics
Where Does Space Radiation Come From?
Nuclear fusion in stars creates light and heavy ions + EM
Stars consists of an abundant amount of Hydrogen (1H = 1 Proton) at high temperatures held in place by gravity
The strong nuclear force pulls two Hydrogen (1H) atoms together overcoming the Columns force and fuses them into a new nucleus
The new nucleus contains 1 proton + 1 neutron
This new nucleus is called Deuterium (D) or Heavy Hydrogen (2H)
Energy is given off during this reaction in the form of a Positron and a Neutrino
The Deuterium (2H) then fuses with Hydrogen (1H) again to form yet another new nucleus
This new nucleus contains 2 protons + 1 neutron
This nucleus is called Tritium or Hydrogen-3 (3H)
Energy is given off during this reaction in the form of a Gamma Ray
Two Tritium nuclei then fuse to form a Helium nucleus
The new Helium nucleus (4H) contains 2 protons + 2 neutrons
Energy is given off in the form of Hydrogen (e.g., protons)

17. Radiation Effects on Electronics
Classes of Ionizing Space Radiation
1) Cosmic Rays
2) Solar Particle Events
3) Trapped Radiation

18. Radiation Effects on Electronics
Classes of Ionizing Space Radiation
Cosmic Rays
Originating for our sun (Solar Wind) and outside our solar system (Galactic)
Mainly Protons and heavier ions
Low flux
Solar Particle Events
Solar flares & Coronal Mass Ejections
Electrons, protons, alpha, and heavier ions
Event activity tracks solar min/max 11 year cycle
Trapped Radiation
Earth’s Magnetic Field traps charged particles
Inner Van Allen Belt holds mainly protons (10-100’s of MeV)
Outer Van Allen Belt holds mainly electrons (up to ~7 MeV)
Heavy ions also get trapped

19. Radiation Effects on Electronics
Which radiation is of most concern to electronics? Concern
Protons (1H)
Makes up ~85% of galactic radiation
Larger Mass than electron (1800x), harder to deflect
Beta Particles (electrons & positrons)
Makes up ~1% of galactic space
More penetrating than alphas
Heavy Ions
Makes up <1% of galactic radiation
High energy (up to GeV) so shielding is inefficient
Neutrons
Uncharged so difficult to stop
No Concern
Alpha Particles (He nuclei)
Makes up ~14% of galactic radiation
~ 5MeV energy level & highly ionizing but…
Low penetrating power (50mm in air, 23um in silicon)
Can be stopped by a sheet of paper
Gamma
Highly penetrating but an EM wave
Lightly ionizing

20. Radiation Effects on Electronics
What are the Effects?
Total Ionizing Dose (TID)
Cumulative long term damage due to ionization.
Primarily due to low energy protons and electrons due to higher, more constant flux, particularly when trapped
Problem #1 – Oxide Breakdown
Threshold Shifts
Leakage Current
Timing Changes

21. Radiation Effects on Electronics
What are the Effects?
Total Ionizing Dose (TID) Cont…
Problem #2 –Leakage Current
Hole trapping slowly “dopes” field oxides to become conductive
This is the dominant failure mechanism for commercial processes

22. Radiation Effects on Electronics
What are the Effects?
2. Single Event Effects (SEE)
Electron/hole pairs created by a single particle passing through semiconductor
Primarily due to heavy ions and high energy protons
Excess charge carriers cause current pulses
Creates a variety of destructive and non-destructive damage “Critical Charge” = the amount of charge deposited to change the state of a gate

23. Radiation Effects on Electronics
What are the Effects?
2. Single Event Effects (SEE) - Non-Destructive (e.g., soft faults)
Single Event Transients (SET)
An induced pulse that can flip a gate
Temporary glitches in combinational logic
Single Event Upsets (SEU)
The pulse is captured by a storage device resulting in a state change

24. Radiation Effects on Electronics
What are the Effects?
2. Single Event Effects (SEE) - Non-Destructive (e.g., soft faults)
Multiple Bit Upsets (MBU)
A single particle causes multiple SEUs due to its path being non-orthogonal
Single Event Functional Interrupt (SEFI)
The system is put into a state that causes function failure - Typically requires reset, power cycle or reprogramming

25. Radiation Effects on Electronics
What are the Effects?
2. Single Event Effects (SEE) – Destructive (e.g., hard faults)
Single Event Latchup (SEL)
Parasitic NPN/PNP transistors are put into positive feedback condition (PNPN).
Runaway current damages device
Due to heavy ions, protons, neutrons.
Single Event Burnout (SEB)
Localized current in body of device turns on parasitic bipolar transistors.
Runaway current causes heat.
Due primarily to heavy ions.
Single Event Gate Rupture (SEGR)
Permanent short through gate oxide due to hole trapping. - Thin oxides are especially immune.
Due to heavy ions.

26. Radiation Effects on Electronics
What are the Effects?
3. Displacement Damage
Cumulative long term damage to protons, electrons, and neutrons
Not an ionizing effect but rather collision damage
Minority Carrier Degradation
Reduced gain & switching speed
Particularly damaging for optoelectronic & linear circuits

27. Current Mitigation Techniques
Shielding
Shielding helps for protons and electrons <30MeV, but has diminishing returns after 0.25”.
This shielding is typically inherent in the satellite/spacecraft design.
Shield Thickness vs. Dose Rate (LEO)

28. Current Mitigation Techniques
Radiation Hardened by Design (RHBD)
Uses commercial fabrication process
Circuit layout techniques are implemented which help mitigate effects
Enclosed Layout Transistors
Eliminates edge of drain terminal
This eliminates any leakage current between source & drain due near edge of gate (STI Region 1)
Guard Rings
Reduces leakage between NMOS & PMOS devices due to hole trapping in Field Oxide (STI Region 2) - Separation of device + body contacts - Adds ~20% increase in area
Thin Gate Oxide
This oxide reduces probability of hold trapping. - Process nodes <0.5um typically are immune to Vgs shift in the gate.

29. Current Mitigation Techniques
Radiation Hardened by Process (RHBP)
An insulating layer is used beneath the channels
This significantly reduces the ion trail length and in turn the electron/hole pairs created
The bulk can also be doped to be more conductive so as to resist hole trapping

30. Current Mitigation Techniques
Architectural
Triple Module Redundancy
Triplicate each circuit
Use a majority voter to produces output
Advantages
Able to address faults in real-time
Simple
Disadvantages
Takes >3x the area
Voter needs to be triplicated also to avoid single-point-of-failure
Doesn’t handle Multiple-Bit-Upsets

31. Current Mitigation Techniques
Architectural Cont…
2. Scrubbing
Compare contents of a memory device to a “Golden Copy”
Golden Copy is contained in a radiation immune technology (fuse-based memory, MROM, etc…)
Advantages
Simple & Effective
Disadvantages
Sequential searching pattern can have latency between fault & repair

32. Current Mitigation Techniques
Architectural Cont…
3. Error Correction Codes
A variety of error detection & correction codes exist (CRC, Hamming, Parity…)
CRCs can be included as part of memory system
Advantages
Quick Handling of Errors
Disadvantages
Faults in a configuration memory can alter circuitry momentarily before error codes performs correction
Does not handle Multiple-Bit Upsets

33. Current Mitigation Techniques
Effects Overview
Primary Concern is Heavy Ions & high energy protons
All computer electronics experience TID and will eventually go out
Heavy Ions cannot be stopped and an architectural approach is used to handle them.

34. Our Approach FPGAs are Uniquely Susceptible Total Ionizing Dose
All gates and memory cells are susceptible to TID due to high energy protons
2. Single Event Effects
SETs/SEUs in the logic blocks
SETs in the routing
SEUs in the configuration memory for the logic blocks (SEFI)
SEUs in the configuration memory for the routing (SEFI)
Radiation Strikes in the Circuit Fabric (Logic + Routing)
Radiation Strikes in the Configuration Memory (Logic + Routing)

35. Our Approach What is needed for FPGA-Based Reconfigurable Computing
SRAM-based FPGAs
To support fast reconfiguration
2. A TID hardened fabric
Thin Gate Oxides to avoid hole trapping and threshold shifting (inherent in all processes)
Radiation Hardened by Design to provide SEL immunity (rings, layout, etc…)
Does This Exist?
Yes, Xilinx Virtex-QV Space Grade FPGA Family
TID Immunity > 1Mrad
RHBD for SEL immunity
CRC
The Final Piece is SEE Fault Mitigation due to Heavy Ions
SEU will happen due to heavy ions, nothing can stop this.
A computer architecture that expects and response to faults is needed.

36. 16 MicroBlaze Soft Processors on an Virtex-6
Our Approach
A Many-Tile Architecture
The FPGA is divided up into Tiles
A Tile is a quantum of resources that:
Fully contains a system (e.g., processor, accelerator)
Can be programmed via partial reconfiguration (PR)
Fault Tolerance
TMR + Spares
Spatial Avoidance of Background Repair
Scrubbing
16 MicroBlaze Soft Processors on an Virtex-6

37. Shuttle Flight Computer (TMR + Spare)
Our Approach
TMR + Spares
3 Tiles run in TMR with the rest reserved as spares.
In the event of a fault, the damaged tile is replaced with a spare and foreground operation continues.
Spatial Avoidance & Repair
The damaged Tile is “repaired” in the background via Partial Reconfiguration.
The repaired tile is reintroduced into the system as an available spare.
3. Scrubbing
A traditional scrubber runs in the background.
Either blind or read-back.
PR is technically a “blind scrub”, but of a particular region of the FPGA.
Shuttle Flight Computer (TMR + Spare)

38. Our Approach Why do it this way?
With Spares, it basically becomes a flow-problem:
If the repair rate is faster than the incoming fault rate, you’re safe.
If the repair rate is slightly slower than the incoming fault rate, spares give you additional time.
The additional time can accommodate varying flux rates.
Abundant resources on an FPGA enable dynamic scaling of the number of spares. (e.g., build a bigger tub in real time)

39. Our Approach Practical Considerations
Foreground operation can continue while repair is conducted in the background. Since scrubbing/PR is typically slower than reinitializing a tile, foreground “down time” is minimized.
Using PR tiles, the system doesn’t need to track the exact configuration memory addresses. Partial bit streams contain all the necessary information about a tile configuration.
PR of a tile also takes care of both SEUs in the circuit fabric & configuration SRAM so the system doesn’t care which one occurred.
The “spares” are held in reset to reduce power. This is as opposed to running in N-MR with everyone voting.

40. Our Approach Modeling Our Approach
We need to compare our approach to a traditional TMR+scrubbing system
We use a Markov Model to predict Mean-Time-Before-Failure
16 tile MicroBlaze system on Virtex-6 (3+13)
 is fault rate
μ is repair rate

41. Our Approach Modeling Our Approach: Fault & Repair Rates
Fault Rate ()
Derived from CREME96 tool for 4 different orbits
Used LET fault data from V4
Repair Rate (μ)
Measured empirically in lab on V6 system

42. Our Approach Modeling Our Approach: Results Baseline System
Proposed System
Improvement
Ok, it looks promising…

43. Our Approach Let’s Build It
Xilinx Evaluation Platforms (Virtex 4/5/6) for Lab Testing
Custom Virtex-6 platform for Flight Testing

44. Our Approach Let’s Test It Local Balloon Flights (MSGC Borealis)
HASP Program
Suborbital Vehicle
4 Flights in MT to 100k ft in 2011/12
Thermal evaluation of form-factor
1st test flight in Sept-12
2nd test flight planned Sept-13
Payload design training (June-12)
Flight planned 2013

45. Conclusion What is Missing What’s Next Faults in the routing MBUs
Addressing Single-Point-of-Failure
What’s Next
Collect flight data
Address above mentioned issues

46. Questions?

47. References Content Images
“Space Transportation Costs: Trends in Price Per Pound to Orbit Fultron Inc Technical Report., September 6, Sammy Kayali, “Space Radiation Effects on Microelectronics”, JPL, [Available Online]:
Holmes-Siedle & Adams, “Handbook of Radiation Effects”, 2nd Edition, Oxford Press 2002.
Thanh, Balk, “Elimination and Generation of Si-Si02 Interface Traps by Low Temperature Hydrogen Annealing”, Journal of Electrochemical Society on Solid-State Science and Technology, July 1998.
Sturesson TEC-QEC, “Space Radiation and its Effects on EEE Components”, EPFL Space Center, June 9, [Available Online]:
Lawrence T. Clark, Radiation Effects in SRAM: Design for Mitigation”, Arizona State University, [Available Online]:
K. Iniewski, “Radiation Effects in Semiconductors”, CRC Press, 2011.
Images
If not noted, images provided by or MSU
Displacement Image 1: Moises Pinada,
Displacement Image 2/3: Vacancy and divacancy (V-V) in a bubble raft. Source: University of Wisconsin-Madison
SRAM Images: Kang and Leblebici, "CMOS Digital Integrated Circuits" 3rd Edition. McGraw Hill, 2003
SEB Images: Sturesson TEC-QEC, “Space Radiation and its Effects on EEE Components”, EPFL Space Center, June 9, 2009.
FPGA Images:
RHBD Images: Giovanni Anelli & Alessandro Marchioro, “The future of rad-tol electronics for HEP”, CERN, Experimental Physics Division, Microelectronics Group, [Available Online]:

48. Misc Slides
Misc Slides

49. Our Approach On-Chip Network
PR Tiles only contain information about sub-system resources.
Routing area & complexity becomes significant in large designs.
FPGA routing is highly susceptible to faults.
Routing consists of programmable switches (multiplexers)
The routing setup is contained in the configuration memory
Faults occurring in configuration memory (SEFI) cause damage PR can’t handle.
Voting on the routing becomes impractical due to the number of nets.
Why do it this way?
Reduce routing complexity compared to verbose approach
Errors can be handled using standard networking error checking