1. A Dynamic Analog Concurrently-Processed Adaptive Chip Malcolm Stagg Grade 11

2. A Dynamic Analog Concurrently- Processed Adaptive Chip Purpose: To design a reconfigurable analog neural network on a chip To design a reconfigurable analog neural network on a chip A chip that can learn over time For any application neural networks are used Improvement over previous designs: Improvement over previous designs: High density High routability On-chip learning Multiple learning algorithms

3. A Dynamic Analog Concurrently- Processed Adaptive Chip Neuron and synapse circuits are created For TSMC 0.35µm CMOS (Complementary Metal Oxide Semiconductor) implementation For TSMC 0.35µm CMOS (Complementary Metal Oxide Semiconductor) implementation Dense layout of transistors for analog arithmetic circuits Dense layout of transistors for analog arithmetic circuits High Density Neurons and Synapses are as small as possible without compromising performance Neurons and Synapses are as small as possible without compromising performance Supports both Backpropagation and Hebbian learning on each cell Supports both Backpropagation and Hebbian learning on each cell

4. A Dynamic Analog Concurrently- Processed Adaptive Chip Neural networks: Have been used for years in many applications, from business and finance to science and engineering Have been used for years in many applications, from business and finance to science and engineering Ideally implemented as arrays of electronic cells Ideally implemented as arrays of electronic cells Many learning algorithms have been designed Many learning algorithms have been designed The first electronic neural network was made in 1951, by Marvin Minsky (called “SNARC”) Carver Mead pioneered chip implementation of neural cells, designing a retina-like neural circuit Intel created a popular analog neural network chip in 1989 (called “ETANN”)

5. A Dynamic Analog Concurrently- Processed Adaptive Chip Backpropagation: Feedback-based learning algorithm (supervised learning) Feedback-based learning algorithm (supervised learning) Requires calculation of the correct output for an input pattern Requires calculation of the correct output for an input pattern Will learn to represent the correct output over time Will learn to represent the correct output over timeHebbian: Non-feedback-based learning algorithm (unsupervised learning) Non-feedback-based learning algorithm (unsupervised learning) Calculates synapse weights based on correlation between input patterns Calculates synapse weights based on correlation between input patterns Will learn to identify and classify input patterns Will learn to identify and classify input patternsHopfield: Advanced memory recall algorithm, using Hebbian learning Advanced memory recall algorithm, using Hebbian learning

6. A Dynamic Analog Concurrently- Processed Adaptive Chip Neuron Block Diagram inout BPin HebbBP BPout Hebb +- BP

7. A Dynamic Analog Concurrently- Processed Adaptive Chip Neuron Cell Diagram

8. A Dynamic Analog Concurrently- Processed Adaptive Chip The neuron design: Supports Backpropagation and Hebbian Supports Backpropagation and Hebbian Uses an SRAM cell to enable/disable algorithm- dependent circuits Uses tanh sigmoid circuit for forwards- propagation (range: [-1, 1]) Uses tanh sigmoid circuit for forwards- propagation (range: [-1, 1]) Derivative as sech 2 for backwards propagation (range: [0, 1]) Derivative as sech 2 for backwards propagation (range: [0, 1])

9. A Dynamic Analog Concurrently- Processed Adaptive Chip Synapse Block Diagram inout BPin BP BPout Hebb BP Weight Update Parameters

10. A Dynamic Analog Concurrently- Processed Adaptive Chip Synapse Cell Diagram

11. A Dynamic Analog Concurrently- Processed Adaptive Chip The synapse design: Supports Backpropagation and Hebbian Supports Backpropagation and Hebbian Uses an SRAM cell to enable/disable algorithm- dependent circuits Uses the Gilbert multiplier cell for forwards- and backwards- propagation Uses the Gilbert multiplier cell for forwards- and backwards- propagation All inputs are ensured to be symmetrical (c+x, c-x) Differential pair-based circuits are used Differential pair-based circuits are used Input range is small for good linearity

12. A Dynamic Analog Concurrently- Processed Adaptive Chip High routability Multiple routing pathways between neurons and synapses Multiple routing pathways between neurons and synapses Using programmable wires and connections Enables use of advanced learning algorithms Enables use of advanced learning algorithms To evolve/change the layout of the network over time This allows the network to become more dense and more efficient, while maintaining a low synapse count

13. A Dynamic Analog Concurrently- Processed Adaptive Chip Programmable WireProgrammable Connection SRAM Analog Switch SRAM Analog Switch

14. A Dynamic Analog Concurrently- Processed Adaptive Chip Routing Cell Diagram

15. A Dynamic Analog Concurrently- Processed Adaptive Chip Circuit Design Created and modified analog and mixed- signal circuits for each learning/propagation task Created and modified analog and mixed- signal circuits for each learning/propagation task Most circuits are implemented using a small number of MOSFET transistors Most circuits are implemented using a small number of MOSFET transistors Transistors are relatively large (5µm length) and interdigitated for good matching Simulations have been completed for all cells CMOS layout has been completed for some of the cells CMOS layout has been completed for some of the cells

16. A Dynamic Analog Concurrently- Processed Adaptive Chip Current Summation

17. A Dynamic Analog Concurrently- Processed Adaptive Chip Tanh and Sech 2 (Sigmoid and Derivative)

18. A Dynamic Analog Concurrently- Processed Adaptive Chip Differential Pairs (for multiplier input)

19. A Dynamic Analog Concurrently- Processed Adaptive Chip Gilbert Multiplier Cell

20. A Dynamic Analog Concurrently- Processed Adaptive Chip Layout: Completed for tanh/sech 2 Completed for tanh/sech 2 First a stick diagram was planned Initial layout planned in Microsoft Visio Drawn in Cadence Virtuoso using unmatched transistors Drawn in Cadence Virtuoso using matched (interdigitated) transistors Planned for other cells, but not yet complete Planned for other cells, but not yet complete

21. A Dynamic Analog Concurrently- Processed Adaptive Chip Initial Virtuoso Layout of tanh (no matching)

22. A Dynamic Analog Concurrently- Processed Adaptive Chip Virtuoso Layout of tanh with interdigitated matching

23. A Dynamic Analog Concurrently- Processed Adaptive Chip Initial Visio layout of tanh (no matching)

24. A Dynamic Analog Concurrently- Processed Adaptive Chip Simulation Graphs Tanh Sigmoid

25. A Dynamic Analog Concurrently- Processed Adaptive Chip Sigmoid: Input Common Mode Range: 0.5V – 1.5V Input Common Mode Range: 0.5V – 1.5V Good accuracy compared to ideal function of tanh Good accuracy compared to ideal function of tanh

26. A Dynamic Analog Concurrently- Processed Adaptive Chip Simulation Graphs Sech 2 Sigmoid

27. A Dynamic Analog Concurrently- Processed Adaptive Chip Sigmoid Derivative: Input Common Mode Range: 0.5V – 1.5V Input Common Mode Range: 0.5V – 1.5V Good accuracy compared to ideal function of sech 2 Good accuracy compared to ideal function of sech 2

28. A Dynamic Analog Concurrently- Processed Adaptive Chip Simulation Graphs Multiplier

29. A Dynamic Analog Concurrently- Processed Adaptive Chip Multiplier: Input Common Mode Range: 2V – 3V Input Common Mode Range: 2V – 3V Symmetrical inputs around center point Symmetrical inputs around center point Good linearity for input swing of ±150mV Good linearity for input swing of ±150mV Performance is reduced as input range is extended Performance is reduced as input range is extended Pre-processing differential pair is required

30. A Dynamic Analog Concurrently- Processed Adaptive Chip Results: A neural network is simulated in C++ A neural network is simulated in C++ OCR problem, of 5 training sets of randomly ordered numerical characters 10x10 pixels 10x10 pixels 5% noise 5% noise Position offset of ±1 pixel horizontally and vertically Position offset of ±1 pixel horizontally and verticallyComparing: 100%-connected network 100%-connected network 20%-connected network 20%-connected network Simulation of a dynamically re-routed network by removing 10% of the synapses over 1000 cycles Simulation of a dynamically re-routed network by removing 10% of the synapses over 1000 cycles

31. A Dynamic Analog Concurrently- Processed Adaptive Chip Results 0 to 1000 training cycles

32. A Dynamic Analog Concurrently- Processed Adaptive Chip Results 200 to 1000 training cycles

33. A Dynamic Analog Concurrently- Processed Adaptive Chip Results 800 to 1000 training cycles

34. A Dynamic Analog Concurrently- Processed Adaptive Chip Conclusions: The dynamic routing algorithm is the ideal method of optimizing performance in a hardware neural network The dynamic routing algorithm is the ideal method of optimizing performance in a hardware neural network Reducing the number of synapses to increase density The cell circuits perform very well with system-made restrictions on input ranges The cell circuits perform very well with system-made restrictions on input ranges Restrictions increase accuracy and linearity The arrangement of routing cells between neurons and synapses allows the routing algorithm to perform very efficiently The arrangement of routing cells between neurons and synapses allows the routing algorithm to perform very efficiently

35. A Dynamic Analog Concurrently- Processed Adaptive Chip Special Thanks To: Victoria Stagg Victoria Stagg My mother, for all of her help and support Andrew Stagg Andrew Stagg My brother, for all of his help and support Dr. Jim Haslett, University of Calgary Dr. Jim Haslett, University of Calgary For use of the ATIPS laboratory, to use Cadence Virtuoso John Carney, Cadence Design John Carney, Cadence Design For the demo license of Cadence Orcad Layout Dr. Vance Tyree, MOSIS Corporation Dr. Vance Tyree, MOSIS Corporation For agreeing to accept a fabrication proposal under the MOSIS education program David Wells, Auton Engineering, Ltd. David Wells, Auton Engineering, Ltd. For printing my trifold