1. Ghent University Compact hardware for real-time speech recognition using a Liquid State Machine Benjamin Schrauwen – Michiel D’Haene David Verstraeten – Jan Van Campenhout Electronics and Information Systems Department Ghent University – Belgium IJCNN 2007

2. 2/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Intro Goal: Create isolated digits speech recognition in digital FPGA hardware Real-time processing As small as possible First introduce LSM based speech recognition Investigate two existing hardware architectures (to fast, to large) Introduce new hardware architecture

3. 3/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 LSM based speech recognition

4. 4/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Ear model

5. 5/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Liquid State Machine Recurrent structures without the training: Reservoir Computing Jaeger (2001): Echo State Networks (engineering) Maass (2002): Liquid State Machines (neuroscience) Steil (2003): weight dynamics of Atiya-Parlos equivalent Fixed, random topology operated in correct dynamic regime Different node types possible: THG, linear, tanh, spiking, … Linear “readout” function which is trained (No local minima, no problems with recurrent structure, one shot learning) On-line computing: prediction at every time-step Any time-invariant filter with fading memory can be learned (with output feedback, universal computing)

6. 6/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Reservoir computing

7. 7/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Influence of parameters Not very important: Connection fraction Exact topology Weight distribution timescale error dynamic regime chaos error reservoir size error

8. 8/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Spiking Neural Networks Incoming spikes influence the membrane potential When certain threshold θ is reached: reset and fire t input output membr θ

9. 9/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Readout and post-processing

10. 10/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Digital spiking neurons SNN: mathematically a more complex model than ANN But: better implementable in hardware No weight multiplications: table look-up Filtering can be implemented using shifts and adds Interconnection only single bit, and sparse communication Asynchronous communication easily implementable

11. 11/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Digital spiking neurons Hardware can take advantage of parallelism But area-speed trade-off: we don’t have to make the implementation faster than needed by the application For trade-off: different implementations with other area-speed needed Possible parallelisms: Network parallelism Neuron/synapse parallelism Arithmetic parallelism We implemented: SPPA: network parallel, neuron serial, arithmetic parallel PPSA: network parallel, neuron parallel, arithmetic serial SPSA: network serial or parallel, neuron serial, arithmetic serial

12. 12/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 SPPA Much like classical CPU [Roggen 2003][Upegui 2005]

13. 13/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 PPSA Use FPGA features [Girau2006] [Schrauwen2006] SRL16 Serial adder

14. 14/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 SPSA Everything serial PEs are very small (4 LUTs) Many parallel PEs possible SIMD controller architecture Memory based interconnect

15. 15/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Area-speed trade-off for speech task Speech task in hardware LSM with 200 neurons 16 kHz processing speed Real-time requirement LUTsmemoryReal-time SPPA13812900 kbit347 PPSA1342658 kbit205 SPSA 10PE488144 kbit2.2 SPSA 5PE489144 kbit1.1 SPSA 1PE489144 kbit0.23

16. 16/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 RC Toolbox Freely available RC Matlab toolbox (www.elis.ugent.be/rct) Simulation models for hardware quantization HW design methodology: Generic network and node settings Readout pipeline Generate network with hardware constraints Evaluate node quantization effects Automatically export to HW description

17. 17/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 System 12 MHz 100 MHz 150 MHz interc 50 MHz Xilinx ML 401

18. 18/18 Compact hardware for real-time speech recognition using a LSM IJCNN – August 13, 2007 Conclusions Hardware real-time speech recognition in HW is possible with very limited hardware Presented novel architecture for SNN implementation in HW Enlarges area/speed design space drastically Uses RC toolbox simulation environment for easy porting Future work: experiment with different applications; add further SNN features such as SDTP, IP, dyn. synapses; and add the possibility to change the weights