2. Introduction to Convolution circuits synthesis image processing, speech processing, DSP, polynomial multiplication in robot control. convolution

3. FIR-filter like structure b4b3 b2b1 +++ a4000 a4*b4 Separate input and output Input and output move synchronized Weights stay in space

4. b4b3 b2b1 +++ a400 a4*b4 a3 a3*b4+a4b3

5. b4b3 b2b1 +++ a3a40 a4*b4 a2 a3*b4+a4b3 a4*b2+a3*b3+a2*b4

6. b4b3 b2b1 +++ a2a3a4 a4*b4 a1 a3*b4+a4b3 a4*b2+a3*b3+a2*b4 a1*b4+a2*b3+a3*b2+a4*b1

7. b4b3 b2b1 +++ a1a2a3 a4*b4 0 a3*b4+a4b3 a4*b2+a3*b3+a2*b4 a1*b4+a2*b3+a3*b2+a4*b1 a1*b3+a2*b2+a3*b1

8. We insert Dffs to avoid many levels of logic b4b3 b2b1 +++ a4a2a3 a4*b4 a4*b3 a4*b2a4*b1

9. b4b3 b2b1 +++ a3a1a2 a4*b4 a4*b3+a3b4 a4*b2+a3b3 a4*b1+a3b2 a3b1

10. b4b3 b2b1 +++ a20a1 a4*b4 a4*b3+a3b4 a4*b2+a3b3+a2b4 a4*b1+a3b2+a2b3 a3b1+a2b2 a2b1 The disadvantage of this circuit is broadcasting

11. We insert more Dffs to avoid broadcasting b4b3 b2b1 +++ a4a2a3 a4*b4 0 00 000

12. b4b3 b2b1 +++ a3a1a2 a4*b4 a3b4 a4b3 0 a400 0 Does not work correctly like this, try something new….

13. b4b3 b2b1 a3a1a2 a4*b4 a3b4a4b3 0 a400 0 a2b4 a1b4 a3b3 a2b3 a1b3 00 0 0 a4b2 a3b2 a2b2 a1b2 0 0 0 a4b1 a3b1 a2b1 First sum Second sum

14. FIR-filter like structure, assume two delays b4b3 b2b1 +++

15. b4b3 b2b1 +++

16. b4b3 b2b1 +++

17. b4b3 b2b1 +++

18. b4b3 b2b1 +++

19. b4b3 b2b1 +++

20. b4b3 b2b1 +++

21. b4b3 b2b1 +++

22. b4b3 b2b1 +++

23. b4b3 b2b1 +++

24. b4b3 b2b1 +++

25. b4b3 b2b1 +++

26. b4b3 b2b1 +++

27. b4b3 b2b1 +++

28. Convolution Algorithm Two loops Patterns of operations on vectors: [a0*b0, a1*b1, … an*bn] 1.vector product is not dot product [a0*b0, a1*b1, … an*bn] a0*b0 + a1*b1 + … + an*bn 2.Dot product (scalar product) = vector product with accumulation a0*b0 + a1*b1 + … + an*bn 3.Polynomial multiplication

29. Example 3: FIR Filter or Convolution

30. Example 3: Convolution There are many ways to implement convolution using systolic arrays, one of them is shown: –u(n) : The input of sequence from left. –w(n) : The weights preloaded in n PEs. –y(n) : The sequence from right (Initial value: 0) and having the same speed as u(n). In this operation each cell’s function is: –1. Multiply the inputs coming from left with weights and output the input received to the next cell. –2. Add the final value to the inputs from right. W0W0 W1W1 W2W2 W3W3 u i ……u 0 y i ……y 0 0 WiWi a in b out a out b in a out = a in b out = b in + a in * w i

31. Each cell operation. W0W0 W1W1 W2W2 W3W3 u i ……u 0 y i ……y 0 0 WiWi a in b out a out b in a out = a in b out = b in + a in * w i Convolution (cont) Systolic array. The input of sequence from left. This is just one solution to this problem Weights in space Inputs and outputs in each cell

32. Convolution Can be 1D, 2D, 3D, etc. Is very important in many applications. Can be implemented efficiently in various architectures. Is an excellent example to compare various computer architectures: –SIMD, – MIMD, –CA, –pipelined, –Systolic.

33. Various Possible Implementations Convolution is very important, we use it in several applications. So let us think what are to implement it Convolution is very important, we use it in several applications. So let us think what are all the possible ways to implement it Convolution Algorithm

34. Bag of Tricks that can be used Preload-repeated-value Replace-feedback-with-register Internalize-data-flow Broadcast-common-input Propagate-common-input Retime-to-eliminate-broadcasting

35. How to invent such circuits? 1.Let us learn from existing designs 2.Let us learn from our own mistakes 3.Let us check all possibilities of moving every piece of data

36. Bogus Attempt at Systolic FIR for i=1 to n in parallel for j=1 to k in place y i += w j * x i+j-1 feedback from sequential implementation Replace with register Inner loop realized in place Stage 1: directly from equation Stage 2: feedback = y i = y i Stage 3: Internal loop in space

37. Bogus Attempt continued: Outer Loop for i=1 to n in parallel for j=1 to k in place y i += w j * x i+j-1 Factorize w j This could work but it has broadcast

38. Bogus Attempt continued: Outer Loop - 2 for i=1 to n in parallel for j=1 to k in place y i += w j * x i+j-1 Because we do not want to have broadcast, we retime the signal w, this requires also retiming of X j

39. Another possibility of retiming for i=1 to n in parallel for j=1 to k in place y i += w j * x i+j-1 Bogus Attempt continued: Outer Loop - 2a

40. Yet another approach is to broadcast common input x i-1 Bogus Attempt continued: Outer Loop - 3 for i=1 to n in parallel for j=1 to k in place y i += w j * x i+j-1

41. What we achieved? We showed several possible beginnigs of creating architectures. They were not successful, but show the principles. We will continue to create architectures, but these attempts were not complete waste of time, they can be used in similar problems successfully. You have to experiment with ideas!!

42. Attempt at Systolic FIR: now internal loop is in parallel Internal loop in parallel

43. Attempt at Systolic FIR: now internal loop is in parallel 1 2 3

44. Outer Loop continuation for FIR filter

45. Continue: Optimize Outer Loop Preload-repeated Value Based on previous slide we can preload weights Wi

46. Continue: Optimize Outer Loop Broadcast Common Value This design has broadcast. Some purists tell this is not systolic as systolic should have all short wires.

47. Continue: Optimize Outer Loop Retime to Eliminate Broadcast We delay these signals y i

48. The design becomes not intuitive. Therefore, we have to explain in detail “How it works” y1=x1w1 x1 x2 inputs outputs Was it a good idea to combine input and output streams to cells?

49. Polynomial Multiplication of 1-D convolution problem

50. Types of systolic structure Convolution problem weight : {w 1, w 2,..., w k } inputs : {x 1, x 2,..., x n } results : {y 1, y 2,..., y n+k-1 } y i = w 1 x i + w 2 x i+1 +...... + w k x i+k-1 (combining two data streams) H. T. Kung’s grouping work assume k = 3 Polynomial Multiplication of 1-D convolution problem

51. A well-known family of systolic designs for convolution computation Given the sequence of weights {w 1, w 2,..., w k } And the input sequence {x 1, x 2,..., x k }, Compute the result sequence {y 1, y 2,..., y n+1-k } Defined by y i = w 1 x i + w 2 x i+1 +... + w k x i+k-1

52. Design B1 - Broadcast input, - move results systolically, - weights stay - (Semi-systolic convolution arrays with global data communication -

53. Design B1 - Broadcast input, - move results systolically, - weights stay - (Semi-systolic convolution arrays with global data communication Previously proposed for circuits to implement a pattern matching processor and for circuit to implement polynomial multiplication. -

54. Types of systolic structure: design B1 wider systolic path (partial result y i move) x3x3 x2x2 x1x1 y3y3 y2y2 y1y1 W1W1 W2W2 W3W3 y in x in y out y out = y in + W  x in W Please analyze this circuit drawing snapshots like in an animated movie of data in subsequent moments of time broadcast Discuss disadvantages of broadcast Results move out

55. Design B2 Inputs broadcast Weights move Results stay

56. Types of systolic structure: Design B2 Inputs broadcast Weights move Results stay w i circulate use multiplier-accumulator hardware w i has a tag bit (signals accumulator to output results) needs separate bus (or other global network for collecting output) W in x in W out y = y + W in  x in W out = W in y x3x3 x2x2 x1x1 y1y1 y2y2 y3y3 W2W2 W3W3 W1W1

57. Design B2 Broadcast input, move weights, results stay [(Semi-) systolic convolution arrays with global data communication] The path for moving y i ’s is wider then w i ’s because of y i ’s carry more bits then w i ’s in numerical accuracy. The use of multiplier- accumulators may also help increase precision of the result, since extra bit can be kept in these accumulators with modest cost. Semisystolic because of broadcast

58. Design F Input move Weights stay Partial results fan-in needs adder

59. Types of systolic structure: design F Input move Weights stay Partial results fan-in needs adder applications : signal processing, pattern matching y 1 ’s Z out = W  x in x out = x in Z out x out x in W x3x3 x2x2 x1x1 W3W3 W2W2 W1W1 ADDER

60. Design F - Fan-in results, move inputs, weights stay - Semi-systolic convolution arrays with global data communication When number of cell is large, the adder can be implemented as a pipelined adder tree to avoid large delay. Design of this type using unbounded fan-in.

61. Design R1 Inputs and weights move in the opposite directions Results stay can use tag bit no bus (systolic output path is sufficient) one-half the cells work at any time

62. Types of systolic structure: Design R1 Inputs and weights move in the opposite directions Results stay can use tag bit no bus (systolic output path is sufficient) one-half the cells are work at any time applications : pattern matching y = y + W in  x in x out = x in W out = W in x1x1 x3x3 x2x2 W1W1 W2W2 y3y3 y2y2 y1y1 W in x in W out y x out

63. Design R1 - Results stay, inputs and weights move in opposite directions - Pure-systolic convolution arrays with global data communication Design R1 has the advan- tage that it dose not require a bus, or any other global net- work, for collecting output from cells. The basic ideal of this de- sign has been used to imple- ment a pattern matching chip.

64. Design R2 Inputs and weights move in the same direction at different speeds Results stay x j ’s move twice as fast as the w j ’s all cells work at any time need additional registers (to hold w value)

65. Types of systolic structure: design R2 Inputs and weights move in the same direction at different speeds Results stay x j ’s move twice as fast as the w j ’s all cells work at any time need additional registers (to hold w value) applications : pipeline multiplier W1W1 W2W2 W3W3 W4W4 W5W5 x3x3 x2x2 x1x1 y1y1 y2y2 y3y3 WWW W y W in W out x in x out y = y + W in  x in W = W in W out = W x out = x in

66. Design R2 - Results stay, inputs and weights move in the same direction but at different speeds - Pure-systolic convolution arrays with global data communication Multiplier-accumulator can be used effectively and so can tag bit method to signal the output of each cell. Compared with R1, all cells work all the time when additional register in each cell to hold a w value.

67. Design W1 Inputs and results move in the opposite direction Weights stay one-half the cells are work constant response time

68. Types of systolic structure: design W1 Inputs and results move in the opposite direction Weights stay one-half the cells are work constant response time applications : polynomial division y out = y in + W  x in x out = x in y in x in y out W x out x1x1 x3x3 x2x2 W1W1 W2W2 y W3W3

69. Design W1 -Weights stay, inputs and results move in opposite direction - Pure-systolic convolution arrays with global data communication This design is fundamental in the sense that it can be naturally extend to perform recursive filtering. This design suffers the same drawback as R1, only appro- ximately 1/2 cells work at any given time unless two inde- pendent computation are in- terleaved in the same array.

70. Overlapping the executions of multiply-and-add in design W1

71. Design W2 Inputs and results move in the same direction at different speeds Weights stay high throughputsall cells work (high throughputs rather than fast response)

72. Types of systolic structure: design W2 Inputs and results move in the same direction at different speeds Weights stay high throughputsall cells work (high throughputs rather than fast response) x W x in x out y in y out y out = y in + W in  x in x = x in x out = x W1W1 W2W2 x5x5 W3W3 x7x7 x3x3 x2x2 x1x1 y1y1 y2y2 y3y3 WWW x4x4 x6x6

73. Design W2 -Weights stay, inputs and results move in the same direction but at different speeds - Pure-systolic convolution arrays with global data communication This design lose one advan- tage of W1, the constant response time. 2-D convolution, This design has been extended to implement 2-D convolution, where high throughputs rather than fast response are of concern.

74. Remarks on Linear Arrays Above designs are all possible systolic designs for the convolution problem. (some are semi-) Using a systolic control path, weight can be selected on- the-fly to implement interpolation or adaptive filtering. We need to understand precisely the strengths and drawbacks of each design so that an appropriate design can be selected for a given environment. For improving throughput, it may be worthwhile to implement multiplier and adder separately to allow overlapping of their execution. (Such as next page show) When chip pin is considered: pure-systolic requires four I/O ports; semi-systolic requires three I/O ports.

75. Retiming of filters

76. FIR circuit: initial design delays Pipelining of x i We insert various numbers of unified delays

77. FIR circuit: registers added below weight multipliers Notice changed timing here We insert delays here

78. FIR Summary: comparison of sequential and systolic

79. Conclusions on 1D and 1.5D Systolic Arrays Systolic arrays are more than processor arrays which execute systolic algorithms. one of the following –A systolic cell takes on one of the following forms: 1.A special purpose cell with hardwired functions, 2.A vector-computer-like cell with instruction decoding and a processing element, 3.A systolic processor complete with a control unit and a processing unit. Smarter processor for SAT, Petrick, etc.

80. Large Systolic Arrays as general purpose computers Large Systolic Arrays as general purpose computers

81. Originally, systolic architectures were motivated for high performance special purpose computational systems that meet the constraints of VLSI, However, it is possible to design systolic systems which: –have high throughputs –yet are not constrained to a single VLSI chip.

82. Problems with systolic array design 1.Hard to design - hard to understand low level realization may be hard to realize 2. Hard to explain remote from the algorithm function can’t readily be deduced from the structure 3.Hard to verify

83. Key architectural issues in designing special-purpose systems special-purpose systems Simple and regular design Simple, regular design yields cost-effective special systems. Concurrency and communication Design algorithm to support high concurrency and meantime to employ only simple blocks. Balancing computation with I/O A special-purpose system should be a match to a variety of I/O bandwidths.

84. Two Dimensional Systolic Arrays Two Dimensional Systolic Arrays In 1978, the first systolic arrays were introduced as a feasible design for special purpose devices which meet the VLSI constraints. These special purpose devices were able to perform four types of matrix operations at high processing speeds: –matrix-vector multiplication, –matrix-matrix multiplication, –LU-decomposition of a matrix, –Solution of triangular linear systems.

85. General Systolic Organization General Systolic Organization

86. Example 2: Example 2: Matrix-Matrix Multiplication All previously shown tricks can be applied

87. Seth Copen Goldstein, CMU A.R. Hurson 2. David E. Culler, UC. Berkeley, 3. 4. Syeda Mohsina Afroze and other students of Advanced Logic Synthesis, ECE 572, 1999 and 2000.Seth Copen Goldstein, CMU A.R. Hurson 2. David E. Culler, UC. Berkeley, 3. Keller@cs.hmc.edu 4. Syeda Mohsina Afroze and other students of Advanced Logic Synthesis, ECE 572, 1999 and 2000. Sources