1. A Weight Pushing Algorithm Michael Güttinger

2. Overview Semirings Weighted finite-state acceptors (WFSAs) The Weight Pushing Algorithm Results

3. Semirings Semiring (K, ⊕, ⊗,, ) K a set ⊕ associative and commutative ⊗ associative a ⊗ (b ⊕ c) = a ⊗ b ⊕ a ⊗ c Identity ( ⊕ ), Identity ( ⊗ ) 0 ⊗ a= a ⊗ 0=0

4. Semirings Examples Probability semiring ( ℝ₊,+,*,0,1) Log semiring ( ℝ₊  {∞}, ⊕,+,∞,0) with ∀ a,b ∈ ℝ  {∞},a ⊕ b =-log(exp(-a)+exp(- b)) Tropical Semiring ( ℝ₊  {∞},min,+, {∞},0)

5. Weighted Finite State Acceptors (WFSAs) WFSA A=(∑,Q,E,i,F,λ,ρ) over a Semiring K ∑ alphabet Q finite set of states E finite set of transitions ⊆ Q  ∑  {ε}  K  Q Initial state i ∈ Q, set of final states F ⊆ Q Initial weight λ,final weight function ρ

6. WFSA Transitions Transition t = (p[t],l[t],w[t],n[t]) Source state p[t], Destination state n[t], Label l[t], Weight w[t]

7. WFSA: Path A path in A is a consecutive transitions with: n[ ] = p[ ] ( ∀ i=1,..,n-1) A successful path π= is a path from the initial state i to a final state F The weight w[π] of path π is: w[π] = λ ⊗ w[ ] ⊗.... ⊗ w[ ] ⊗ ρ(n[ ]) Two WFSAs are equivalent when they associate the same weight with any given input string. Equivalent WFSAs may have weights distributed differently along their paths.

8. Motivation for Weight Pushing The weight pushing algorithm will build an equivalent automata whose weight distribution is better for pruning during speech recognition. This means pushing the weights as far towards the initial state as possible. The final weights assigned to all arcs leaving any given state will sum to unity.

9. Weight Pushing Potential function V:Q  K – {0}: The weights are updated by: λ  λ ⊗ V(i) ∀ e E,w[e]  [V(p[e])] ⁻ ¹ ⊗ (w[e] ⊗ V(n[e])) ∀ f F, ρ(f)  [V(f)] ⁻ ¹ ⊗ ρ[f] D[q] = w[ π ] V(q) is equal to the shortest distance from q to any final state.

10. Weighted Acceptor A V(q)= min w[ π ] Tropical Semiring V[0]=0; V[1]=0; V[2]=10; V[3]=0; w[e]  [V(p[e])] ⁻ ¹ + (w[e] +V(n[e])) w[e]  w[e] +V(n[e])- V(p[e])

11. Result of Pushing A over the Tropical Semiring

12. Weighted Acceptor A V(q)= ⊕ w[ π ] Log - Semiring w[e]  [V(p[e])] ⁻ ¹ ⊗ (w[e] ⊗ V(n[e])) λ  λ ⊗ V(i)

13. Result of Pushing A over Log Semiring

14. Consequence of Pushing In the Tropical Semiring the shortest path from each state to a final State has weight 0 In the Automaton we obtained from A by pushing over the log semiring: At each state,the outgoing weights sum to 1 With classical minimization the size of both automatons can be reduced

15. Conditions for Computing V(q ) The semiring K is divisible, when A k-closed Semiring is a semiring for which there exists a k such that:

16. Algorithm for Computing V(q) Source-Shortest-Distance(q); (tropical Semiring) For j= 1 to |Q| do d[j] =r[j]= ∞ ( ) d[j] an estimate of the shortest d[q] = r[q] =0 ( ) distance from q to j S={q}r[j] the total weight add to d[j]since While S  the last time j was extracted from S do node=head(S) DEQUEUE(S) R=r[node] r[node]=∞ ( ) for each e ∈ E[q] do if d[n[e]]  min (d[n[e]],(R+w[e]) //d[n[e]]  d[n[e]] ⊕ (R ⊗ w[e]) then d[n[e]]  min (d[n[e]],(R+w[e]) // d[n[e]] = d[n[e]] ⊕ (R ⊗ w[e]) r[n[e]]  min (r[n[e]],(R+w[e]) // r[n[e]] = r[n[e]] ⊕ (R ⊗ w[e]) if n[e] ∉ S then ENQUEUE(S,n[e]) d[q]=0

17. Algorithm to compute V(q) Source-Shortest-Distance(q); () For j= 1 to |Q| do d[j] =r[j]= d[j] an estimate of the shortest d[q] = r[q] = distance from q to j S={q} r[j] the total weight add to d[j]since While S  the last time j was extracted from S do node=head(S) DEQUEUE(S) R=r[node] r[node]=0 for each e ∈ E[q] do if d[n[e]]  d[n[e]] ⊕ (R ⊗ w[e]) then d[n[e]] = d[n[e]] ⊕ (R ⊗ w[e]) r[n[e]] = r[n[e]] ⊕ (R ⊗ w[e]) if n[e] ∉ S then ENQUEUE(S,n[e]) d[q]=0

18. Some facts about Weight Pushing Using tropical or log semiring pushing in the minimization step result in equivalent machines. But the distribution of the weights differs often radically The log semirring benefits speech recognition pruning Using the tropical semiring can be harmful in some cases

19. Experiments and Results

20. 40k-word NAB Task:North American BusinessNews Vocabulary size: 40K Words Trigram language model with 4 000 000 transitions Triphonic acoustic model Compaq alpha processor

21. X Real Time Word Accuracy 40k-word NAB

22. 160k-word NAB Task:North American BusinessNews Vocabulary size: 160K Words 6-gram language model with 40 000 000 transitions Triphonic acoustic model Compaq alpha processor

23. X Real Time Word Accuracy 160-word NAB

24. Summary The principle of weight pushing. The algorithm for the shortest distance. In consequence of the weight pushing algorithm is that speech recognition will be much faster.