A Weight Pushing Algorithm Michael Güttinger

Overview Semirings Weighted finite-state acceptors (WFSAs) The Weight Pushing Algorithm Results

Semirings Semiring (K, ⊕, ⊗,, ) K a set ⊕ associative and commutative ⊗ associative a ⊗ (b ⊕ c) = a ⊗ b ⊕ a ⊗ c Identity ( ⊕ ), Identity ( ⊗ ) 0 ⊗ a= a ⊗ 0=0

Semirings Examples Probability semiring ( ℝ₊,+,*,0,1) Log semiring ( ℝ₊  {∞}, ⊕,+,∞,0) with ∀ a,b ∈ ℝ  {∞},a ⊕ b =-log(exp(-a)+exp(- b)) Tropical Semiring ( ℝ₊  {∞},min,+, {∞},0)

Weighted Finite State Acceptors (WFSAs) WFSA A=(∑,Q,E,i,F,λ,ρ) over a Semiring K ∑ alphabet Q finite set of states E finite set of transitions ⊆ Q  ∑  {ε}  K  Q Initial state i ∈ Q, set of final states F ⊆ Q Initial weight λ,final weight function ρ

WFSA Transitions Transition t = (p[t],l[t],w[t],n[t]) Source state p[t], Destination state n[t], Label l[t], Weight w[t]

WFSA: Path A path in A is a consecutive transitions with: n[ ] = p[ ] ( ∀ i=1,..,n-1) A successful path π= is a path from the initial state i to a final state F The weight w[π] of path π is: w[π] = λ ⊗ w[ ] ⊗.... ⊗ w[ ] ⊗ ρ(n[ ]) Two WFSAs are equivalent when they associate the same weight with any given input string. Equivalent WFSAs may have weights distributed differently along their paths.

Motivation for Weight Pushing The weight pushing algorithm will build an equivalent automata whose weight distribution is better for pruning during speech recognition. This means pushing the weights as far towards the initial state as possible. The final weights assigned to all arcs leaving any given state will sum to unity.

Weight Pushing Potential function V:Q  K – {0}: The weights are updated by: λ  λ ⊗ V(i) ∀ e E,w[e]  [V(p[e])] ⁻ ¹ ⊗ (w[e] ⊗ V(n[e])) ∀ f F, ρ(f)  [V(f)] ⁻ ¹ ⊗ ρ[f] D[q] = w[ π ] V(q) is equal to the shortest distance from q to any final state.

Weighted Acceptor A V(q)= min w[ π ] Tropical Semiring V[0]=0; V[1]=0; V[2]=10; V[3]=0; w[e]  [V(p[e])] ⁻ ¹ + (w[e] +V(n[e])) w[e]  w[e] +V(n[e])- V(p[e])

Result of Pushing A over the Tropical Semiring

Weighted Acceptor A V(q)= ⊕ w[ π ] Log - Semiring w[e]  [V(p[e])] ⁻ ¹ ⊗ (w[e] ⊗ V(n[e])) λ  λ ⊗ V(i)

Result of Pushing A over Log Semiring

Consequence of Pushing In the Tropical Semiring the shortest path from each state to a final State has weight 0 In the Automaton we obtained from A by pushing over the log semiring: At each state,the outgoing weights sum to 1 With classical minimization the size of both automatons can be reduced

Conditions for Computing V(q ) The semiring K is divisible, when A k-closed Semiring is a semiring for which there exists a k such that:

Algorithm for Computing V(q) Source-Shortest-Distance(q); (tropical Semiring) For j= 1 to |Q| do d[j] =r[j]= ∞ ( ) d[j] an estimate of the shortest d[q] = r[q] =0 ( ) distance from q to j S={q}r[j] the total weight add to d[j]since While S  the last time j was extracted from S do node=head(S) DEQUEUE(S) R=r[node] r[node]=∞ ( ) for each e ∈ E[q] do if d[n[e]]  min (d[n[e]],(R+w[e]) //d[n[e]]  d[n[e]] ⊕ (R ⊗ w[e]) then d[n[e]]  min (d[n[e]],(R+w[e]) // d[n[e]] = d[n[e]] ⊕ (R ⊗ w[e]) r[n[e]]  min (r[n[e]],(R+w[e]) // r[n[e]] = r[n[e]] ⊕ (R ⊗ w[e]) if n[e] ∉ S then ENQUEUE(S,n[e]) d[q]=0

Algorithm to compute V(q) Source-Shortest-Distance(q); () For j= 1 to |Q| do d[j] =r[j]= d[j] an estimate of the shortest d[q] = r[q] = distance from q to j S={q} r[j] the total weight add to d[j]since While S  the last time j was extracted from S do node=head(S) DEQUEUE(S) R=r[node] r[node]=0 for each e ∈ E[q] do if d[n[e]]  d[n[e]] ⊕ (R ⊗ w[e]) then d[n[e]] = d[n[e]] ⊕ (R ⊗ w[e]) r[n[e]] = r[n[e]] ⊕ (R ⊗ w[e]) if n[e] ∉ S then ENQUEUE(S,n[e]) d[q]=0

Some facts about Weight Pushing Using tropical or log semiring pushing in the minimization step result in equivalent machines. But the distribution of the weights differs often radically The log semirring benefits speech recognition pruning Using the tropical semiring can be harmful in some cases

Experiments and Results

40k-word NAB Task:North American BusinessNews Vocabulary size: 40K Words Trigram language model with transitions Triphonic acoustic model Compaq alpha processor

X Real Time Word Accuracy 40k-word NAB

160k-word NAB Task:North American BusinessNews Vocabulary size: 160K Words 6-gram language model with transitions Triphonic acoustic model Compaq alpha processor

X Real Time Word Accuracy 160-word NAB

Summary The principle of weight pushing. The algorithm for the shortest distance. In consequence of the weight pushing algorithm is that speech recognition will be much faster.