Finite-State Automata Shallow Processing Techniques for NLP Ling570 October 5, 2011

Roadmap Finite-state automata variants Non-deterministic Finite-State Automata (NFA) Probabilistic Finite-State Automata (PFSA) Finite-State Transducers (FST) Intro CARMEL

FSAs Formally A Finite-State Automaton (FSA) is a 5-tuple: A set of states Q { q 0,q 1,q 2,q 3,q 4 } A finite alphabet Σ {b,a,!} A state state q 0 A set of accepting states {q 4 } A transition function Q x Σ  Q

Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition

Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA

Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a

Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a ε-transitions Transitions that consume no input

Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a ε-transitions Transitions that consume no input Multiple start states Transitions: Q x (Σ U ε)  2 S

Sheeptalk NFA Multiple choice:

Sheeptalk NFA Multiple choice: ε-transitions

NFA & DFA What is the relationship b/t NFAs and DFAs?

NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages

NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA

NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA If NFA has N states, up to 2 N states in DFA

NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA If NFA has N states, up to 2 N states in DFA Why use both?

Managing Non-Determinism Approaches to non-determinism

Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative

Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative Look-ahead: Look ahead to see which choice to take

Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative Look-ahead: Look ahead to see which choice to take Parallelism: At choice points, consider all paths in parallel Basically the NFA  DFA conversion process

Recognition in NFAs Two standard approaches:

Recognition in NFAs Two standard approaches: Transformation: Convert NFA  DFA Perform standard DFA recognition

Recognition in NFAs Two standard approaches: Transformation: Convert NFA  DFA Perform standard DFA recognition Search: Explicitly model recognition as state-space search Perform backtracking as necessary

Recognition in NFA For strings in the language:

Recognition in NFA For strings in the language: There exists some state sequence leading to a final state

Recognition in NFA For strings in the language: There exists some state sequence leading to a final state Not all paths necessarily lead to an accept state For strings not in the language:

Recognition in NFA For strings in the language: There exists some state sequence leading to a final state Not all paths necessarily lead to an accept state For strings not in the language: No paths lead to an accept state

NFA Recognition as Search Search problem:

NFA Recognition as Search Search problem: Start search-state (start node, beginning of input)

NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input)

NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions

NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions Path cost

NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions Path cost Use standard search algorithms e.g. Depth-first, breadth first, A*

Example

Breadth-first Search

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata E.g. Concatenation

Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata E.g. Concatenation

Equivalences and Operations Closure: Union:

Equivalences and Operations Closure: Union:

Equivalences and Operations Closure: Union:

Finite-State Tools Toolkits: AT&T Finite-State Toolkit: NLTK: CARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel

Finite-State Tools Toolkits: AT&T Finite-State Toolkit: NLTK: CARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel Functions: Convert Regex to FSA Convert FSA to Regex Convert NFA to DFA Perform operations: concatenation, disjunction, *, etc

Probabilistic Finite-State Automata (PFA)

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R +

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R +

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R +

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R + s.t.

PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R + s.t.

PFA Example Example Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) = 0.8 n *0.2 Example due to F. Xia

PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) = 0.8 n *0.2 Example due to F. Xia

PFA PFA augments NFAs with probabilities on arcs

PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs)

PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths.

PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths. Tech. issue: Numerical underflow:

PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths. Tech. issue: Numerical underflow: Replace product of probs with sum of log probs Tasks: Given an input, find most probable state sequence Given an input, find its probability etc (We’ll do these when we generalize to HMMs)

Weighted FSA Similar to PFA Arcs associated with weight (not probability)

Finite-State Transducers

FSTs Finite automaton that maps between two strings Automaton with two labels/arc input:output

FST Applications Tokenization Segmentation Morphological analysis Transliteration Translation Speech recognition Spoken language understanding

Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects

Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages

Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages FST as translator: Reads an input string and prints output string

Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages FST as translator: Reads an input string and prints output string FST as set relator: Computes relations between sets

FSTs, Formally

FSTs & Regular Relations FSAs: equivalent to regular languages

FSTs & Regular Relations FSAs: equivalent to regular languages FSTs: equivalent to regular relations Sets of pairs of strings

FSTs & Regular Relations FSAs: equivalent to regular languages FSTs: equivalent to regular relations Sets of pairs of strings Regular relations: For all (x,y) in Σ 1 x Σ 2, {(x,y)} is a regular relation The empty set is a regular relation If R 1,R 2 are regular relations, R 1  R 2, R 1 U R 2 and R 1 * are regular relations

Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to !

Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to ! Composition: If T 1 is a transducer from I 1 to O 2 and T 2 is a transducer from O 2 to O 3, then T 1 T 2 is a transducer from I 1 to O 3

Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to ! Composition: If T 1 is a transducer from I 1 to O 2 and T 2 is a transducer from O 2 to O 3, then T 1 T 2 is a transducer from I 1 to O 3

CARMEL

Defining an FSA/FST final-state (from-state (to-state “input-sym” “outputsym”? weight?)*) ….. First line is final state Lines define state transitions input-sym (in quotes) required output-sym, weight optional *e* is the representation for ε patas:/NLP_TOOLS/ml_tools/fsm/carmel/latest/bin/carmel

FSA Example q4 (q0 (q1 “b”)) (q1 (q2 “a”)) (q2 (q3 “a”)) (q3 (q2 *e*)) (q3 (q4 “!”))

FST Example q1 (q0 (q0 “a” “A”)) (q0 (q1 “b” “b”)) (q1 (q1 “b” “B”))

WFST Example S (S (S “they” “PRO” 1.0)) (S (S “can” “AUX” 0.99)) (S (S “can” “VERB” 0.01)) (S (S “fish” “NOUN” 0.7)) (S (S “fish” “VERB” 0.3))

Basic Functions carmel –k N mywfst Prints top N most probable paths through mywfst carmel –Ok N mywsft Print top N most probable outputs from mywfst carmel fst1 fst2 Composes fst1 and fst2 cat input_file | carmel –sli mywfst Converts 1 st line of input_file to fst & composes (left) -b flag: batch processes & prints best path

HW #2 Finite-State Automata Goals: Build Finite-State Automata to represent Regex Employ modern FST package to build FSA acceptor Implement FSA acceptor yourselves