1. Finite-State Automata Shallow Processing Techniques for NLP Ling570 October 5, 2011

2. Roadmap Finite-state automata variants Non-deterministic Finite-State Automata (NFA) Probabilistic Finite-State Automata (PFSA) Finite-State Transducers (FST) Intro CARMEL

3. FSAs Formally A Finite-State Automaton (FSA) is a 5-tuple: A set of states Q { q 0,q 1,q 2,q 3,q 4 } A finite alphabet Σ {b,a,!} A state state q 0 A set of accepting states {q 4 } A transition function Q x Σ  Q

4. Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition

5. Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA

6. Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a

7. Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a ε-transitions Transitions that consume no input

8. Non-deterministic Finite-State Automata Deterministic FSA: Given current state x input, at most one legal transition Sources of non-determinism in NFA More than one transition for q i x a ε-transitions Transitions that consume no input Multiple start states Transitions: Q x (Σ U ε)  2 S

9. Sheeptalk NFA Multiple choice:

10. Sheeptalk NFA Multiple choice: ε-transitions

11. NFA & DFA What is the relationship b/t NFAs and DFAs?

12. NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages

13. NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA

14. NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA If NFA has N states, up to 2 N states in DFA

15. NFA & DFA What is the relationship b/t NFAs and DFAs? NFAs and DFAs are equivalent Accept same languages Standard transformation from NFA to DFA: Create new state for each equivalence class in NFA If NFA has N states, up to 2 N states in DFA Why use both?

16. Managing Non-Determinism Approaches to non-determinism

17. Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative

18. Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative Look-ahead: Look ahead to see which choice to take

19. Managing Non-Determinism Approaches to non-determinism Backup/backtracking: At choice points, mark state, input If fail, return and try alternative Look-ahead: Look ahead to see which choice to take Parallelism: At choice points, consider all paths in parallel Basically the NFA  DFA conversion process

20. Recognition in NFAs Two standard approaches:

21. Recognition in NFAs Two standard approaches: Transformation: Convert NFA  DFA Perform standard DFA recognition

22. Recognition in NFAs Two standard approaches: Transformation: Convert NFA  DFA Perform standard DFA recognition Search: Explicitly model recognition as state-space search Perform backtracking as necessary

23. Recognition in NFA For strings in the language:

24. Recognition in NFA For strings in the language: There exists some state sequence leading to a final state

25. Recognition in NFA For strings in the language: There exists some state sequence leading to a final state Not all paths necessarily lead to an accept state For strings not in the language:

26. Recognition in NFA For strings in the language: There exists some state sequence leading to a final state Not all paths necessarily lead to an accept state For strings not in the language: No paths lead to an accept state

27. NFA Recognition as Search Search problem:

28. NFA Recognition as Search Search problem: Start search-state (start node, beginning of input)

29. NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input)

30. NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions

31. NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions Path cost

32. NFA Recognition as Search Search problem: Start search-state (start node, beginning of input) Goal test (final node, end of input) Successor function Transitions Path cost Use standard search algorithms e.g. Depth-first, breadth first, A*

34. Example

43. Breadth-first Search

44. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

45. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

46. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

47. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata

48. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata E.g. Concatenation

49. Equivalences and Operations FSAs equivalent to Regular Languages (Regex, etc) Can be shown by induction Demonstrate primitive regex ops imitated by automata E.g. Concatenation

50. Equivalences and Operations Closure: Union:

51. Equivalences and Operations Closure: Union:

52. Equivalences and Operations Closure: Union:

53. Finite-State Tools Toolkits: AT&T Finite-State Toolkit: http://www.research.att.com/~fsmtools/fsm NLTK: www.nltk.org CARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel

54. Finite-State Tools Toolkits: AT&T Finite-State Toolkit: http://www.research.att.com/~fsmtools/fsm NLTK: www.nltk.org CARMEL (ISI): /NLP_TOOLS/ml_tools/FST/carmel Functions: Convert Regex to FSA Convert FSA to Regex Convert NFA to DFA Perform operations: concatenation, disjunction, *, etc

55. Probabilistic Finite-State Automata (PFA)

56. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q

57. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R +

58. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R +

59. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R +

60. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R + s.t.

61. PFA Definition A Probabilistic Finite-State Automaton is a 7-tuple: A set of states Q An alphabet Σ A set of transitions: δsubset Q x Σ x Q Initial state probabilities: Q  R + Transition probabilities: δ  R + Final state probabilities: Q  R + s.t.

62. PFA Example Example Example due to F. Xia

63. PFA Example Example I(q 0 )=1 I(q 1 )=0 Example due to F. Xia

64. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 Example due to F. Xia

65. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 Example due to F. Xia

66. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) Example due to F. Xia

67. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) Example due to F. Xia

68. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) = 0.8 n *0.2 Example due to F. Xia

69. PFA Example Example I(q 0 )=1 I(q 1 )=0 F(q 0 )=0 F(q 1 )=0.2 P(q 0,a,q 1 )=1; P(q 1,b,q 1 ) =0.8 P(ab n ) = I(q 0 )*P(q 0,a,q 1 )*P(q 1,b,q 1 ) n *F(q 1 ) = 0.8 n *0.2 Example due to F. Xia

70. PFA PFA augments NFAs with probabilities on arcs

71. PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs)

72. PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths.

73. PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths. Tech. issue: Numerical underflow:

74. PFA PFA augments NFAs with probabilities on arcs Probability of a path through PFA is product of transition probabilities (and initial/final state probs) Probability of a string is the sum of accepting paths. Tech. issue: Numerical underflow: Replace product of probs with sum of log probs Tasks: Given an input, find most probable state sequence Given an input, find its probability etc (We’ll do these when we generalize to HMMs)

75. Weighted FSA Similar to PFA Arcs associated with weight (not probability)

76. Finite-State Transducers

77. FSTs Finite automaton that maps between two strings Automaton with two labels/arc input:output

78. FST Applications Tokenization Segmentation Morphological analysis Transliteration Translation Speech recognition Spoken language understanding

79. Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects

80. Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages

81. Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages FST as translator: Reads an input string and prints output string

82. Approaches to FSTs FST as recognizer: Takes pair of input:output strings Accepts if in language, o.w. rejects FST as generator: Outputs pairs of strings in languages FST as translator: Reads an input string and prints output string FST as set relator: Computes relations between sets

83. FSTs, Formally

84. FSTs & Regular Relations FSAs: equivalent to regular languages

85. FSTs & Regular Relations FSAs: equivalent to regular languages FSTs: equivalent to regular relations Sets of pairs of strings

86. FSTs & Regular Relations FSAs: equivalent to regular languages FSTs: equivalent to regular relations Sets of pairs of strings Regular relations: For all (x,y) in Σ 1 x Σ 2, {(x,y)} is a regular relation The empty set is a regular relation If R 1,R 2 are regular relations, R 1  R 2, R 1 U R 2 and R 1 * are regular relations

87. Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to !

88. Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to ! Composition: If T 1 is a transducer from I 1 to O 2 and T 2 is a transducer from O 2 to O 3, then T 1 T 2 is a transducer from I 1 to O 3

89. Regular Relation Operations Regular relations are closed under: Inversion: Switching input and output labels If T maps from I to O, T -1 maps from O to ! Composition: If T 1 is a transducer from I 1 to O 2 and T 2 is a transducer from O 2 to O 3, then T 1 T 2 is a transducer from I 1 to O 3

90. CARMEL

91. Defining an FSA/FST final-state (from-state (to-state “input-sym” “outputsym”? weight?)*) ….. First line is final state Lines define state transitions input-sym (in quotes) required output-sym, weight optional *e* is the representation for ε patas:/NLP_TOOLS/ml_tools/fsm/carmel/latest/bin/carmel

92. FSA Example q4 (q0 (q1 “b”)) (q1 (q2 “a”)) (q2 (q3 “a”)) (q3 (q2 *e*)) (q3 (q4 “!”))

93. FST Example q1 (q0 (q0 “a” “A”)) (q0 (q1 “b” “b”)) (q1 (q1 “b” “B”))

94. WFST Example S (S (S “they” “PRO” 1.0)) (S (S “can” “AUX” 0.99)) (S (S “can” “VERB” 0.01)) (S (S “fish” “NOUN” 0.7)) (S (S “fish” “VERB” 0.3))

95. Basic Functions carmel –k N mywfst Prints top N most probable paths through mywfst carmel –Ok N mywsft Print top N most probable outputs from mywfst carmel fst1 fst2 Composes fst1 and fst2 cat input_file | carmel –sli mywfst Converts 1 st line of input_file to fst & composes (left) -b flag: batch processes & prints best path

96. HW #2 Finite-State Automata Goals: Build Finite-State Automata to represent Regex Employ modern FST package to build FSA acceptor Implement FSA acceptor yourselves