1 Faster algorithms for string matching with k mismatches Adviser : R. C. T. Lee Speaker: C. C. Yen Journal of Algorithms, Volume 50, Issue 2, February 2004, Pages Amihood Amir, Moshe Lewenstein and Ely Porat

2 String matching with k mismatches Input: A text T with length n, a pattern P with length m and a mismatching threshold k Output: Each sub-string S of T where HD(S,P)

3 The basic idea of following algorithms The authors discuss the number of distinct symbols in the pattern and design algorithms to solve the problems efficiently in different cases. Example: P = ACAABD The number of distinct symbols of P is 4.

4 Three cases of the number of distinct symbols in pattern The paper discusses the following three cases; k is the maximal number of mismatches allowed. 1.There are at least 2k distinct symbols. 2.There are less than distinct symbols. 3.The number of distinct symbols is between and 2k.

5 Case 1: At least 2k distinct symbols There are two stages in the algorithm. 1. Marking Identify potential starts of the pattern and do a crude pruning of the potential candidates. 2. Verification Verify which of the potential candidates is indeed a pattern which occurs. In this case, the algorithm takes linear time to solve string matching with k mismatches problem.

6 The basic idea of this paper is as follow 1)Let A={a 1 a 2… a 2k } be a set of distinct alphabets appearing in P. 2)Let P be the shortest prefix of P containing A. 3)Let the length of P be C. 4)Let S be a substring of T of length C. 5)Suppose among the 2k distinct alphabets in A which also appear in S, there are d matches between Pand S, as shown below 6)Then, obviously, among 2k locations in P,there are 2k-d mismatches. 7)If, then, we may ignore S totally. C d matches S P

7 But, how can we determine d ? We may use a position table

8 Marking stage of Case1 Let{a 1…., a 2k }be 2k different alphabet symbols appearing in the pattern and let i j be the smallest index in the pattern where a j appears,j=1….,2k. Create a position table S 1 … S 2k to represent distinct symbols in pattern P and pos 0 … pos 2k are their first appearance locations on P. Example symbolsACBD pos0134 S0S0 S1S1 S2S2 S3S3 pos 0 pos 1 pos 2 pos 3 T = ACBBDACTADIKQDABD…. = T 0 … T n-1 P = ACABDAE k =

9 We need scan the text T for each t i,, if we can find a j,, such that t i =s j, add 1 to location i - pos j of an array X. If i – pos j is less than 0, we ignore it. X is an array with size n and all elements of X are 0 initially. 4310pos DBCA symbols S0S0 S1S1 S2S2 S3S3 pos 0 pos 1 pos 2 pos 3 S 0 … S 3 represent 2k distinct symbols in pattern P and pos 0 … pos 3 are their first appearance locations on P. T = ACBBDACTADIKQDABD…. = T 0 … T n-1 P = ACABDAE k = X = ….

10 After the scanning is completed, we obtain the following array : X= For every X(a)=b, we know that there are b matches 2k distinct character between T(a, a+c-1) and P(0, c-1). There are at least 2k- b mismatches.Since b k. We may ignore T(a,a+c-1) in our case, since X= We need to examine only T(0,4) and T(5,9).We ignore all other substrings

11 Lemma 1 For Case 1, let n denote the length of text and k be maximal number of mismatches allowed. There are at most n/k candidate locations. Proof : The total number of addition to the X array is at most n because the algorithm tests T(i), i=1,2….n. Let the number of locations whose numbers are larger than k be a Then

12 Through Lemma 1, we know that at most n/k candidate locations remain. But not all candidate locations are starting points of matches with k maximal number of mismatches. P = ACABDAE T = ACBBDACTADIKQDABD…. X = …. There are four other mismatches, so the candidate location is not a starting point of match with k maximal number of mismatches. Take T(5) as an example:

13 We must verify which candidate locations are starting points of matches with k maximal number of mismatches.

14 Verification stage of Case1 The authors use the Kangaroo Method to verify whether a location has k maximal number of mismatches in O(k). T = ABCCABDADBDETADBAADFDAAEERDXTDADCT… P = ETBDBCCDFDC We shall not elaborate on this method because it was presented before

15 Time complexity of Case 1 We take O(n) time in marking stage, where n is the length of the text. According to Lemma 1, we have at most n/k candidate locations. Using Kangaroo method, we take O(k) time to verify a remained candidate location. Thus, we take O(n) time for the verification stage.

16 Case 2: Less than distinct symbols We can use the Boolean Convolution method to solve the problem for this case.

17 Thus it is obvious that Hamming distance can be found by convolution Let A=abac and B=acdc For this case HD(A,B)=2 Convolution a b a c c d c a matches HD(A,B)=2

18 Using Fast Fourier Transforms (FFT), Boolean Convolution can be done in O(nlogm). Our alphabet size is We take times to solve the problem for Case 2.

19 Case 3: The number of distinct symbols is between and 2k Definition: frequent symbol: A symbol appears in the pattern at least times. k = 2,, P = baccdbdd d is a frequent symbol. Example

20 Two Sub-cases of Case 3 Case3-1 There are at least frequent symbols in the pattern. Case3-2 There are less than frequent symbols in pattern.

21 Case 3-1:at least frequent symbols There are two stages in the algorithm for this case. (1)Marking stage Identify potential starts of the pattern and do a crude pruning of the potential candidates. (2)Verification stage Verify which of the potential candidate is indeed a pattern which occurs. Verification stage will be done by Kangaroo Method.

22 Example Let P = ABCAABBDBAA and k = 4 There are 4 ( 4 is between and 2k) distinct symbols in P and A, B are frequent symbols. There are 2 (= )frequent symbols. Marking stage of Case 3-1 We pick arbitrarily frequent symbols and convert this problem to mismatch problem with dont care. T = ABCABDCABBCFADDABC

23 Mismatch problem with dont care Input: A text T with length n and a pattern P with length m. where g are the characters in the pattern which are not dont care symbols. and the rest are Φ(dont care). Output: The numbers of mismatches between pattern and each sub-string of T with length m. Only mismatches of the g pattern characters are counted. The number of mismatches ABΦAABBΦBAΦ ABCABDCABBCFADDABD T = P = 4 ABΦAABBΦBAΦ

24 Mismatch problem with dont care Input: A text T with length n and a pattern P with length m. where g are the characters in the pattern which are not dont care symbols. and the rest are Φ(dont care). Output: The numbers of mismatches between pattern and each sub-string of T with length m. Only mismatches of the g pattern characters are counted. The number of mismatches ABΦAABBΦBAΦ ABCABDCABBCFADDABD T = P = 47 ABΦAABBΦBAΦ

25 Mismatch problem with dont care Input: A text T with length n and a pattern P with length m. where g are the characters in the pattern which are not dont care symbols. and the rest are Φ(dont care). Output: The numbers of mismatches between pattern and each sub-string of T with length m. Only mismatches of the g pattern characters are counted. The number of mismatches ABΦAABBΦBAΦ ABCABDCABBCFADDABD T = P = 477 ABΦAABBΦBAΦ

26 Mismatch problem with dont care Input: A text T with length n and a pattern P with length m. where g are the characters in the pattern which are not dont care symbols. and the rest are Φ(dont care). Output: The numbers of mismatches between pattern and each sub-string of T with length m. Only mismatches of the g pattern characters are counted. The number of mismatches ABΦAABBΦBAΦ ABCABDCABBCFADDABD T = P = 2477 ABΦAABBΦBAΦ

27 Mismatch problem with dont care can be solved in (Amir et, 1997), where n is the length of text T, m is the length of pattern P and g are the characters in the pattern which are not dont care symbols.

28 All locations with at most k mismatches of frequent symbols are our candidate locations where matches with k maximal number of mismatches start. The number of mismatches ABCABDCABBCFADDABD T = P = ABΦAABBΦBAΦ k = 4 Example

29 Lemma 2 for Case 3-1 Let {a 1,….,a }be frequent symbols. Then there exist in the text at most locations where there is a pattern occurrence with no more than k errors Proof The total number of mark is at most n because the algorithm tests T(i), i=1,2….n. Let the number of locations which have marks larger than k be a Then

30 We convert marking stage to mismatch problem with dont care and take to solve mismatch problem with dont care problem. According to lemma 2 for Case3-1, there are candidate locations and we take O(k) time to verify one candidate location. Verification stage for Case3-1 takes time.

31 Case 3-2:less than frequent symbols First, we can check the number of mismatches by using convert all frequent symbols to Φ (dont care symbol). Let P = ABCAABGDBAA and k = 5 There are 5 ( 5 is between and 2k) distinct symbols in P and A are frequent symbols. There are 1 (< )frequent symbols. T = ABCABDCABBCFADDABC Example

32 Two cases are discussed after we convert all frequent symbols to Φ There are less than 2k remaining symbols There are at least 2k remaining symbols.

33 Case3-2-1 There are less than 2k remaining symbols There are less than 2k remaining symbols and the rest are dont care symbols. Finding mismatches of remaining symbols can be solved as a mismatch problem with dont care and takes time. T = ABCABDCABBCFADDABCΦBCΦΦBGDBΦΦ P = mismatches of remaining = symbols 3 ΦBCΦΦBGDBΦΦ

34 Case3-2-1 There are less than 2k remaining symbols There are less than 2k remaining symbols and the rest are dont care symbols. Finding mismatches of remaining symbols can be solved as a mismatch problem with dont care and takes time. T = ABCABDCABBCFADDABCΦBCΦΦBGDBΦΦ P = mismatches of remaining = symbols 3 ΦBCΦΦBGDBΦΦ 5

35 Case3-2-1 There are less than 2k remaining symbols There are less than 2k remaining symbols and the rest are dont care symbols. Finding mismatches of remaining symbols can be solved as a mismatch problem with dont care and takes time. T = ABCABDCABBCFADDABCΦBCΦΦBGDBΦΦ P = mismatches of remaining = symbols 3 ΦBCΦΦBGDBΦΦ 56

36 Case3-2-1 There are less than 2k remaining symbols There are less than 2k remaining symbols and the rest are dont care symbols. Finding mismatches of remaining symbols can be solved as a mismatch problem with dont care and takes time. T = ABCABDCABBCFADDABCΦBCΦΦBGDBΦΦ P = mismatches of remaining = symbols 3 ΦBCΦΦBGDBΦΦ 564

37 Case3-2-1 There are less than 2k remaining symbols There are less than 2k remaining symbols and the rest are dont care symbols. Finding mismatches of remaining symbols can be solved as a mismatch problem with dont care and takes time. T = ABCABDCABBCFADDABCΦBCΦΦBGDBΦΦ P = mismatches of remaining = symbols 3 ΦBCΦΦBGDBΦΦ 5644

38 All locations which have less than k mismatches of all frequent symbols and remaining symbols are matches which we want.

39 Conclusion: The problem for Case can be solved in time

40 Case3-2-2 There are at least 2k remaining symbols There are two stages in algorithm for this case. (1)Marking stage Identify potential starts of the pattern and do a crude pruning of the potential candidates. (2)Verification stage Verify which of the potential candidates is indeed a pattern which occurred. Verification stage will be done by Kangaroo Method.

41 Marking stage of Case We pick arbitrarily 2k remaining symbols and convert all symbols to Φ(dont care symbols) except 2k remaining symbols which we picked. Marking stage of Case3-2-2 can be solved as mismatch problem with dont care in time.

42 Conclusion: The problem for Case can be solved in time

43 Thank you