Fast and Memory-Efficient Regular Expression Matching for Deep Packet Inspection Publisher : ANCS’ 06 Author : Fang Yu, Zhifeng Chen, Yanlei Diao, T.V. Lakshman and Randy H. Katz Presenter : Yu-Hsiang Wang Date : 2010/11/17 1

Outline Introduction DFA Analysis for Individual Regular expression Regular Expression Rewrites Regular Expressions Grouping Evaluation results 2

Introduction A theoretical worst case study [14] shows a single regular expression of length n can be expressed as an NFA with O(n) states. When the NFA is converted into a DFA, it may generate O(Σ n ) states. (Σ : a finite set of input symbols, 2 8 symbols from the ASCII code) The processing complexity for each character in the input is O(1) in a DFA, but is O(n 2 ) for an NFA when all n states are active at the same time. 3

Introduction To handle m regular expressions, two choices are possible: - processing them individually in m automata : O(m) - compiling m regular expressions into a composite DFA : O(1) 4

Design Consideration Completeness of matching result: Pattern : ab* Input : abbb -Exhaustive Matching : a, ab, abb,abbb -Non-overlapping Matching : a (or abbb) left-most longest match, shortest match results DFA execution model for substring matching : patterns without ^ attached at the beginning. - Repeated search :Start scanning from one position, if no match, start again at the next position -One-pass search :.* is pre-pended to each pattern without ^ 5

DFA Analysis We use Exhaustive Matching and One-pass search Typical patterns in network payload scanning applications 6

Case 4 : DFA of Quadratic size if an input contains multiple Bs, the DFA needs to remember the number of Bs it has seen and their locations 7

Case 4 Rewrites Rewrite Rule(1) Rewriting is enabled by relaxing the requirement of exhaustive matching to that of non-overlapping matching the new pattern essentially implements non-overlapping left- most shortest match. Ex: ^SEARCH\s+[^\n]{1024}  ^SEARCH\s [^\n]{1024} input : SEARCH\s\s... \s aa... a number of states linear in j because it has removed the ambiguity for matching \s

Case 5 : DFA of Exponential Size we need to remember all possible effects of the preceding As as they may yield different results when combined with subsequent inputs. 9 AAB ABA BCD O BCD X

Case 5 : DFA of Exponential Size Often for detecting buffer overflow attempts :.*AUTH\s[^\n]{100} DFA needs to remember all the possible AUTH\s : DFA > 10000states -A second AUTH\s can either match [^\n]{100} or be counted as a new match of the start of the pattern AUTH\s Can’t be efficiently processed by an NFA-based approach either 10 AUTH\s[\^n] 100 states ε NFA for.*AUTH\s[^\n]{100} Input AUTH\sAUTH\s AUTH\s\s AUTH\s\s\s …

Case 5 Rewrites Only the first AUTH\s matters -If there is a ‘\n’ within the next 100 bytes None of the AUTH\s matches the pattern -Otherwise, the first AUTH\s and the following characters have already matched the pattern Rewrite the pattern to: ([^A]|A[^U]|AU[^T]|AUT[^H]|AUTH[^\s]|AUTH\s[^\n]{0,99}\n) *AUTH\s[^\n]{100} generates a DFA of only 106 states 11

Regular Expressions Grouping Some composite patterns generate DFA of exponential sizes interaction ： two patterns interact with each other if their composite DFA contains more states than the sum of two individual ones 12

Regular Expressions Grouping Multi-core architectures (ex: IXP 2800 NPU,16 processing unit) Goal : design an algorithm that divides regular expressions into several groups, so that one processing unit can run one or several composite DFAs. the size of local memory of each processing unit is quite limited -Compute pair-wise interactive results, form a graph -Pick a pattern with the fewest interactions to the new group -Keep adding patterns until reaching limit 13

Regular Expressions Grouping 14

Evaluation results Effect of Rule Rewriting -L7-filter: protocol identifiers (70 regular expression) -Bro: intrusion patterns (2781 regular expression) -SNORT: No regular expression in April out of 4867 regular expressions as of Jan

Evaluation results Effect of Grouping Multiple Patterns 16

Evaluation results 17