1 Fast and Memory-Efficient Regular Expression Matching for Deep Packet Inspection Fang Yu Microsoft Research, Silicon Valley Work was done in UC Berkeley, jointly with Zhifeng Chen (Google Inc); Yanlei Diao (Umass, Amherst); T. V. Lakshman (Bell Labs); Randy H. Katz (UC Berkeley)

Regular Expressions Flexible way to describe pattern –Example: for detecting yahoo messenger traffic ^(ymsg|ypns|yhoo).?.?.?.?.?.?.? [lwt].*\xc0\x80 Used in many payload scanning applications –L7-filter: protocol identifiers –Bro: intrusion patterns –SNORT: No regular expression in April out of 4867 intrusion rules contain regular expressions as of Jan

Challenges Features specific to packet scanning applications Large set of patterns, order of 100s or 1000s SnortL7-filterXML filtering # of regular expressions analyzed , ,000 % of patterns with wildcards “., +, ?, *”74.9%75.7%50% -100% Average # of wildcards per pattern % of patterns with class “[ ]”31.6%52.8%0 Average # of classes per pattern % of patterns with length restrictions on classes or wildcards 56.3%21.4% 00 3

Design Space 4 Automata-based Approaches DFA-basedNFA-based Patterns (A|B)C and (A|D)E A group of states can be activated simultaneously Only one state is activated High percentage of wildcards  NFA-based approaches can be slow, sometimes less than 1Mb/s Repeated ScanOne Pass Scan Start scanning from one position, if no match, start again at the next position Good for parsers Packets may not contain any patterns No guarantee of high speed Scan the input only once Fast and deterministic throughput Add.* before patterns Some patterns generate very large DFA m Individual DFA for m patterns One composite DFA for m patterns O(m) processing complexity for each input character O(1) processing complexity for each input character Rewrite techniques to reduce memory usage Make DFA-based approach feasible Contributions Selectively group patterns into k groups (e.g., k=3) Avoid exponential memory growth Further speed up matching process (Space Problem)

DFA Sizes of Regular Expressions Pattern featuresExample# of states % of patterns Average # of states 1) Explicit strings with k characters^ABCD.*ABCD k+125.1% ) Wildcards^AB.*CD.*AB.*CD k % ) Patterns with ^, a wildcard, and a length restriction j ^AB.{j+}CD ^AB.{0, j}CD ^AB.{j}CD O(k*j)44.7% ) Patterns with ^, a class of characters overlaps with the prefix, and a length restriction j ^A+[A-Z]{j}DO(k+j 2 ) j~ % ) Patterns with a length restriction j, where a wildcard or a class of characters overlaps with the prefix.*AB.{j}CD.*A[A-Z]{j+}D O(k+2 j ) j~ %> Typical patterns in network payload scanning applications Rewrite Rule 2 Focus of this talk Rewrite Rule 1

Design Considerations Completeness of matching results for one pattern –Complete matching Report all the possible substrings E.g., a pattern ab* and an input abbb –Four possible matches, i.e., a, ab, abb, and abbb –Non-overlapping matching Common practice: left-most longest match, shortest match results In most payload scanning applications, for one pattern, reporting non-overlapping matching result is sufficient 6

Patterns with Exponential DFA Sizes Often for detecting buffer overflow attempts, e.g.,.*AUTH\s[^\n]{100} DFA needs to remember all the possible AUTH\s –A second AUTH\s can either match [^\n]{100} or be counted as a new match of the start of the pattern AUTH\s –Generate a DFA of >100,000 states Can’t be efficiently processed by an NFA-based approach either 7 AUTH\s[\^n] 100 states ε NFA for.*AUTH\s[^\n]{100} Input AUTH\sAUTH\s AUTH\s\s AUTH\s\s\s …

Rewriting Intuition Only the first AUTH\s matters –If there is a ‘\n’ within the next 100 bytes None of the AUTH\s matches the pattern –Otherwise, the first AUTH\s and the following characters have already matched the pattern  Rewrite the pattern to: ([^A]|A[^U]|AU[^T]|AUT[^H]|AUTH[^\s]|AUTH\s[^\n]{0,99}\n)*AUTH\s[^\ n]{100} generates a DFA of only 106 states This rewritten pattern –Report different numbers of matches from the original pattern in identifying complete matches –Equivalent in identifying non-overlapping patterns 8

Rewriting Effect on the SNORT Rule Set 9 Pattern featuresExample# of states % of patterns Average # of states 1) Explicit strings with k characters^ABCD.*ABCD k+125.1% ) Wildcards^AB.*CD.*AB.*CD k % ) Patterns with ^, a wildcard, and a length restriction j ^AB.{j+}CD ^AB.{0, j}CD ^AB.{j}CD O(k*j)44.7% ) Patterns with ^, a class of characters overlaps with the prefix, and a length restriction j ^A+[A-Z]{j}DO(k+j 2 ) O(k+j) 5.11% ) Patterns with a length restriction j, where a wildcard or a class of characters overlaps with the prefix.*AB.{j}CD.*A[A-Z]{j+}D O(k+2 j ) O(k+j) 6.27%>2 214 v

Rewriting Effect on the SNORT Rule Set Created scripts to automatically rewrite patterns 10 Type of RewriteRule SetNumber of Patterns Average Length Restriction DFA Reduction Rate Rewrite Rule for Quadratic case Snort17370>98% Bro000 Rewrite Rule for Exponential Case Snort19344>99% Bro >99% –After rewriting, patterns in SNORT and Bro can be compiled into DFAs

Design Choices 11 Automata-based Approaches DFA-basedNFA-based Repeated ScanOne Pass Scan m Individual DFA for m patterns One composite DFA for m patterns O(m) processing complexity for each input character O(1) processing complexity for each input character Rewrite techniques to reduce memory usage Make DFA-based approach feasible Contributions Selectively group patterns into k groups (e.g., k=3) Further speedup matching process Avoid exponential memory growth

State Explosion Problem Randomly adding patterns from the L7-filters into one DFA 12

Interactions of Regular Expressions Some patterns generate DFA of exponential sizes –E.g., A DFA for pattern.*AB.*CD and.*EF.*GH 13

Grouping Algorithms –Fixed local memory limitation ( NPU or multi-core architectures) Compute pair-wise interactive results, form a graph Keep adding patterns until reaching limit –Pick a pattern with the fewest interactions to the new group –Fixed total memory limitation (General single-core CPU architecture) First compute the DFA of individual patterns and compute the leftover memory size Distribute the leftover memory evenly among ungrouped expressions 14

Experimental Setup Regular expression pattern sets –Linux application layer filer (L7-filter): 70 regular expressions –Pattern sets from Bro intrusion detection systems HTTP related patterns: 648 patterns Payload related patterns: 223 patterns Packet traces: –MIT dump: with viruses and worms –Berkeley dump: normal traffic Scanners: –Generated one pass scanning DFA scanner –A NFA-based scanner Pcregrep –A repeated scanning DFA parser generated by flex 15

Grouping Results for Patterns in L7-filter (70 patterns) Total DFA state Limit Groups Compilation Time (s) Results of grouping algorithms for fixed total memory Sum of individual DFAs No extra memory cost 70/12=5.83 times less processing per character 6.83MB of memory 70/3=23.3 times less processing per character No grouping

Throughput Analysis 17 For Linux L7-filter (70 patterns) Using PCs with 3Ghz single core CPU and 4GB memory

Comparisons to Other Approaches Throughputs (Mb/s) Memory Consumption (KB) MIT dumpBerkeley dump Linux L7-filter (70 patterns) NFA DFA RP DFA OP 3 groups Bro HTTP (648 patterns) NFA DFA RP DFA OP 1 group Bro Payload (223 patterns) NFA DFA RP DFA OP 4 groups NFA—Pcregrep DFA RP – Flex generated DFA-based repeated scan engine DFA OP – Our DFA one pass scanning engine DFA OP is 48 to 704 times faster over the NFA implementation times faster than the commonly used DFA-based parser Use 2.6 to 8.4 times memory

Conclusions High speed regular expression matching scheme –Proposed two rewrite rules DFA-based approach is possible with our rewriting rules Can rewrite complicated patterns from our pattern sets In other pattern sets, there may be patterns not covered by our rewriting rules. –Developed grouping algorithm to selectively group patterns together Orders of magnitude faster than existing solutions –Can be applied to FPGA or ASIC based approaches as well 19