# Backtracking Algorithmic Complexity Attacks Against a NIDS

## Presentation on theme: "Backtracking Algorithmic Complexity Attacks Against a NIDS"— Presentation transcript:

Backtracking Algorithmic Complexity Attacks Against a NIDS
Randy Smith, Cristian Estan, Somesh Jha University of Wisconsin–Madison

Algorithmic Complexity Attacks
Vulnerable algorithm: algorithm whose worst case differs from typical case. The larger the difference, the more vulnerable the algorithm. Examples: Algorithm Average Worst Quicksort O(n log n) O(n2) Hash lookup constant O(n)

Algorithmic Complexity Attacks
Algorithmic Complexity Attack – an attacker induces worst-case behavior in a vulnerable algorithm. Common observable effect is denial of service. Crosby and Wallach: induced worst-case behavior in hash function implementations. “Algorithms are now part of the attack surface” (Crosby and Wallach, 2003)

Are NIDS vulnerable? NIDS and IPS are ubiquitous, but…
Do they contain vulnerable algorithms? Can they be exploited? YES! Only need 1 packet every 3 seconds.

Two attack vectors in an evasion attempt: 1st—alg. complexity attack targeting the NIDS 2nd—true attack targeting the network Effect of an algorithmic complexity attack: (NIDS) Packets enter network unexamined (fail-closed IPS) Packets are dropped

Main results In Snort, vulnerability in rule-matching
worst-case vs. typical case: 6 orders of magnitude. “Backtracking Attack” Easily exploitable through packet payloads Improved rule-matching algorithm limits running time differences to within 1 order of magnitude.

Outline Snort rule matching Inducing backtracking attacks
Countermeasures Measurement results Conclusion

Snort Rule Matching alert tcp \$EXT_NET any -> \$HOME_NET 99
content:”fmt=”; //P1 content:”player=”; //P3 content:”overflow”,relative; //P5 alert tcp \$EXT_NET any -> \$HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; pcre:”/^(mp3|ogg)/”,relative; //P2 pcre:”/.exe|.com/”,relative; //P4 sid:5678)

Snort Rule Matching Rule matches!
alert tcp \$EXT_NET any -> \$HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) Rule matches! fmt=acc player=default fmt=mp3 rate=14kbps

Matching the packet P1 alert tcp \$EXT_NET any -> \$HOME_NET 99
(msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 P3 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 P4 content:”overflow”,relative; //P5 sid:5678) P5 Rule matches! fmt=acc player=default fmt=mp3 rate=14kbps

Inducing Backtracking attacks
P1,P2,P3,P4 match in 3 positions each P5 never matches alert tcp \$EXT_NET any -> \$HOME_NET 99 (msg:”ReelAudio jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) Leads to excessive packet traversals! fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe fmt=acc player=default fmt=mp3 rate=14kbps

Matching the malicious packet
alert tcp \$EXT_NET any -> \$HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) P2 P3 P4 P4 P5 P4 P5 P5 P5 P5 P5 P5 fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

Are real rules vulnerable?
Rule number Processing (s/GB) Slowdown Same proto All traffic 3682 (SMTP) 30,933,874 232,936X 1,501,644X 2611 (Oracle) 6,220,768 56,296X 301,979X 1382 (IRC) 1,956,858 134,031X 94,993X 2403 (NetBIOS) 357,777 490X 17,368X 1755 (IMAP) 89,181 444X 4,329X

Safer backtracking Memoization: maintain a table of subproblem “answers”; never evaluate a predicate twice at the same starting payload offset alert tcp \$EXT_NET any -> \$HOME_NET 99 (msg:”AudioPlayer jukebox exploit”; content:”fmt=”; //P1 pcre:”/^(mp3|ogg)/”,relative; //P2 content:”player=”; //P3 pcre:”/.exe|.com/”,relative; //P4 content:”overflow”,relative; //P5 sid:5678) Identify constrained predicate sequences Monotone memoization: don’t re-evaluate monotone predicates that have been evaluated at lower offsets

Reductions in processing cost
4 11 18 P5 P4 P2 P3 P5 P4 P2 P3 P2 7 14 21 P3 28 35 42 P4 P4 P4 46 50 54 P5 P5 P5 P5 P5 P5 P5 P5 P5 fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe

Outline Snort rule matching Inducing backtracking attacks
Protecting against backtracking attacks Measurement results Conclusion

Slowdown factor w.r.t. same protocol
Measurement results Rule number Slowdown factor w.r.t. same protocol Before w/ Memo+ 3682 (SMTP) 232,936X 0.95X 2611 (Oracle) 56,296X 1.57X 1382 (IRC) 134,031X 6.00X 2403 (NetBIOS) 490X 0.17X 1755 (IMAP) 444X 0.46X

Live experiment topology
Background Traffic AC Attack True Attack

Live experiment Background Traffic @ 10Mbps AC Attack
Targets Snort SMTP rule 3682 Directed at sendmail server True Attack: NIMDA 300 exploit attempts, sent 1 byte per second. New exploit started every second.

Live experiment results
Attack Description Exploits Detected Required Rate (kbps) Control (No attack) 300/300 -- 2 packets every 60 s. 220/300 0.4 1 packet every 5 s. 4/300 2.4 1 packet every 3 s. 0/300 4.0 20 packets initially 0.8

Conclusions NIDS operation is complex. Many opportunities for vulnerable algorithms. In Snort, rule-matching is vulnerable and can be exploited by an attacker. Memoization, along with other semantics-preserving operations, significantly reduces vulnerability. Other vulnerable algoritms exist.

Backtracking Algorithmic Complexity Attacks Against a NIDS
Thank you.