Presentation on theme: "Backtracking Algorithmic Complexity Attacks Against a NIDS"— Presentation transcript:
1 Backtracking Algorithmic Complexity Attacks Against a NIDS Randy Smith, Cristian Estan, Somesh JhaUniversity of Wisconsin–Madison
2 Algorithmic Complexity Attacks Vulnerable algorithm: algorithm whose worst case differs from typical case. The larger the difference, the more vulnerable the algorithm.Examples:AlgorithmAverageWorstQuicksortO(n log n)O(n2)Hash lookupconstantO(n)
3 Algorithmic Complexity Attacks Algorithmic Complexity Attack – an attacker induces worst-case behavior in a vulnerable algorithm.Common observable effect is denial of service.Crosby and Wallach: induced worst-case behavior in hash function implementations.“Algorithms are now part of the attack surface” (Crosby and Wallach, 2003)
4 Are NIDS vulnerable? NIDS and IPS are ubiquitous, but… Do they contain vulnerable algorithms? Can they be exploited?YES! Only need 1 packet every 3 seconds.
5 Evading a NIDS Attacker’s Goal: Evade NIDS Two attack vectors in an evasion attempt:1st—alg. complexity attack targeting the NIDS2nd—true attack targeting the networkEffect of an algorithmic complexity attack:(NIDS) Packets enter network unexamined(fail-closed IPS) Packets are dropped
6 Main results In Snort, vulnerability in rule-matching worst-case vs. typical case: 6 orders of magnitude.“Backtracking Attack”Easily exploitable through packet payloadsImproved rule-matching algorithm limits running time differences to within 1 order of magnitude.
11 Inducing Backtracking attacks P1,P2,P3,P4 match in 3 positions eachP5 never matchesalert tcp $EXT_NET any -> $HOME_NET 99(msg:”ReelAudio jukebox exploit”;content:”fmt=”; //P1pcre:”/^(mp3|ogg)/”,relative; //P2content:”player=”; //P3pcre:”/.exe|.com/”,relative; //P4content:”overflow”,relative; //P5sid:5678)Leads to excessive packet traversals!fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exefmt=acc player=default fmt=mp3 rate=14kbps
12 Matching the malicious packet alert tcp $EXT_NET any -> $HOME_NET 99(msg:”AudioPlayer jukebox exploit”;content:”fmt=”; //P1pcre:”/^(mp3|ogg)/”,relative; //P2content:”player=”; //P3pcre:”/.exe|.com/”,relative; //P4content:”overflow”,relative; //P5sid:5678)P2P3P4P4P5P4P5P5P5P5P5P5fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe
13 Are real rules vulnerable? Rule numberProcessing (s/GB)SlowdownSame protoAll traffic3682 (SMTP)30,933,874232,936X1,501,644X2611 (Oracle)6,220,76856,296X301,979X1382 (IRC)1,956,858134,031X94,993X2403 (NetBIOS)357,777490X17,368X1755 (IMAP)89,181444X4,329X
14 Safer backtrackingMemoization: maintain a table of subproblem “answers”; never evaluate a predicate twice at the same starting payload offsetalert tcp $EXT_NET any -> $HOME_NET 99(msg:”AudioPlayer jukebox exploit”;content:”fmt=”; //P1pcre:”/^(mp3|ogg)/”,relative; //P2content:”player=”; //P3pcre:”/.exe|.com/”,relative; //P4content:”overflow”,relative; //P5sid:5678)Identify constrained predicate sequencesMonotone memoization: don’t re-evaluate monotone predicates that have been evaluated at lower offsets
15 Reductions in processing cost 41118P5P4P2P3P5P4P2P3P271421P3283542P4P4P4465054P5P5P5P5P5P5P5P5P5fmt=mp3fmt=mp3fmt=mp3player=player=player=.exe.exe.exe
17 Slowdown factor w.r.t. same protocol Measurement resultsRule numberSlowdown factor w.r.t. same protocolBeforew/ Memo+3682 (SMTP)232,936X0.95X2611 (Oracle)56,296X1.57X1382 (IRC)134,031X6.00X2403 (NetBIOS)490X0.17X1755 (IMAP)444X0.46X
18 Live experiment topology Background TrafficAC AttackTrue Attack
19 Live experiment Background Traffic @ 10Mbps AC Attack Targets Snort SMTP rule 3682Directed at sendmail serverTrue Attack: NIMDA300 exploit attempts, sent 1 byte per second.New exploit started every second.
20 Live experiment results Attack DescriptionExploitsDetectedRequired Rate (kbps)Control (No attack)300/300--2 packets every 60 s.220/3000.41 packet every 5 s.4/3002.41 packet every 3 s.0/3004.020 packets initially0.8
21 ConclusionsNIDS operation is complex. Many opportunities for vulnerable algorithms.In Snort, rule-matching is vulnerable and can be exploited by an attacker.Memoization, along with other semantics-preserving operations, significantly reduces vulnerability.Other vulnerable algoritms exist.
22 Backtracking Algorithmic Complexity Attacks Against a NIDS Thank you.