MITACS-PINTS Prediction In Interacting Systems Project Leader : Michael Kouriztin

Network Security Search and Rescue Defence Investing Environmental Monitoring Fraud Detection Nonlinear Filtering Modeling Observing

Countering Espionage in Cyber-Warfare: Detecting Stealthy Portscans Jarett Hailes Surrey Kim Michael Kouritzin Wei Sun 5th MITACS IT-Theme Meeting October 19, 2003

Outline Problem of detecting stealthy port scans Simulations Clustering model & Filtering equation Computer workable approximation

Port scanning: method for discovering network vulnerabilities Reconnaissance stage of a hacker attacks. “Probes” target network via sending packets Port Scanning

Stealthy Techniques Slow Scans : to obscure the attack, an attacker could do the scan very slowly. : Multiple source scans : using multiple sources using multiple sources Idle scanning : bouncing scans from dumb "zombie" host. Spoofed Source IP : sending large number of packets with only one as the real source.

Current Solutions Existing solutions are : Prone to false alarms and miss detects Easily foiled by new scanning techniques Insufficient information (black and white solutions) Cause for unacceptable downtime Extensive human management required

End goal : to probe 30 ports on 10 hosts. Scanning Technique: Half-open SYN Scan and t Scanning Technique: Half-open SYN Scan and to obscure the attack : may use multiple computers (i.e. source IP addresses). may use multiple computers (i.e. source IP addresses). may use may use dumb "zombie" host to bounce scans. slows down scan rate slows down scan rate sends 300 packets in random order sends 300 packets in random order Example

Detection Problem To detect whether or not there is a port scanner present. Via Filtering and Bayesian Model selection Only SYN packets are considered (i.e. No packet flag information used yet) Assume the traffic rates for target hosts

Portscan Detector Results

Traffic Summary Signal to Noise Ratio ,000 1,000,000 Number of Packets Normal Network Traffic Packets : 923,424 Port Scanner Packets : 428

Challenges and Future Work Enormous State Space : Localization : IP spoofing : Stealthy hacker scans all ports certain number of times, decreasing scan rate and using to reduce suspicion

To obscure the attack, an attacker could do the scan very slowly. Unless the target system is normally idle (in which case one packet to a non-listening port is enough for the admin to notice, not a likely real world situation), it is possible to make the delay between ports large enough for this to be likely not recognized as a scan. A way to hide the origin of a scan, while still receiving the information, is to send a large amount (say, 999) of spoofed "port scans", and only one scan from the real source address. Even if all the scans (1000 of them) are detected and logged, there's no way to tell which of the source addresses is real. All we can tell is that we've been port scanned. Idle scanning - a clever side-channel attack allows for the scan to be bounced off a dumb "zombie" host. Intrusion detection system (IDS) reports will finger the innocent zombie as the attacker. Stealthy Techniques

Clustering Model Model packet traffic as marked point process with marks, i.e. packet headers – (Destination, Source, Flags), in Network traffic mixture of two types Normal traffic rate: Normal traffic rate: Malicious & stealthy traffic rate: Malicious & stealthy traffic rate: depends on all previous scans depends on all previous scans Hacker can have stealthy strategy – e.g. scan network host port over so many days Hacker can have stealthy strategy – e.g. scan network host port over so many days Which packets are due to port scans? )(u  ),( t u  S ),( t u 

Filtering Approach New Nonlinear Filtering Approach Provides probabilistic information Provides probabilistic information Other bw Other bw Choose acceptable ratio of miss detect to false alarm Choose acceptable ratio of miss detect to false alarm Asymptotically optimal Asymptotically optimal

Normal Traffic Poisson measure – randomly distributes points across marks, rates, time Number of points in disjoint regions independent Number of points in disjoint regions independent Desired expected number of points everywhere Desired expected number of points everywhere Normal Traffic = Observation noise that must be “filtered out’’    i tvASVU iii tvA ],0[],0[),,( ]),0[],,0[,(  )()(1),( ],0[),0[ 1)](,0[1 dsddutAY tA u       

Port Scanning Buried in this noise is the signal = count of Port Scan packets at various marks Port Scan signal or cluster: Observation = observed traffic: )()(1),( ],0[),0[ 2)],(,0[ dsddutA tA u s         ),(),(),( 1 tAtAYtAY 

Simulation Example End goal : to probe 30 ports on 10 target hosts. Normal Traffic Rates : Cluster dependent scanning rate : Host

Bayesian Model Selection Detecting whether or not there is anomalous traffic on observed computer system. Bayes factor satisfies

Nonlinear Filtering Goal: Approximate Idea: Choose that does not depend on Choose that does not depend on  Then, calculations are simple Then, calculations are simple Reference probability measure method There is artificial probability Q where is Poisson measure with intensity There is artificial probability Q where  is Poisson measure with intensity P(A) = L(t) Q(A) for events A occuring by t; L is martingale P(A) = L(t) Q(A) for events A occuring by t; L is martingale  )),(|(),(tssYAPtA t  )(u

Filtering Equation Unnormalized conditional port scan distribution Then, we approximate Real-world conditional probability satisfies  )),(|)()),(((),(tssYtLtfEtf Q    (1)

Workable Approximation (I) Under general conditions and after modest work we find and prove: Under general conditions and after modest work we find and prove: In probability on pathspace for each fixed observation Y, i.e. in quenched sense. Here

Workable Approximation (II) Equation (1) is still unworkable so we let... S Ex: Suppose S is 1-dimensional... Number of Packets in Each Cell

Workable Approximation (III) Substituting into (1) and approximating counting measures on S with counting measures on with at most L N particles, one finds Here

Workable Approximation (IV) We also discretize amplitude to yield Markov chain approximation Suppose is sequence satisfying Let

Workable Approximation (V) Our Markov chain solves The approximation is given by:

Characterizing and Tracking the signal