1. MITACS-PINTS Prediction In Interacting Systems Project Leader : Michael Kouriztin

2. Network Security Search and Rescue Defence Investing Environmental Monitoring Fraud Detection Nonlinear Filtering Modeling Observing

3. Countering Espionage in Cyber-Warfare: Detecting Stealthy Portscans Jarett Hailes Surrey Kim Michael Kouritzin Wei Sun 5th MITACS IT-Theme Meeting October 19, 2003

4. Outline Problem of detecting stealthy port scans Simulations Clustering model & Filtering equation Computer workable approximation

5. Port scanning: method for discovering network vulnerabilities Reconnaissance stage of a hacker attacks. “Probes” target network via sending packets Port Scanning

6. Stealthy Techniques Slow Scans : to obscure the attack, an attacker could do the scan very slowly. : Multiple source scans : using multiple sources using multiple sources Idle scanning : bouncing scans from dumb "zombie" host. Spoofed Source IP : sending large number of packets with only one as the real source.

7. Current Solutions Existing solutions are : Prone to false alarms and miss detects Easily foiled by new scanning techniques Insufficient information (black and white solutions) Cause for unacceptable downtime Extensive human management required

8. End goal : to probe 30 ports on 10 hosts. Scanning Technique: Half-open SYN Scan and t Scanning Technique: Half-open SYN Scan and to obscure the attack : may use multiple computers (i.e. source IP addresses). may use multiple computers (i.e. source IP addresses). may use may use dumb "zombie" host to bounce scans. slows down scan rate slows down scan rate sends 300 packets in random order sends 300 packets in random order Example

9. Detection Problem To detect whether or not there is a port scanner present. Via Filtering and Bayesian Model selection Only SYN packets are considered (i.e. No packet flag information used yet) Assume the traffic rates for target hosts

11. Portscan Detector Results

13. Traffic Summary Signal to Noise Ratio 0 100000 200000 300000 400000 500000 600000 700000 800000 900,000 1,000,000 Number of Packets Normal Network Traffic Packets : 923,424 Port Scanner Packets : 428

14. Challenges and Future Work Enormous State Space : Localization : IP spoofing : Stealthy hacker scans all ports certain number of times, decreasing scan rate and using to reduce suspicion

15. To obscure the attack, an attacker could do the scan very slowly. Unless the target system is normally idle (in which case one packet to a non-listening port is enough for the admin to notice, not a likely real world situation), it is possible to make the delay between ports large enough for this to be likely not recognized as a scan. A way to hide the origin of a scan, while still receiving the information, is to send a large amount (say, 999) of spoofed "port scans", and only one scan from the real source address. Even if all the scans (1000 of them) are detected and logged, there's no way to tell which of the source addresses is real. All we can tell is that we've been port scanned. Idle scanning - a clever side-channel attack allows for the scan to be bounced off a dumb "zombie" host. Intrusion detection system (IDS) reports will finger the innocent zombie as the attacker. Stealthy Techniques

16. Clustering Model Model packet traffic as marked point process with marks, i.e. packet headers – (Destination, Source, Flags), in Network traffic mixture of two types Normal traffic rate: Normal traffic rate: Malicious & stealthy traffic rate: Malicious & stealthy traffic rate: depends on all previous scans depends on all previous scans Hacker can have stealthy strategy – e.g. scan network host port over so many days Hacker can have stealthy strategy – e.g. scan network host port over so many days Which packets are due to port scans? )(u  ),( t u  S ),( t u 

17. Filtering Approach New Nonlinear Filtering Approach Provides probabilistic information Provides probabilistic information Other bw Other bw Choose acceptable ratio of miss detect to false alarm Choose acceptable ratio of miss detect to false alarm Asymptotically optimal Asymptotically optimal

18. Normal Traffic Poisson measure – randomly distributes points across marks, rates, time Number of points in disjoint regions independent Number of points in disjoint regions independent Desired expected number of points everywhere Desired expected number of points everywhere Normal Traffic = Observation noise that must be “filtered out’’    i tvASVU iii tvA ],0[],0[),,( 1 111 1]),0[],,0[,(  )()(1),( ],0[),0[ 1)](,0[1 dsddutAY tA u       

19. Port Scanning Buried in this noise is the signal = count of Port Scan packets at various marks Port Scan signal or cluster: Observation = observed traffic: )()(1),( ],0[),0[ 2)],(,0[ dsddutA tA u s         ),(),(),( 1 tAtAYtAY 

20. Simulation Example End goal : to probe 30 ports on 10 target hosts. Normal Traffic Rates : Cluster dependent scanning rate : Host123456789101.000.0010.0051.011.012.02.00.020.010.02

21. Bayesian Model Selection Detecting whether or not there is anomalous traffic on observed computer system. Bayes factor satisfies

22. Nonlinear Filtering Goal: Approximate Idea: Choose that does not depend on Choose that does not depend on  Then, calculations are simple Then, calculations are simple Reference probability measure method There is artificial probability Q where is Poisson measure with intensity There is artificial probability Q where  is Poisson measure with intensity P(A) = L(t) Q(A) for events A occuring by t; L is martingale P(A) = L(t) Q(A) for events A occuring by t; L is martingale  )),(|(),(tssYAPtA t  )(u

23. Filtering Equation Unnormalized conditional port scan distribution Then, we approximate Real-world conditional probability satisfies  )),(|)()),(((),(tssYtLtfEtf Q    (1)

24. Workable Approximation (I) Under general conditions and after modest work we find and prove: Under general conditions and after modest work we find and prove: In probability on pathspace for each fixed observation Y, i.e. in quenched sense. Here

25. Workable Approximation (II) Equation (1) is still unworkable so we let... S Ex: Suppose S is 1-dimensional... Number of Packets in Each Cell

26. Workable Approximation (III) Substituting into (1) and approximating counting measures on S with counting measures on with at most L N particles, one finds Here

27. Workable Approximation (IV) We also discretize amplitude to yield Markov chain approximation Suppose is sequence satisfying Let

28. Workable Approximation (V) Our Markov chain solves The approximation is given by:

29. Characterizing and Tracking the signal