1. CSC 382/582: Computer Security
Integrity Management
CSC 382/582: Computer Security

2. Models of Intrusion Detection
Misuse detection
You know what’s bad.
Attempt to detect bad items.
Anti-virus and anti-spyware tools.
Anomaly detection
You know what’s good.
Attempt to detect deviations from good state.
Host Intrusion Detection Systems (HIDS).
CSC 382/582: Computer Security

3. Theory of Malicious Code
Theorem 22-1: It is undecidable whether an arbitrary program contains a computer virus.
Proof:
Define virus v as TM program that copies v to other parts of the tape, while not overwriting any part of v.
Reduce to Halting Problem: T’ running code V’ reproduces V iff running T on V halts.
Theorem 22-2: It is undecidable whether an arbitrary program contains malicious logic.
CSC 382/582: Computer Security

4. CSC 382/582: Computer Security
Detecting Malware
Signature-based
Look for known patterns in malicious code.
Defeated by polymorphic viruses.
Smart scanning
Skips junk instructions inserted by poly engines.
Skips whitespace/case changes in macro viruses.
Decryption
Brute-forces simple XOR-based encryption.
Checks decrypted text against small virus sig to decide whether has plaintext or not.
CSC 382/582: Computer Security

5. CSC 382/582: Computer Security
Detecting Malware
Code Emulation
Execute potential malware on VM.
Scan VM memory after certain # iterations.
Watch instructions for decryptor profile.
Code Optimization.
Optimize away junk instructions and odd techniques used by polymorphic viruses.
CSC 382/582: Computer Security

6. CSC 382/582: Computer Security
Detecting Malware
Heuristics
Code execution starts in last section.
Suspicious code redirection.
Suspicious section ACLs or size.
Suspicious library routine imports.
Hard-coded pointers into OS kernel.
Neural Network Heuristics
IBM researchers trained neural net to recognize difficult polymorphic viruses.
Released in Symantec antivirus.
CSC 382/582: Computer Security

7. CSC 382/582: Computer Security
Detecting Malware
Behavior-based
Watch for known actions from malicious code.
Network access signature of worm.
Unexpected use of dangerous system calls.
Integrity Checking
Host-based Intrusion Detection System.
Record MAC, size, dates, ACL of files.
Periodically check for changes.
ex: Tripwire, AIDE
CSC 382/582: Computer Security

8. CSC 382/582: Computer Security
References
Ross Anderson, Security Engineering, Wiley, 2001.
Matt Bishop, Computer Security: Art and Science, Addison-Wesley, 2003.
William Cheswick, Steven Bellovin, and Avriel Rubin, Firewalls and Internet Security, 2/e, 2003.
Fred Cohen,
Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3/e, O’Reilly & Associates, 2003.
Cyrus Peikari and Anton Chuvakin, Security Warrior, O’Reilly & Associates, 2003.
Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003.
Ed Skoudis, Counter Hack Reloaded 2/e, Prentice Hall, 2006.
Peter Szor, The Art of Computer Virus Research and Defense, Addison-Wesley, 2005.
CSC 382/582: Computer Security