1. Learning to Live with an Advanced Persistent Threat
John Denune
IT Security Director
University of California, San Diego

2. ACT Infrastructure services
Database Administration
Data Center
Active Directory
Security
Telecom
Decentralized
100 OU’s
800 OU Admins
Networking
ID Management
UNIX and Windows Support

3. ACT Security Policy and Compliance SSL Certs 9 Staff Firewall
Anti-virus and FDE
Forensics
VPN
Decentralized
100 OU’s
800 OU Admins
Patch Management
Incident Response
Vulnerability Assessment
Intrusion Detection

4. It’s not Opportunistic
What is an APT?
It’s not Opportunistic
Low level criminal activity
Spam, phishing, WAREZ
Off the shelf attacks, higher likelihood of AV detection

5. APT Varied Attacks Espionage Technical Targeted Patient Skilled
Corporate
State-Sponsored
You have something they want and they will spend a lot of time trying to get it
Months or years
Off the shelf, but also custom malware including zero-day
Extremely low detection rates
Technical, phishing, phone calls
Dropping infected USB drives in parking lot or keystroke loggers on lab keyboards
Often tied so some sort of espionage, either corporate to get insider information or state sponsored to get military info
Hacktivism to expose information or long term DDOS to make a point
Good old fashioned theft
Skilled
Theft
Hacktivism
Physical threats
Social Engineering

6. APT Lifecycle External Recon Initial Compromise Establish Foothold
Escalate Privileges
Internal Recon
Expand
Complete Mission

7. Initial Detection June 2012 Got lucky AV alert on ACT server.
OU Admins compromised through unrelated staff account on VPN
Only 1 of 4 pieces of malware detected
Changed password, rebuild servers
Happened again the following night with another unrelated VPN account
Also found several other computers in unrelated departments, also OU admins compromised
Third time password changes and re-used

11. Initial Detection June 2012 Got lucky AV alert on ACT server.
OU Admins compromised through unrelated staff account on VPN
Only 1 of 4 pieces of malware detected
Changed password, rebuild servers
Happened again the following night with another unrelated VPN account
Also found several other computers in unrelated departments, also OU admins compromised
Third time password changes and re-used

12. Pay attention to anti-virus alerts
Lesson #1
Pay attention to anti-virus alerts
Too many sysadmins view a detection as AV doing it’s job IF they even monitor at all
Modern malware loves company and almost always brings friends

13. Don’t (completely) rely on your anti-virus product
Lesson #2
Don’t (completely) rely on your anti-virus product
Low detection rates, especially for custom malware

14. Where possible, track IP’s instead of blocking them
Lesson #3
Where possible, track IP’s instead of blocking them
Only had IP blocks

15. Initial Recon Initial Compromise
February 2012
Initial Compromise
April 2012
Going through org charts, reading about projects

16. Gh0st RAT

17. Make your local FBI agent your new best friend
Lesson #4
Make your local FBI agent your new best friend
Insight into goals
Any others being attacked from same group
Assistance analyzing malware
Help with management
This attack is different. Not a patch, rebuild and you’re done
There are those who are hacked and know it, and those who are hacked and don’t
IRPS and Dali Lama

18. Have a secure communications plan in place
Lesson #5
Have a secure communications plan in place
Security staff had PGP keys but most sysadmins did not
Voice mail unreliable due to unified messaging
Attackers were definitely reading

19. Log everything, especially authentication,
Lesson #6
Log everything, especially authentication,
netflow and DNS
AD logs are ugly and chatty HUGE
Information spread out over several lines using different infor (IP, system name, etc) so context is difficult
Netflow to understand where they are going within the network. VPN netflow added
DNS is HUGE but can provide a lot of insight, especially when connected through VPN.
Tremendous amount of data

20. Dynamic DNS Beaconing $ nslookup host.somehackedsite.com
** server can't find host.somehackedsite.com: NXDOMAIN
host.somehackedsite.com has address

21. Attack timing
All attacks took place Sunday – Thursday between the hours of 6pm and 3am Pacific
This was somebody’s job
Insight on when we could make system changes when the attackers weren’t active

22. Attack Path

23. You don’t need to crack passwords when you can just pass a hash
Malware Observations
You don’t need to rely on a lot of malware when you’ve already got a long list of credentials
You don’t need to crack passwords when you can just pass a hash

24. NTLM Authentication
DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful.
Server sends the username, challenge and encrypted response to the DC.
Client encrypts the challenge with the user hash and sends it back to the server.
User provides username and password. Client computes hash, stores it in memory and throws away the plaintext password.
Server sends a challenge to the client.
Client sends username to server.
LSASS Local Security Authority Subsystem Service

25. Interactive Authentication
Client computes LM and NTLM hash and stores them in memory. Plaintext password is reversibly encrypted and stored in memory.
Password hash is salted with username and stored in registry.
LSASS Local Security Authority Subsystem Service

26. NTLM Authentication Client sends username to server.
LSASS Local Security Authority Subsystem Service

27. NTLM Authentication Server sends a challenge to the client.
LSASS Local Security Authority Subsystem Service

28. NTLM Authentication
Client encrypts the challenge with the user hash and sends it back to the server.
LSASS Local Security Authority Subsystem Service

29. NTLM Authentication
Server sends the username, challenge and encrypted response to the DC.
LSASS Local Security Authority Subsystem Service

30. NTLM Authentication
DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful.
LSASS Local Security Authority Subsystem Service

31. Administrator Hash
So, let’s say the domain administrator RDP’s to the client…
Domain Admin NTLM hash now stored in client memory.

32. Pass the Hash Attacker compromises client… Steals hashes from memory…
Accesses both server and domain controller

33. GAME OVER Pass the Hash Attacker compromises client…
Steals hashes from memory…
GAME OVER
Accesses both server and domain controller

34. Mitigations Change passwords multiple times per day
Fast track two factor authentication
Compartmentalized passwords
Separate user and admin credentials
Minimize lateral trust
Scan entire domain for scheduled tasks
Rebuild Domain Controlers
Authentications that used a hash but didn’t have a corresponding interactive login

35. Reconsider traditional password best practices
Lesson #7
Reconsider traditional password best practices
How often do you change your password?
A lot of best practice is based on outdated information
Keystroke loggers and phishing have invalidated most of that thinking
How long do you want the attackers to have access to your systems before kick them a=out and force them to reacquire creds?

36. Good passwords?
*tecno9654postgres A Matt Hale Tribute CD would be cool.. Access-Control-Allow-Origin Abundance4me2day Bulletformyvalentine123 Elementarymydearwatson Putin is nothing but commie scum. Video killed the radio star? antcolonyoptimization

37. Emergency Action September 2012
Tried to capture of upper org chart of one of the targeted departments
Webmail to check cred, POP to download
Swatting flies
So many compromised credentials
Reset the playing field

38. Effectively and securely communicating a password change is hard
Lesson #8
Effectively and securely communicating a password change is hard
Met with campus sysadmins to spread the message internally
helpdesk
Campus announcements
Prominent notices on official campus web pages
Just before quarter started
Fac, staff, priv role accounts
Avoid Sept 11.
35000 accounts,
Many disabled outright as not been used
5 day rolling disable
Huge phishing in the following weeks
Try a few, and back off

39. We are not alone Not just a windows problem
Some backlash on whether AD could be trusted
As we starting protecting more and more AD creds, attackers tried local accounts in an attempt to hide their activity

40. Reengagement
July 2013

41. ACT

42. Parting Thoughts Detection can be subtle and an art
Have a good AD Team
Logging visibility is essential
Regular password changes are a MUST
Be prepared to re-image any system
Firewalls to prevent lateral movement
Separation of user and admin credentials
Require two-factor for OU Admins
FBI has now confirmed other activity from this particular group

43. A New Hope

44. A New Hope Strengthened LSASS to prevent credential dumps
Many processes no longer store credentials in memory
Better ways to restrict local account use over the network
RDP use without putting the credentials on the remote computer
Addition of a new Protected Users group, whose members' credentials cannot be used in remote PtH attacks

45. Further Reading Know Your Digital Enemy – Anatomy of a Gh0st RAT
Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft Techniques
APT1: Exposing One of China's Cyber Espionage Units

46. “If ignorant both of your enemy and yourself, you are certain to be in peril.”
― Sun Tzu, The Art of War