1. Learning to Live with an Advanced Persistent Threat
EDUCAUSE 2013
October 17th, 2013
John Denune
IT Security Director

2. ACT Infrastructure services
Database Administration
Data Center
Active Directory
Security
Telecom
Decentralized
100 OU’s
800 OU Admins
Networking
ID Management
UNIX and Windows Support

3. It’s not Opportunistic
What is an APT?
It’s not Opportunistic
Low level criminal activity
Spam, phishing, WAREZ
Off the shelf attacks, higher likelihood of AV detection

4. APT Varied Attacks Espionage Technical Targeted Patient Skilled
Corporate
State-Sponsored
You have something they want and they will spend a lot of time trying to get it
Months or years
Off the shelf, but also custom malware including zero-day
Extremely low detection rates
Technical, phishing, phone calls
Dropping infected USB drives in parking lot or keystroke loggers on lab keyboards
Often tied so some sort of espionage, either corporate to get insider information or state sponsored to get military info
Hacktivism to expose information or long term DDOS to make a point
Good old fashioned theft
Skilled
Theft
Hacktivism
Physical threats
Social Engineering

5. APT Lifecycle External Recon Initial Compromise Establish Foothold
Escalate Privileges
Internal Recon
Expand
Complete Mission

6. Initial Detection June 2012 Got lucky AV alert on ACT server.
OU Admins compromised through unrelated staff account on VPN
Only 1 of 4 pieces of malware detected
Changed password, rebuild servers
Happened again the following night with another unrelated VPN account
Also found several other computers in unrelated departments, also OU admins compromised
Third time password changes and re-used

7. Pay attention to anti-virus alerts
Lesson #1
Pay attention to anti-virus alerts
Too many sysadmins view a detection as AV doing it’s job IF they even monitor at all
Modern malware loves company and almost always brings friends

8. Don’t (completely) rely on your anti-virus product
Lesson #2
Don’t (completely) rely on your anti-virus product
Low detection rates, especially for custom malware

9. Where possible, track IP’s instead of blocking them
Lesson #3
Where possible, track IP’s instead of blocking them
Only had IP blocks

10. Initial Recon Initial Compromise
February 2012
Initial Compromise
April 2012
Going through org charts, reading about projects

11. Gh0st RAT

12. Make your local FBI agent your new best friend
Lesson #4
Make your local FBI agent your new best friend
Insight into goals
Any others being attacked from same group
Assistance analyzing malware
Help with management
This attack is different. Not a patch, rebuild and you’re done
There are those who are hacked and know it, and those who are hacked and don’t
IRPS and Dali Lama

13. Have a secure communications plan in place
Lesson #5
Have a secure communications plan in place
Security staff had PGP keys but most sysadmins did not
Voice mail unreliable due to unified messaging
Attackers were definitely reading

14. Log everything, especially authentication,
Lesson #6
Log everything, especially authentication,
netflow and DNS
AD logs are ugly and chatty HUGE
Information spread out over several lines using different infor (IP, system name, etc) so context is difficult
Netflow to understand where they are going within the network. VPN netflow added
DNS is HUGE but can provide a lot of insight, especially when connected through VPN.
Tremendous amount of data

15. Dynamic DNS Beaconing $ nslookup host.somehackedsite.com
** server can't find host.somehackedsite.com: NXDOMAIN
host.somehackedsite.com has address

16. Attack timing
All attacks took place Sunday – Thursday between the hours of 6pm and 3am Pacific
This was somebody’s job
Insight on when we could make system changes when the attackers weren’t active

17. Attack Path

18. You don’t need to crack passwords when you can just pass a hash
Malware Observations
You don’t need to rely on a lot of malware when you’ve already got a long list of credentials
You don’t need to crack passwords when you can just pass a hash

19. NTLM Authentication
DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful.
Server sends the username, challenge and encrypted response to the DC.
Client encrypts the challenge with the user hash and sends it back to the server.
User provides username and password. Client computes hash, stores it in memory and throws away the plaintext password.
Server sends a challenge to the client.
Client sends username to server.
LSASS Local Security Authority Subsystem Service

20. Administrator Hash
So, let’s say the domain administrator RDP’s to the client…
Domain Admin NTLM hash now stored in client memory.

21. GAME OVER Pass the Hash Attacker compromises client…
Steals hashes from memory…
GAME OVER
Accesses both server and domain controller

22. Mitigations Change passwords multiple times per day
Fast track two factor authentication
Compartmentalized passwords
Separate user and admin credentials
Minimize lateral trust
Scan entire domain for scheduled tasks
Rebuild Domain Controlers
Authentications that used a hash but didn’t have a corresponding interactive login

23. Emergency Action September 2012
Tried to capture of upper org chart of one of the targeted departments
Webmail to check cred, POP to download
Swatting flies
So many compromised credentials
Reset the playing field

24. Reconsider traditional password best practices
Lesson #7
Reconsider traditional password best practices
How often do you change your password?
A lot of best practice is based on outdated information
Keystroke loggers and phishing have invalidated most of that thinking
How long do you want the attackers to have access to your systems before kick them a=out and force them to reacquire creds?

25. Effectively and securely communicating a password change is hard
Lesson #8
Effectively and securely communicating a password change is hard
Met with campus sysadmins to spread the message internally
helpdesk
Campus announcements
Prominent notices on official campus web pages
Just before quarter started
Fac, staff, priv role accounts
Avoid Sept 11.
35000 accounts,
Many disabled outright as not been used
5 day rolling disable
Huge phishing in the following weeks
Try a few, and back off

26. We are not alone Not just a windows problem
Some backlash on whether AD could be trusted
As we starting protecting more and more AD creds, attackers tried local accounts in an attempt to hide their activity

27. Reengagement
July 2013

28. ACT

29. Parting Thoughts Detection can be subtle and an art
Have a good AD Team
Logging visibility is essential
Regular password changes are a MUST
Be prepared to re-image any system
Firewalls to prevent lateral movement
Separation of user and admin credentials
Require two-factor for OU Admins
FBI has now confirmed other activity from this particular group

30. A New Hope Strengthened LSASS to prevent hash dumps
Many processes no longer store credentials in memory
Better ways to restrict local account use over the network
RDP use without putting the credentials on the remote computer
Addition of a new Protected Users group, whose members' credentials cannot be used in remote PtH attacks

31. Further Reading Know Your Digital Enemy – Anatomy of a Gh0st RAT
Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft Techniques
APT1: Exposing One of China's Cyber Espionage Units

32. “If ignorant both of your enemy and yourself, you are certain to be in peril.”
― Sun Tzu, The Art of War