Learning to Live with an Advanced Persistent Threat John Denune IT Security Director University of California, San Diego jdenune@ucsd.edu

ACT Infrastructure services Database Administration E-mail Data Center Active Directory Security Telecom Decentralized 100 OU’s 800 OU Admins Networking ID Management UNIX and Windows Support

ACT Security Policy and Compliance SSL Certs 9 Staff Firewall Anti-virus and FDE Forensics VPN Decentralized 100 OU’s 800 OU Admins Patch Management Incident Response Vulnerability Assessment Intrusion Detection

It’s not Opportunistic What is an APT? It’s not Opportunistic Low level criminal activity Spam, phishing, WAREZ Off the shelf attacks, higher likelihood of AV detection

APT Varied Attacks Espionage Technical Targeted Patient Skilled Corporate State-Sponsored You have something they want and they will spend a lot of time trying to get it Months or years Off the shelf, but also custom malware including zero-day Extremely low detection rates Technical, phishing, phone calls Dropping infected USB drives in parking lot or keystroke loggers on lab keyboards Often tied so some sort of espionage, either corporate to get insider information or state sponsored to get military info Hacktivism to expose information or long term DDOS to make a point Good old fashioned theft Skilled Theft Hacktivism Physical threats Social Engineering

APT Lifecycle External Recon Initial Compromise Establish Foothold Escalate Privileges Internal Recon Expand Complete Mission

Initial Detection June 2012 Got lucky AV alert on ACT server. OU Admins compromised through unrelated staff account on VPN Only 1 of 4 pieces of malware detected Changed password, rebuild servers Happened again the following night with another unrelated VPN account Also found several other computers in unrelated departments, also OU admins compromised Third time password changes and re-used

Initial Detection June 2012 Got lucky AV alert on ACT server. OU Admins compromised through unrelated staff account on VPN Only 1 of 4 pieces of malware detected Changed password, rebuild servers Happened again the following night with another unrelated VPN account Also found several other computers in unrelated departments, also OU admins compromised Third time password changes and re-used

Pay attention to anti-virus alerts Lesson #1 Pay attention to anti-virus alerts Too many sysadmins view a detection as AV doing it’s job IF they even monitor at all Modern malware loves company and almost always brings friends

Don’t (completely) rely on your anti-virus product Lesson #2 Don’t (completely) rely on your anti-virus product Low detection rates, especially for custom malware

Where possible, track IP’s instead of blocking them Lesson #3 Where possible, track IP’s instead of blocking them Only had IP blocks

Initial Recon Initial Compromise February 2012 Initial Compromise April 2012 Going through org charts, reading about projects

Gh0st RAT

Make your local FBI agent your new best friend Lesson #4 Make your local FBI agent your new best friend Insight into goals Any others being attacked from same group Assistance analyzing malware Help with management This attack is different. Not a patch, rebuild and you’re done There are those who are hacked and know it, and those who are hacked and don’t IRPS and Dali Lama

Have a secure communications plan in place Lesson #5 Have a secure communications plan in place Security staff had PGP keys but most sysadmins did not Voice mail unreliable due to unified messaging Attackers were definitely reading e-mail

Log everything, especially authentication, Lesson #6 Log everything, especially authentication, netflow and DNS AD logs are ugly and chatty HUGE Information spread out over several lines using different infor (IP, system name, etc) so context is difficult Netflow to understand where they are going within the network. VPN netflow added DNS is HUGE but can provide a lot of insight, especially when connected through VPN. Tremendous amount of data

Dynamic DNS Beaconing $ nslookup host.somehackedsite.com ** server can't find host.somehackedsite.com: NXDOMAIN host.somehackedsite.com has address 10.2.3.4

Attack timing All attacks took place Sunday – Thursday between the hours of 6pm and 3am Pacific This was somebody’s job Insight on when we could make system changes when the attackers weren’t active

Attack Path

You don’t need to crack passwords when you can just pass a hash Malware Observations You don’t need to rely on a lot of malware when you’ve already got a long list of credentials You don’t need to crack passwords when you can just pass a hash

NTLM Authentication DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful. Server sends the username, challenge and encrypted response to the DC. Client encrypts the challenge with the user hash and sends it back to the server. User provides username and password. Client computes hash, stores it in memory and throws away the plaintext password. Server sends a challenge to the client. Client sends username to server. LSASS Local Security Authority Subsystem Service

Interactive Authentication Client computes LM and NTLM hash and stores them in memory. Plaintext password is reversibly encrypted and stored in memory. Password hash is salted with username and stored in registry. LSASS Local Security Authority Subsystem Service

NTLM Authentication Client sends username to server. LSASS Local Security Authority Subsystem Service

NTLM Authentication Server sends a challenge to the client. LSASS Local Security Authority Subsystem Service

NTLM Authentication Client encrypts the challenge with the user hash and sends it back to the server. LSASS Local Security Authority Subsystem Service

NTLM Authentication Server sends the username, challenge and encrypted response to the DC. LSASS Local Security Authority Subsystem Service

NTLM Authentication DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful. LSASS Local Security Authority Subsystem Service

Administrator Hash So, let’s say the domain administrator RDP’s to the client… Domain Admin NTLM hash now stored in client memory.

Pass the Hash Attacker compromises client… Steals hashes from memory… Accesses both server and domain controller

GAME OVER Pass the Hash Attacker compromises client… Steals hashes from memory… GAME OVER Accesses both server and domain controller

Mitigations Change passwords multiple times per day Fast track two factor authentication Compartmentalized passwords Separate user and admin credentials Minimize lateral trust Scan entire domain for scheduled tasks Rebuild Domain Controlers Authentications that used a hash but didn’t have a corresponding interactive login

Reconsider traditional password best practices Lesson #7 Reconsider traditional password best practices How often do you change your password? A lot of best practice is based on outdated information Keystroke loggers and phishing have invalidated most of that thinking How long do you want the attackers to have access to your systems before kick them a=out and force them to reacquire creds?

Good passwords? *tecno9654postgres A Matt Hale Tribute CD would be cool.. Access-Control-Allow-Origin Abundance4me2day Bulletformyvalentine123 Elementarymydearwatson Putin is nothing but commie scum. Video killed the radio star? antcolonyoptimization

Emergency Action September 2012 Tried to capture e-mail of upper org chart of one of the targeted departments Webmail to check cred, POP to download e-mail Swatting flies So many compromised credentials Reset the playing field

Effectively and securely communicating a password change is hard Lesson #8 Effectively and securely communicating a password change is hard Met with campus sysadmins to spread the message internally helpdesk Campus announcements Prominent notices on official campus web pages Just before quarter started Fac, staff, priv role accounts Avoid Sept 11. 35000 accounts, Many disabled outright as not been used 5 day rolling disable Huge phishing in the following weeks Try a few, and back off

We are not alone Not just a windows problem Some backlash on whether AD could be trusted As we starting protecting more and more AD creds, attackers tried local accounts in an attempt to hide their activity

Reengagement July 2013

ACT

Parting Thoughts Detection can be subtle and an art Have a good AD Team Logging visibility is essential Regular password changes are a MUST Be prepared to re-image any system Firewalls to prevent lateral movement Separation of user and admin credentials Require two-factor for OU Admins FBI has now confirmed other activity from this particular group

A New Hope

A New Hope Strengthened LSASS to prevent credential dumps Many processes no longer store credentials in memory Better ways to restrict local account use over the network RDP use without putting the credentials on the remote computer Addition of a new Protected Users group, whose members' credentials cannot be used in remote PtH attacks

Further Reading Know Your Digital Enemy – Anatomy of a Gh0st RAT http://www.mcafee.com/us/resources/white-papers/foundstone/wp-know-your-digital-enemy.pdf Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft Techniques http://www.microsoft.com/en-us/download/details.aspx?id=36036 APT1: Exposing One of China's Cyber Espionage Units http://intelreport.mandiant.com/Mandiant_APT1_Report.pdf

“If ignorant both of your enemy and yourself, you are certain to be in peril.” ― Sun Tzu, The Art of War