1. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Exposing Stormworm Brandon Enright, bmenrigh@ucsd.edu Available at: http://noh.ucsd.edu/~bmenrigh/exposing_storm.ppt Data available at: http://noh.ucsd.edu/~bmenrigh/storm_data.tar.bz2

2. Part 0: Research Credit Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

3. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Storm Research Credit: Although most of this work is my own research, this presentation build heavily off of the research, support, and help of others. All errors, inaccuracies, or omissions added by Brandon Enright. Thanks to: Joe Stewart Josh Gabe Lawrence Andre Ludwig UCSD Sysnet Stefan Savage Chris Kanich

4. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Part 1: Intro to Stormworm

5. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Storm Worm is a malicious program discovered in late January, 2007. The name ‘Storm Worm’ is derived from one of the subject lines used in the initial email attack: “230 dead as storm batters Europe.” Storm’s primary method of spreading remains email social engineering. Other common AV names: Peacomm Nuware Zhelatin CME711 Storm Vitals: Tibs Peed Dorf See: [1] [2] Note: I generally refer to “Stormworm” or “Storm Worm” as “Storm”. The phrase “worm” is no more accurate than “bot”, “Trojan”, “backdoor” or other labels. Storm is truly hybrid malware.

6. Storm Vitals Continued: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Storm is one of the first pieces of malware to use a P2P network for command and control (C&C). The use of P2P has made Storm one of the most resilient, successful bots ever. The lack of centralized C&C has made Storm able to easily resist attempts to shut down the network and has evolved continuously to stay ahead of the AV industry and researchers. Features: Uses P2P (Overnet/Kademlia) Uses fast-flux DNS for hosting on named sites Binary has gone through many revisions Features of P2P network have evolved with time Hides on machine with rootkit technology See: [3]

7. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Storm Capabilities: As Storm has evolved, it has gained a number of capabilities to aid it in malicious activity. Capabilities: Spam (implemented via templates) Spread (using spam) ICMP Echo flood TCP SYN flood Proxy connections Download and executed file Update

8. Storm Activities: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Aside from spreading via ‘eCards’ and other social engineering emails, the Storm network has been used for many malicious money-making activities. Malicious activities: Pump and Dump spam using: PDF file with embedded images Excel spreadsheet file with embedded images MP3 text-to-speech recordings Plain-text emails Phishing emails DDoS against other groups and organizations Automatic DDoS of researchers probing Storm web proxies See: [4]

9. Storm “Research” and Media Coverage: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations The media has made Storm one of the most written and talked about malicious programs ever. The numbers and information from researches and reported on by the media has evolved in a whirlwind of back-of-the-envelope calculations and sensationalist journalism. Common estimates about Storm: The network size is between 1 and 50 million bots The network is the worlds most powerful super computer The network sends tens of billions of emails a day The network has been growing in size The decentralized nature of the network makes it impossible to track Fortunately, most of these estimates are inaccurate or completely wrong. See: [5] [6] [7]

10. Part 2: Storm Malware Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

11. Storm Malware: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Several in-depth papers have been written about the Storm binary and some of its capabilities. Storm has gone through several generations that have changed how it behaves, installs, and hides on the system. Here is a rough overview of the evolution: Unpacked/Unencrypted driver: wincom32.sys Packed driver: wincom32.sys Packed driver: windev- -.sys Packed driver: vdo- -.sys Packed driver: spooldr.sys, Packed exe: spooldr.exe TCP functionality added to P2P network Anti-virtualization and anti-debugging code added, patching of tcpip.sys See: [8] [9] [10] [11]

15. Part 3: Overnet/DHT Overview Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

16. Overnet/Kademlia: Overnet is a Distributed Hash Table (DHT) network based on the Kademlia algorithm. This is the same protocol that older versions of eDonkey use. DHT algorithms use a linear key space (generally 128 bits) to uniquely identify nodes and published network content. All DHT algorithms have some way of computing logical distance between two nodes or between a node and published content. Kademlia computes distance between nodes by XORing their published hashes. When a peer wants to find content in the network, it computes (or is given) the hash of that content and then searches adjacent peers. Those peers respond with their adjacent peers that are closer. This is repeated until the searching peer gets close enough to the content that a node there will be able to provide a search result. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations See: [12] [13] [14]

17. Overnet Continued: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Because of the distributed nature of Overnet, there is no central peer list. The list of active peers and published content is distributed in many small pieces in the memory of all the nodes participating in the network. It is the lack of a centralized list of peers and the dynamic nature of what is stored in the DHT that makes Storm so resilient to attack. It is also the primary reason why casual researchers and security enthusiasts often chalk the Storm network up as impossible to shut down or to even track or estimate the size of. Fortunately, Overnet is a simple protocol and once implemented in a semi- generic program can easily and quickly be used to crawl the whole Storm network.

18. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Bootstrapping Overnet: Since there isn’t a centralized peer list, a new peer can not join the network without being given a short list of peers to bootstrap from. Storm is always distributed with an embedded peer bootstrap file. When run, Storm extracts this file to disk.

19. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Bootstrapping Overnet Continued: [peers] 00cfed21483926536128f06ceb479d8a=ACCCD8EE150B01 019fc63b3137a6806e5fcd70b1a5139a=9A25428D1EBF01 2d32b3f2a0476d167929bf7e6d0c1aa2=543F05B2177201 2d736cb7f396677aa569f61a9994bca2=D51AD5961A7500 2d8103bfe0756a2a5798ea2a1d7bebe8=A135A605177F01 2d8aae15f6e40821c9031084a28b4cae=52EE4FD51E0201 2daf33b3329cf8e09df3ea7f95534dd8=53FE44ED297B00 … The Storm bootstrap file format is shown below: Each line is a single hex-encoded peer in this format: =

20. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Bootstrapping Overnet Continued: Although the Overnet ID hash space is linear, it is easy to represent as a 2D directed graph. The following slide is a simple bootstrap graph where peers are colored based on the round in which they were discovered.

21. Bootstrapping Peer Round 2 Round 1 Round 4 Round 3

22. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Maintaining Overnet Connectivity: The Overnet network Storm uses is extremely dynamic. Peers come and go and can change OIDs frequently. In order to stay “well connected” peers must periodically search for themselves to find nearby peers: Storm Node See: [15]

23. Overnet Message Passing: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Overnet has three basic message types to facilitate proper function of the network: Connect: A peer uses connect messages to report their OID to other peers and to receive a list of peers somewhat close to the peer. Search: A peer uses search messages to find resources and other nodes based on OID. Publicize: A peer uses publicize messages to report ownership of network resources (OIDs) so that other peers can find the resource later.

24. Part 4: Storm Overnet Network Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

25. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Analyzing Storm Overnet Traffic: Although Overnet is a simple protocol and Storm uses a rather small subset of it, analyzing thousands of packets can still be difficult. Luckily, the eDonkey protocol dissector included with Wireshark does an excellent job of identifying Storm’s Overnet traffic. See: [16]

26. Decoding Messages:

27. Publicize Message:

28. Connect Message:

29. Connect Reply Message:

30. Search Message:

31. Search Next Message:

32. Other Messages: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations While analyzing Storm Overnet traffic, you will notice other Overnet messages like IPQuery being used. Some versions of Storm make more use out of these messages than others. They are not required for the basic operations of Storm and don’t need to be implemented in a crawler. Documentation can for these other messages can be found via searching or the Resources/References section at the end of this presentation.

33. Changes to the Network: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations The Storm network has undergone several changes to make it more resilient to attack and to better suit the needs of the creators/operators. Rough list of changes: Addition of TCP component to load proxying peers table Change of OID generation code Enhanced usage of IPQuery to detect NAT Addition of TCP commands via custom TCP overlay network Shift from heavy usage of Search to Publicize Addition of encryption to Overnet protocol Credit: Joe Stewart, SecureWorks

34. Part 5: Building a Crawler Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

35. Crawler Basics: Storm’s use of Overnet is rather simple. There are several shortcuts to writing a crawler that won’t significantly affect its performance. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Shortcuts: Crawler does not have to respond to probes sent to it Publicize does not need to be implemented IPQuery and other miscellaneous messages do not need to be implemented ICMP errors do not need to be detected Response OIDs can be completely ignored No TCP functionality is needed All communication can (and probably should) be done with a single UDP socket Crawler does not need to wait for or timeout responses

36. Crawler Outline: The simplest crawler design is probably to follow the basic algorithm used by older versions of Storm when bootstrapping. Basic outline: 1. Read in peers list (bootstrap file) 2. Send each peer connect message, add responders to list 3. Loop, sending random subset connect or search or publicize message 4. Go into select() loop, retrieving and parsing responses 5. Periodically clean out dead peers 6. Goto step 3 Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

37. Pitfalls to avoid: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Storm has gone through many buggy revisions to get to where it is now. Add to that researchers poisoning the network or sending malformed messages and there are a number of things to look out for. Pitfalls: Don’t trust than an IP and port are valid just because you received a response from it (think port 0, multicast, etc) Don’t send to obviously invalid IPs (private IPs, unallocated networks, etc) Don’t waste memory (think C, Perl, Python, etc, not Java) Don’t let single host tie up your select() loop by sending packets too quickly (no really easy fix, detect, bail out of loop, block) Don’t run behind firewall, NAT, or any device that does connection tracking

38. Optimizations: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations The unencrypted Storm network is a mess. There is so much data, most of it stale or fake, that if the crawler wastes time processing/sending too much data it won’t do its job well. Potential optimizations: Use a RAW ICMP socket to detect errors, delete those peers Blacklist abusers/poisoners/extremely buggy versions of Storm Reduce delay between send() and select() or use threads to do both Increase your OS’s UDP buffer/queue as much as possible (sysctl) Aggressively detect and remove dead peers Implement a minimum removed time to cut down on re-trying dead/fake peers If crawling the encrypted network(s), allow for multiple keys

39. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Part 6: Sample Perl Code

40. Sockets: my $udpproto = getprotobyname ('udp'); my $icmpproto = getprotobyname ('icmp'); my $siaddr = inet_aton ($bind_addr); my $spaddr = sockaddr_in ($bind_port, $siaddr); socket (SOCKET, PF_INET, SOCK_DGRAM, $udpproto) or die "Failed to create socket: $!\n"; setsockopt (SOCKET, SOL_SOCKET, SO_RCVBUF, $socket_buffer) or die "Failed to set socket buffer: $!\n"; socket (ICMPSOCKET, PF_INET, SOCK_RAW, $icmpproto) or die "Failed to create ICMP socket: $!\n"; bind (SOCKET, $spaddr) or die "Failed to bind: $!\n"; Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

41. Bootstrap select(): # Recieve any bootstrap connect replies we may have gotten my ($rin, $rout); $rin = ''; vec ($rin, fileno (SOCKET), 1) = 1; while ((select ($rout = $rin, undef, undef, $bootstrap_time)) && (scalar (keys %peers) < 1000)) { my $MESSAGE; my $rhpaddr = recv (SOCKET, $MESSAGE, PACKET_SIZE, 0) or die "Recieve (bootstrap) failed on socket : $!\n"; (my $rport, my $rhaddr) = sockaddr_in($rhpaddr); my $raddr = inet_ntoa ($rhaddr); Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

42. is_search_next(): Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations sub is_search_next { my $MESSAGE = shift; unless (length $MESSAGE >= 19) { return 0; } my ($kproto, $type, $hash, $number) = unpack ('C C a[16] C', $MESSAGE); return (($kproto == 0xE3) && ($type == 0x0F) && ($number > 0)); }

43. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations parse_connect_reply(): sub parse_connect_reply { my $MESSAGE = shift; my $packed_ip = shift; my ($kproto, $type, $number) = unpack ('C C v', $MESSAGE); # Skip the header and go right to the peers $MESSAGE = substr $MESSAGE, 4; for (my $i = 0; $i < $number; $i++) { last if (length $MESSAGE < 23); my ($phash, $phaddr, $pport) = unpack ('a[16] a[4] v', $MESSAGE); my $paddr = inet_ntoa ($phaddr); add_peer ($paddr, $pport, $phash, 'ConnectReported', $packed_ip); # Go to the next peer $MESSAGE = substr $MESSAGE, 23; }

44. Part 7: Stormdrain: Crawler States

45. Stormdrain: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Stormdrain is the name of a Perl crawler I wrote in June, 2007. Stormdrain is a glorified finite-state-machine simulator. State information is maintained for each peer and as Stormdrain probes the network peers transition to different states depending on how they respond. Stormdrain saw the peak of the Storm network in early July at about 200k live/active/reachable hosts online at a given time (about 1.5M in a day). Unfortunately, recording of all peer data was not implemented until mid-July so this peak was not recorded. Stormdrain info: 2315 lines of Perl 37 GiB of data recorded Over 10 billion packets sent

46. Stormdrain Peer States: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Although Stormdrain’s design principles are simple, the way peers transition through states in the state machine can be challenging to understand. The following slides describe the basics of each of the basic states: Basic Peer States: Live Active Dead Removed Unknown

47. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Live State: Any peer that has recently sent a Overnet packet to the crawler is counted as live. The Overnet packet could be a response to a probe initially sent by Stormdrain or a probe initiated by the peer itself. NAT’d and firewalled peers can be live. If they don’t respond to probes though they will be quickly transitioned into a dead peer. Any peer can become a live peer by sending a probe to the crawler. This includes removed peers. Spoofed source packets will cause an errant live peer until it is transitioned to dead for not responding to active probing.

48. Active State: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations An active peer is a live, actively responding peer. All active peers are also live peers. The active state is an attempt to identify peers that are online and reachable at a given moment. NAT’d and firewalled peers typically can not become active. Active peers must have been probed with connects at least 3 times, searches at least 3 times, must have responded at least 3 times, and must have responded at least once in the last 5 probes. Most peers that are reachable and useful to the group controlling Storm are active.

49. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Dead and Unknown States: A dead peer is a peer is a peer that at some point was live (or active) but hasn’t responded to a large number of probes and so is probably no longer connected/participating in the network. Dead peers stick around for a while trying to be ‘revived’ before being removed. Unknown peers are peers that aren’t in any other state (including removed). Unknown peers are peers that were reported in a connect reply or search next message by some live peer. Unknown peers could become live (or active) peers. If an unknown peer doesn’t respond to any probes to become live they are transitioned directly to removed (skipping the dead state). The vast majority of unknown peers are fake/stale peers and are not participants in the network.

50. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Removed State: The removed state is a unique state in Stormdrain. The removed state is not an instantaneous measurement of what a peer is doing. It is a 2 hour black-list. The removed state exists to reduce the amount of work Stormdrain has to spend on unknown peers. Without the removed state, stale peers would be reported by a live peer, stay unknown for some time, then be deleted. They would go right back into the unknown state and start over if they were reported by a live peer again though. Removed peers are not probed by Stormdrain at all. A removed peer can not be added back to the state table as an unknown peer until it is cleared after 2 hours. A removed peer can move directly to the live state by sending a probe to the crawler. The reason the removed state is so much larger than the others is that it is 2 hours of data. Many of the removed peers are stale peers that were live or active recently but have gone offline or have become otherwise unreachable.

51. Part 8: Stormdrain: Crawler Data

52. Stormdrain Data: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Starting in early September Stormdrain was modified to report live data in addition to the hourly snapshots. This was done so that MRTG could be used to graph the network in real-time. The live nature of these statistics make them much more sensitive to bugs and down-time. Stats have also been generated for the hourly data dumps and included in the large flat file. All times in the stats and graphs are UTC. The raw stats data files for each point in time are included in the data archive. They can be used to generate higher resolution graphs. Proper attribution is required. I can be contacted via email with questions regarding the stats or requests for an explanation of the various anomalies.

53. Total Peers and Active Peers: The blue peers count is all peers being probed at a time. This includes live, active, dead, and unknown states. The peers line is not the size of the network. The active line is much closer to the instantaneous size of the network. It can be seen in the month and year chart that Microsoft made a measurable dent in the network with the MRT Storm (Nuwar) release. See: [17]

54. Live and Active Peers: The blue live peer count line is the most accurate estimate of the size of the network for a point in time that Stormdrain can give.

55. Just Encrypted Live and Active Peers: This graph is the result of crawling the encrypted portion of the Storm network. Currently only one key is known by Stormdrain. There is evidence of more than one key in use. These encrypted peers are included in the totals in the other graphs. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

56. Removed Peers and Total Peers: The blue line is the 2 hour window of removed peers. The green area is the number of peers being worked on by Stormdrain, not the active or live count. The absolute scale is not meaningful, just the relative change over time. These graphs depict one view some of the as-of- yet unexplained trends in the Storm network.

57. Part 9: Resources and References Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations

58. Resources and References: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Basics: [1] http://en.wikipedia.org/wiki/Storm_wormhttp://en.wikipedia.org/wiki/Storm_worm [2] http://www.websense.com/securitylabs/blog/blog.php?BlogID=147http://www.websense.com/securitylabs/blog/blog.php?BlogID=147 [3] http://en.wikipedia.org/wiki/Storm_botnethttp://en.wikipedia.org/wiki/Storm_botnet [4] http://www.eweek.com/article2/0,1759,2169497,00.asphttp://www.eweek.com/article2/0,1759,2169497,00.asp Media: [5] http://www.informationweek.com/news/201804528http://www.informationweek.com/news/201804528 [6] http://www.neoseeker.com/news/story/7103/http://www.neoseeker.com/news/story/7103/ [7] http://blog.washingtonpost.com/securityfix/2007/10/the_storm_worm_m aelstrom_or_te.html http://blog.washingtonpost.com/securityfix/2007/10/the_storm_worm_m aelstrom_or_te.html

59. Resources and References Continued: Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Malware: [8] http://www.secureworks.com/research/threats/view.html?threat=storm-wormhttp://www.secureworks.com/research/threats/view.html?threat=storm-worm [9] http://www.websense.com/securitylabs/blog/blog.php?BlogID=141http://www.websense.com/securitylabs/blog/blog.php?BlogID=141 [10] http://www.cyber-ta.org/pubs/StormWorm/http://www.cyber-ta.org/pubs/StormWorm/ [11] http://www.reconstructer.org/papers/Peacomm.C%20- %20Cracking%20the%20nutshell.ziphttp://www.reconstructer.org/papers/Peacomm.C%20- %20Cracking%20the%20nutshell.zip Kademlia: [12] http://www.barsoom.org/~agthorr/papers/infocom-2006-kad.pdfhttp://www.barsoom.org/~agthorr/papers/infocom-2006-kad.pdf [13] http://en.wikipedia.org/wiki/Kademliahttp://en.wikipedia.org/wiki/Kademlia [14] http://xlattice.sourceforge.net/components/protocol/kademlia/specs.htmlhttp://xlattice.sourceforge.net/components/protocol/kademlia/specs.html

60. Exposing Stormworm Brandon Enright, UCSD ACT/Network Operations Resources and References Continued: Storm Overnet Network: [15] http://www.usenix.org/events/hotbots07/tech/full_papers/grizzard/grizzard.pdfhttp://www.usenix.org/events/hotbots07/tech/full_papers/grizzard/grizzard.pdf [16] https://opensvn.csie.org/mlnet/trunk/docs/overnet.txthttps://opensvn.csie.org/mlnet/trunk/docs/overnet.txt [17] http://blogs.technet.com/antimalware/archive/2007/09/20/storm-drain.aspxhttp://blogs.technet.com/antimalware/archive/2007/09/20/storm-drain.aspx Me: Brandon Enright, bmenrigh@ucsd.edubmenrigh@ucsd.edu