1. Michigan HTCIA March 29 th, 2006 Usenet Abuse Primer

2. Introductions Mark Lachniet (mlachniet@analysts.com) Technical Director, Security Services Group Responsible for technical oversight of all security offerings including live and static forensic analysis Certified Information Systems Auditor (CISA) Certified Information Systems Security Professional (CISSP) Frequent presenter at local educational conferences (MACUL, MAEDS, MIEM) Technical certifications from Novell, Microsoft, Linux Professional Institute, etc. Formerly the I.S. Director at Holt Public Schools Also an instructor for Walsh College’s MSIA program NOT a Law Enforcement Officer - YMMV

3. Introductions D/Sgt. Michael Harrington,CFCE, EnCE Forensic Analyst Michigan State Police Past President MIHTCIA HTCIA International Treasurer 2004 HTCIA International Secretary 2005 Gentoo and Sam Adams Enthusiast Email:linuxchimp@gmail.com

4. Warning! Due to the subject matter, there may be offensive content in this presentation No graphic images will intentionally be displayed, but be warned, there is definitely disturbing content out there You have the capability to do the hands-on exercises, but as time is very limited this will be on a “do it if you can keep up” basis The dev server (dev.lachniet.com) will be available for testing after this presentation. If you would like to use it, please send me an e- mail. I cannot guarantee how long I will keep the server up, but its free

5. Agenda Usenet overview NZB files Finding Usenet content Usenet anonymity Legal liability Yenc encoding Investigation Demonstration(s)

6. Why do we care? Usenet is very commonly being used for piracy and pornography, and very little information about it is generally known in the Law Enforcement community There may be legal liability for employers Usenet is widely distributed and heavily used Usenet can be extremely difficult to investigate – anonymous services that market based on not logging are commonplace Usenet content, particularly the Yenc format of binary encoding, is not well supported by conventional forensic tools

7. The Paper and Future Training This presentation is based on a whitepaper, developed by Mark and Mike, which will soon be released We are considering putting together a longer training session based on this content (2hrs isn’t enough) We would be interested in both feedback on the paper prior to release, and in a venue for doing a longer training session Contact me after the session or at mark@lachniet.com if you are interested

8. The History of Usenet Usenet is a distributed Internet discussion system that evolved from a general purpose UUCP network of the same name. Users, sometimes called Usenetters, read and post email-like messages (called "articles") to a number of distributed newsgroups, categories that resemble bulletin board systems in most respects. The medium is sustained among a large number of servers, which store and forward messages to one another. Usenet is of significant cultural importance in the networked world, having given rise to, or popularized, many widely recognized concepts and terms such as "FAQ" and "spam".

9. Usenet Today Usenet is like a bulletin board – users can post messages which can then be read by other users The messages themselves generally resemble a standard e-mail Newsgroups such as alt.binaries.games are used to organize the messages by topic, and are the “buckets” in which the messages are stored Usenet servers talk amongst themselves to share messages and synchronize Thus, if you post a message to alt.binaries.games on a commercial server such as Giganews, it will soon be replicated to other servers around the world Refer to RFC 0977 and RFC 1036 for more technical details

10. Usenet Groups There are a variety of newsgroups for just about anything you could ever want or need. As of 3/26/06 there are 105,228 groups carried by Giganews.com Groups that are distributed worldwide are split into seven classifications: comp, misc, news, rec, soc, sci, and talk Conversely, the alt tree of Usenet is anarchy incarnate, and has less oversight For example: –alt.adjective.noun.verb.verb.verb –alt.american.olympians.choke.choke.choke –alt.christnet.bible-thumpers.convert.convert.convert –alt.binaries.games –alt.binaries.pictures.erotica.early-teens*

11. Example Usenet Message Headers Path: border1.nntp.dca.giganews.com!nntp.giganews.com!feed2.newsreader.com!newsreader.com!npeer.de.kpn- eurorings.net!news.tele.dk!news.tele.dk!small.news.tele.dk!news.astra web.com!newsrouter-eu.astraweb.com!eweka!hq- usenetpeers.eweka.nl!81.171.88.219.MISMATCH!newsreader30.ewe ka.nl!not-for-mail From: "Apollo" Subject: were can i download the series? Newsgroups: alt.binaries.battlestar-galactica Date: Sun, 1 Jan 2006 16:47:29 +0100 Lines: 7 Message-ID: Organization: Eweka Internet Services NNTP-Posting-Host: Eweka Internet Services X-Trace: Posted by Eweka Internet Services, http://www.eweka.nl X-Complaints-To: abuse@n-o-s-p-a-m.eweka.nl Xref: number1.nntp.dca.giganews.com alt.binaries.battlestar- galactica:824361

12. Example Usenet Message Body This message is in yEnc format. If your newsreader cannot display this message, please visit http://www.ydecode.com/ and download yEnc decoder. =ybegin line=128 size=24064 name=hello.doc úù;=JËÛD ****************h*- *()3*0***********+***T********:**V***+***()))****S***))))))) )))))))))))))))))))))))))))))))))))))))) ))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))))) ))))))))))))))))))))))))))))))))))))))) )Ïë*©Š3=n**<é******:*****0**m2**8*Œ”Œ”±±****************** 3=n@*X:**®**®**m*******************************))9**** *****))9**

13. Similarities to e-mail Messages There are several similarities to regular e-mail messages We have a From: heading and a Subject: as well as a Date: and other similar looking headers However, some key differences exist. For example, rather than a To: field, we have a Newsgroups: field You can’t really trust the From: field to be an actual person – it is easy to spoof. Only a complete idiot would post illegal content with their real e-mail address. One thing you can *probably* trust is the Message- ID field. This field shows you what news server originally received the message. Although this could also be spoofed by setting up your own Usenet server, the common criminal probably isn’t that smart.

14. Usenet Message Propagation In layman’s terms, my interpretation, Usenet propagation works something like the following: A client with access to a Usenet server posts a message. This communication is between the client and the server directly, and usually takes place over TCP port 119. Let us suppose that the client had posted their message to alt.binaries.battlestar- galactica. The server receives the message and assigns it a unique Message-ID field. The server communicates with its Usenet peers (other Usenet servers). For each message that it has received locally, it checks with its peers to see if any of them subscribe to the alt.binaries.battlestar- galactica groups.

15. Usenet Message Propagation If the peer servers do subscribe, they communicate between themselves to see if they already have a copy of the message. This is based on the Message-ID field. If they subscribe to the newsgroup but do not have the specific message, it is transmitted to the peer. If the server already has the message, it ignores that message. A second client (the consumer) connects to their Usenet server (most likely a server in a completely different part of the world). They pull up a list of messages for the newsgroups for which they are subscribed, and the message in question is listed. If the consumer client wishes to download the message, they do so, and the content is transferred to their computer.

16. Usenet Binaries Just as in e-mail, it is possible to encode a binary in a text format and transmit it Some methods to encode binary content include BASE64, BinHex, UUencode, Quoted Printable and the Usenet-specific yEnc format. More information on this will be included in Mike Harrington’s Forensic Analysis (later on) In the case of large files such as CD Images, video content, etc. the limit 10,000 lines of text is often exceeded In some cases, a binary file may fit within the limit of a single Usenet message. For example, small images and smaller files may be entirely self- contained.

17. Demonstration – A Small Binary A small repository of files to do the exercises can be found at http://dev.lachniet.com First we’ll install NewsReactor and subscribe to our test groups Download the install from http://dev.lachniet.com, or http://www.daansystems.com/newsreactor/, and run it Click next a bunch of times We’ll then configure our dev Usenet server and subscribe to the groups

18. Configuring the Usenet Server Go to File-> Options and add the server The server is dev.lachniet.com The server doesn’t require authentication – it’s wide open for now

19. Subscribe to Groups Ignore my accidental extra groups Click on the Groups tab, and then click on the “newsserver” button to download a list of groups on the server. Select alt.binaries.mihtcia.n aughty Click OK

20. View the Contents of the Group Click on the alt.binaries.mihtcia naughty link

21. View Raw “inappropriate” Message Double-click on the last message to pull up the raw text This is what you might find if you analyzed a NNTP server, or a client that had downloaded the mssage

22. View the Decoded Binary Close the window and this time right-click on the message and select “grab and open” Hey, it was the 70’s

23. Multi-Part Postings There is a maximum number of lines, so large files will be broken up when posted Some clients (e.g. Outlook) won’t reassemble them, while smarter clients (e.g. NewsReactor) will This is *different* from multi-part archives, which we’ll talk about later This is due to the maximum number of lines that are allowed in a Usenet message

24. Multi-part Postings Example - OE You can see a single file broken into 9 chunks in OE – note 517kb per message

25. Multi-part Postings Example - NR NewsReactor aggregates and combines them for you

26. Multi-part Archives Splitting large binary postings (such as a CD ISO image) into smaller files is useful, but not convenient and fault tolerant For example, if you lose any single part of the posting, you will not have a valid binary and will have to download it again This can happen if a part is lost, corrupt, or never uploaded It is better to break a large files into multi-part archive files of a more manageable size such as 4mb. That way, if part of it is corrupt, you only have to download 4mb instead of 650mb The most popular way to split up large files is to use WinRAR

27. Multi-part Archive Example Look at the Babylon 5 Videos posting:

28. Multi-part Archives - WinRAR In the above example, if you successfully download all of the RAR files (1-9) you can then open the archive and uncompress it This is how most software and pornography is distributed currently Also note that NewsReactor gives you a high-level title heading for the entire series of files, and allows you to download them all with a single click Lets demonstrate this…

29. Demonstration – Multi-part Archives Select the high-level message, right click and hit “grab” and you can see it downloading the parts

30. Demonstration – Multi-part Archives Now click on “browse” to open the directory they were downloaded to Make sure WinRar is installed (get it from dev.lachniet.com if it isn’t) Double-click on the first file (b5cd.part01.rar) and you’ll see the contents of the combined RAR file

31. Demonstration – Multi-part Archives Click “Extract to” and select c:\ Navigate to c:\b5cd to view the files Double-click on “loader.exe” if you have an unhealthy interest in science fiction and mythology

32. Parity Archives Multi-part archive files are all well and good, but the Usenet (let alone the Internet!) is not a reliable medium for transmitting large amounts of data. In particular, with the Usenet, it is common for some parts of a large archive file to be lost (generating a “fill” request for the poster to resend the missing parts). To accommodate this problem, enterprising software engineers came up with a way to create parity files This parity system is very similar to a RAID-5 disk array in the hardware world. The long and short of it is that with PAR and PAR2 software, it is possible to recreate complete archives, even if some portions of an archive are missing. I used WinPAR (on dev.lachniet.com) to create parity archives for the Babylon 5 videos

33. Demonstration – PAR files Navigate to your hard drive where all of the RAR files are stored. Delete 1 of the 9 archive files and attempt to open the archive You’ll still be able to see what the contents of the file are, but when you try to extract it to c:\ you’ll get an error

34. Demonstration – PAR files Now, double-click on the PAR2 file “b5cd.vol01+02.PAR2”

35. Demonstration – PAR files You can now “reconstitute” the missing file from the PAR files by clicking on “repair” At this point you could then open the RAR files and uncompress the archive There are several implications for LEO’s: –Evidence may exist on the hard drive but not be findable until you combine multi-part postings and/or multi-part archives –Searches for yenc strings and archive files (not just RAR but also ZIP, PAR, PAR2, etc.) should be included in all searches –Even partial files (one rar out of many) will still have a directory table, so you may be able to get interesting search terms out of it

36. NZB Files Since each Usenet message is unique, and has a unique identifier such as Message-ID: it is possible to create a type of index file that makes it easy to download binary content NZB files are specifically designed for this purpose Many programs and search sites will automatically create NZB files for you, so you don’t have to find the content the hard way

37. NZB File Example alt.binaries.old.games Xns9734CAA2DE2A8orisitdunnocom@194.152.65.251 Xns9734CAAE875D0orisitdunnocom@194.152.65.251

38. NZB File Example In the previous example, we can see that the binary is “MISC OLDGAMES gone FREEWARE vol 3” and is 6,736,639 bytes We can tell that the original server to receive the posting was 194.152.65.251 We can see a series of unique message ID’s that the server assigned (e.g. Xns9734CAA2DE2A8orisitdunnocom) Using a modern newsreader, we could simply open that NZB file and start downloading

39. Finding Usenet Content Since there are so many newsgroups, and so many messages, it is sometimes difficult to find specific content Just the headers on an active newsgroup can be gigabytes in size Fortunately, there are a number of online search engines that can be used to narrow down what you are looking for

40. http://groups.google.com Intended for text searches Allows you to search a LOT of historic data, going back (at least) into the early 90’s This is also a great site for doing background checks on people who have been around for a while In 1992, nobody ever heard about archiving, and probably were a little more loose with their postings than they would be nowadays

41. http://www.yabse.com/index.php Will find binary content and make a NZB for youb You can use a URL such as http://www.yabse.com/index.php?q=freeware

42. http://www.guba.com Guba provides you not only with a search engine for binary content, but a means of directly downloading it You don’t need to use a news client at all – just get a subscription to Guba, and search for the content you want You can then simply download the complete binary from the web site You can also convert it to a different format, so you can watch it on your iPod or PSP

43. http://www.guba.com

44. Usenet Anonymity As previously mentioned, it can be tough to track down a Usenet poster even under the best of circumstances Names, servers and message IDs can all be spoofed In addition, a number of Usenet service providers make a business out of running systems that intentionally don’t keep any logs This could make it very tough to catch the person who already posted, but you might be able to catch them if they do it more than once Example Privacy policy (from easynews.com) –Easynews takes your privacy seriously. We have one of the most aggressive privacy policies in the industry. –Easynews does not monitor or log downloads. –No identifying information is placed in your Usenet posts. Your posts are virtually anonymous with all X- Headers removed.

45. Usenet  E-Mail Gateways There are also allegedly services that will forward between Usenet and E- Mail The software for this is available, but I don’t know of a commonly used service This might further obscure forensic evidence, especially when combined with anonymous remailers

46. Legal Liability As an employer or ISP, you may have some liability in regards to Usenet What about employees downloading warez or porn from work? What about hosting a NNTP server without proper access control? What if someone uploaded illegal material to it? Certainly the usual “sexually charged workplace” issues apply here The Business Software Alliance could also take you to task for pirated software

47. Investigation – on the Client There may be traces of Usenet files on a workstation that you could turn up, either in allocated space, or in deleted / slack space. It would behoove a forensic investigator to include looking for evidence of Usenet abuse in their standard operating procedures. Keyword searches might include: –usenet –nntp –news –binaries (or alt.binaries) –known NNTP server names and IP addresses (you might find a personal firewall log or something with entries to a Usenet server, even if the program was deleted) –Yenc strings (see later slides)

48. Investigation – on the Client File searches might include: –.RAR archive files –.PAR and.PAR2 parity files –.NFO description files –.NZB batch files –.ZIP files –Known NNTP clients (in file space and the registry) –Known Usenet search engines in browser caches (Guba, etc.)

49. Investigation – On the Network If you had access to the network of either a suspect ISP or a suspect computer, you could use a protocol analyzer to identify suspicious activity. As previously noted, it is possible to get Usenet binaries entirely over web connections using gateways, as well as by watching for actual NNTP traffic. Using firewall logs, or a protocol analyzer such as Ethereal you might look for connections such as: –HTTP / HTTPS (TCP port 80 and 443) connections to known Usenet web servers such as Guba –NNTP connections to any host on TCP port 119 –NNTP over SSL connections to any host on TCP or UDP port 563 –Traffic with a payload matching the keywords listed previously Of course, a crafty criminal will use the NNTP over SSL encryption option, or tunnel all of their connections through a SSH tunnel or something.

50. Investigation – At the Provider Theoretically, you can get a provider to help you with a court order of some kind. However, as we noted previously, a lot of them don’t keep any records at all, so this may be difficult If you are going to get any information at all from a provider, you’ll need to have that unique Message-ID field to work with. It is relatively certain that if you see a message with a Message- ID such as “Xns9734CAA2DE2A8orisitdunnocom@194.152.65.251” that the machine with the IP address of 194.152.65.251 was the one that originally took the posting You might also look at the PATH header. For example, our previous example had a path of: Border1.nntp.dca.giganews.com!nntp.giganews.com!feed2.new sreader.com!newsreader.com!npeer.de.kpn- eurorings.net!news.tele.dk!news.tele.dk!small.news.tele.dk!ne ws.astraweb.com!newsrouter-eu.astraweb.com!eweka!hq- usenetpeers.eweka.nl!81.171.88.219.MISMATCH!newsreader 30.eweka.nl!not-for-mail This might help you find a person with logs in the event of a spoofed Message-ID

51. Legal Citations Coming soon! We are hoping to do a review of cases for citable precedent

52. yEncode in Computer Forensics What is yEncode? –Encoding scheme for transmitting binary information in email and newsgroups –yEncode takes advantage of the entire 8- bit character set resulting in output only 1- 2% bigger than the original binary(compare to 40% for traditional 7 bit encoding)

53. yEncode in Computer Forensics Header –Single yEncoded binaries always begin with a header that contains an escape character(‘=‘), the keyword ‘ybegin’, and followed with parameters ‘line’, ‘size’ and ‘name’ as in the following example =yenc line=128 size-123456 name=mybinary.dat The filename must always be the last item on the line.

55. Trailer(footer) –Always begins with escape character ‘=‘ and ‘yend’ keywordand MUST contain the size of the original unencoded binary(in bytes) as in the below example =yend size=123456 yEncode in Computer Forensics

57. Verifying Integrity –yEncoded documents may include a 32 bit CRC value in the trailer to held decoders evaluate the integrity of the binary as seen below =yend size=123456 crc32=abcdef123 yEncode in Computer Forensics

58. Multi-part files –Due to size binaries are frequently split into multiple parts for transmission –This results in frequent unusable binaries due to missing parts and/or data corruption. –Multi-parts have standard ‘ybegin’ line and then additional keyword ‘part’ in header

59. yEncode in Computer Forensics Multi-part files (cont’d) –The keyword part specifies part number and identify it as a multi-part file –If ‘part’ is included an additional ‘ypart’ keyword line must follow which specifies the information about the part

60. yEncode in Computer Forensics Multi-part files(cont’d) –‘ypart’ keyword requires a being and end keyword specifying the information about the part –The file must end with a modified ‘ypart’ trailer line-an additional ‘part’ is added to specify the part number and must match the original one in the header.

61. yEncode in Computer Forensics Multi-part file (cont’d) –The trailer in a multi-part file must also contain a ‘pcrc’; keyword representing the CRC32 value of the preceding encoded part. It is also possible to encounter a CRC value for the entire encoded binary. –The ‘size’ keyword in multi-part trailers represents the size of the file part not the entire file

62. yEncode in Computer Forensics Multi-part files (cont’d) –To verify integrity a decoder must re- compute ‘begin’ and ‘end’ values in the ‘ypart’ line. –If the expected part size differs from the part size in the ‘yend’ line the file is corrupt. =ybegin part=1 total=10 line=128 size=500000 name=binary.exe =ypart begin=1 end=100000

63. yEncode in Computer Forensics

65. Suggestions for Post-Mortem Analysis yEncode in Computer Forensics

66. Our scope is going to be limited to three more popular newsgroup applications –Outlook Express –Mozilla Thunderbird –Free Agent from Forte yEncode in Computer Forensics

67. Outlook Express –EnCase Encase will search for and automatically decodes OE DBX files. EnCase will separate attachments for viewing- but will not reassemble multi-part files Pictures in newsgroups MUST be viewed in the email view they do not show up in the gallery Bookmarking must be done from the email view yEncode in Computer Forensics

68. FTK –FTK will not extract newsgroup DBX files into an easily readable format –FTK will show the yEncoded binary so you can copy out the binary for further processing yEncode in Computer Forensics

70. Thunderbird –FTK There is currently no implementation in FTK to read newsgroups from Thunderbird –EnCase The same applies in Encase for Thunderbird as applies for OE. Again, you must view attachments in the email view and bookmark there as well yEncode in Computer Forensics

71. Forte Agent –FTK There is currently no implementation in FTK to read newsgroups from Forte Agent-however like dbx files the yENC encoding can be seen and exported for analysis –EnCase The same applies in Encase for Thunderbird as applies for OE. Again, you must view attachments in the email view and bookmark there as well yEncode in Computer Forensics

72. Manual Decoding of yEncoded Binaries yEncode in Computer Forensics

73. Yenc32 (www.yenc32.com/download.php) –Free (yes, I said “free”) decoder that can be used with Outlook Express, Thunderbird and Forte agent as well as other newsgroup readers –Standalone program as well as right click integration with the gui –Tutorials on how to decode the above available online (http://www.yenc32.com/support.php) yEncode in Computer Forensics

74. Email Examiner (Mailbag Assistant) –Will read the downloaded newsgroup messages but will not decode the attachments (All common mailbox types-OE Thunderbird and Forte agent). –EMEX will tell you the status of the message (i.e. whether or not the binary still resides on the server or has been downloaded) –To decode with EMEX export the emails as a generic mailbox with the extension “.yenc” Right click on the file and decode with yenc32. yEncode in Computer Forensics

78. Linux –Convert DBX files to mbox format and import into a mail reader of choice. Decode using a yEnc decoder for Linux. http://www.yenc.org/linux.htm –For tips on how to decode email to a flat mbox format please see the whitepaper “Analysing Exchange and mbox emails using open source software” at http://www.forensicfocus.com/computer-forensics- papers yEncode in Computer Forensics

79. Thunderbird –Thunderbird as a newsgroup reader supports yEnc binaries(though it will not reconstruct multi-part files) –Convert dbx mailboxes to an mbox format ( as outlined in the aforementioned paper) and import into Thunderbird. Export your binaries. yEncode in Computer Forensics

80. Searches –Regular expression or keyword searches ‘=yenc’ or ‘=yend’ or ‘=ypart’ ‘name=xxx’ if the name of the suspect binary is known Use a tool such as Foremost (Linux or Cygwin), SMART or Data lifter with the ability to carve using header and footer-note this will not get CRC values or size since these values can be optional in the yEncoding standard yEncode in Computer Forensics

81. Questions and Comments Thanks!