1 Information Systems 2/26/03 Tom Coppeto Mark Silis MIT Mail System Update 26 February 2003

2 Information Systems 2/26/03 Tom Coppeto Mark Silis Agenda Introduction to the mail system New Features Ongoing Work

3 Information Systems 2/26/03 Tom Coppeto Mark Silis The Mail System Mailhub Internet MIT Users DMZ (MX mit.edu) Outgoing Post Office Other MIT Mailers

4 Information Systems 2/26/03 Tom Coppeto Mark Silis The Outgoing Mailers The first hop in all originating mail –A relay dedicated to application response sender addressing –Catches some mail anomalies –Will be the place where future tightening occurs Outgoing Internet MIT Mail System MIT Users

5 Information Systems 2/26/03 Tom Coppeto Mark Silis The mit.edu (DMZ) Mailers Provides queuing for external deliveries to keep the internal queues small The place outside mailers beat up Contains an alias database (users & lists) of 132,496 entries DMZ Internet MIT Mail System MIT Users

6 Information Systems 2/26/03 Tom Coppeto Mark Silis The Mail Hub The core of the mail system Creates a 3-tier system that optimizes for local traffic and better insulates the system from Internet anomalies Also contains an aliases database

7 Information Systems 2/26/03 Tom Coppeto Mark Silis The Post Offices Provides storage Accepts deliveries from the mailhub and dmz Supports pop/krb4, imap/krb4 and imap/ssl This has been the focal point of the recent upgrades

8 Information Systems 2/26/03 Tom Coppeto Mark Silis Post Office Upgrade quotas 250MB max message size 25MB new software –Spam management –Auto response

9 Information Systems 2/26/03 Tom Coppeto Mark Silis About Spam No one likes it There’s no way to exactly know if a message is or is not wanted We can offer some heuristics that allow you to decide what to do We will not intentionally delete messages on your behalf This requires processing the content of messages which is new and very expensive to do centrally With the software and hardware available today, we believe we can pull this off

10 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Management New functionality added to post offices Comprised of: –spam scoring –spam filtering client side server side –automatic expunging of server side spam folders

11 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Scoring header & body analysis – scores may be positive or negative –the higher the score, the more likely it is spam –more specifically, the more tests it failed –this is inexact score of 7.5 required to flag the message as spam –user configurable

12 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Scoring header Subject includes "viagra" header Subject contains "Your Bills" or similar 0.7 header Missing Date: header 1.37 body Yahoo! Groups message header From: address is in the user's white-list -100 body HTML has a low ratio of text to image area 1.101

13 Information Systems 2/26/03 Tom Coppeto Mark Silis To: From: "James D. Bruce" Subject: important: please read X-Spam-Score: 9.5 X-Spam-Level: ********* (9.5) X-Spam-Flag: YES X-Scanned-By: MIMEDefang 2.28 Spam Scoring (2)

14 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Scoring (3) –set scoring threshold –allow lists specify list of From: addresses that you never want to be flagged as spam –deny lists specify list of From: addresses that you always want to be flagged as spam

15 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Filtering - Client Side client side filtering –set up your client to filter on the X- Spam-Flag field –set up your client to filter on the X- Spam-Level field (requires no threshold configuration) see

16 Information Systems 2/26/03 Tom Coppeto Mark Silis Spam Filtering - Server Side You may elect to have messages flagged as spam filed to a separate IMAP folder Must be using IMAP to see the messages sent to this folder This folder counts as quota You can also elect to have this folder expunged after 14 days

17 Information Systems 2/26/03 Tom Coppeto Mark Silis Spamscreen Web Page

18 Information Systems 2/26/03 Tom Coppeto Mark Silis Spamscreen Web Page (2)

19 Information Systems 2/26/03 Tom Coppeto Mark Silis Spamscreen Web Page (3)

20 Information Systems 2/26/03 Tom Coppeto Mark Silis Automatic Response Delivery automatic replies upon delivery Challenges are technical –Don’t melt down mail system (will it happen? probably) and cultural –Not everyone thinks this is a feature (spam filtering to the rescue!)

21 Information Systems 2/26/03 Tom Coppeto Mark Silis Automatic Response

22 Information Systems 2/26/03 Tom Coppeto Mark Silis SMTP Authentication MIT mail relays abused by spammers Outgoing is a quasi-open relay Need to further tighten outgoing to stop this The answer is SMTP authentication Only authorized users should be allowed to originate mail sent through the mail system

23 Information Systems 2/26/03 Tom Coppeto Mark Silis Problems Under Full Load Deliveries outbound from spam filtering too inefficient under load (fixed) The new cyrus software dependent on a single process (some workarounds) The database cache size too small (fixed) Some messages may bypass scoring in the event of a process failure or the need to handle a backlog (we just need to get better at managing this)

24 Information Systems 2/26/03 Tom Coppeto Mark Silis Statistics

25 Information Systems 2/26/03 Tom Coppeto Mark Silis Statistics (2)

26 Information Systems 2/26/03 Tom Coppeto Mark Silis Summary of Changes Post office components upgraded Quotas increased to 250MB Max message size increased to 25MB Spam scoring Optional spam re-filing Auto responder Web based controls of spam and auto response

27 Information Systems 2/26/03 Tom Coppeto Mark Silis Stabilize the system Rollout spam and responder functionality Solidify recommended clients Upgrade the hardware/software on the outgoing, mailhub and dmz mailers Enable SMTP authentication on outgoing mailers Simplify the system by centralizing the spam scoring Look at new features Next Steps