9 UTORmail Re-Architecture UTORmail has scaled from 1995 one small server to today 13 servers. From 2 GB to 3/4 TB. We asked whether existing architecture is suitable to meet future demands. Some of the issues addressed follow...
10 UTORmail Re-Architecture: servers Examined single server vs multiple servers. Concluded existing multi-server is effective. Concluded lack of infrastructure for many more servers so will consolidate to fewer bigger ones.
11 UTORmail Re-Architecture: Linux Currently using Sun servers: examined Sun versus Linux Intel servers Tested Sun & Intel models considering buying. After heavy discounting Sun was 2x the price Linux was 2.5x faster encrypting, up to 12x faster in some post office functions. Linux Intel wins.
12 UTORmail Re-Architecture: Storage Storage use doubles every year High performance requirement. Management nightmare: – A disk failure a month – Only protection is nightly backups – A restore from backups could cost hours of down time and loss of a day worth of new messages – Difficult to balance load across disks – Uneven distribution causing problems for backups
14 De-couple Server Upgrades from Disks for Storing Mail UTORmail server upgrades need to happen before September. Requirements and proposals for disk storage for storing mail will be brought forward at another time. (SAN, disk array, JBOD, RAID, etc.) Disk storage solution cannot be put in before September
16 Message Store: Normal Growth In past added 1-2 servers per year normal growth – Increase in number of customers – Increase in transactions per customer – Increase in size of messages – Increase in use of http://my.utoronto.ca webmail which pounds the message stores Regular Alarms now: trivial transactions taking too long (>10seconds). Regularly peaks of 15-20 seconds. Maximum peaks up to 45 seconds.
17 Message Store: UTORauth Current message store connections in clear: spy can see everything including passwords. Must encrypt for UTORauth. Encryption (SSL is standard) requires huge amounts of CPU power Existing Message Store CPUs too slow. Looked at alternatives: encryption co-processor vs new servers, latter more cost-effective Conclusion: Need to upgrade all message stores
18 Message Store: not disks We propose to upgrade the server but leave the existing disks (will copy all message files from Solaris to Linux format using same disks) Will bring forth proposal for alternative to existing disks at a later point in time.
19 Message Store: Inaction Consequence Can't accommodate normal growth in accounts (let alone double cohort) Passwords flow in clear: forget UTORauth Existing customers will see >1minute delays when clicking to view messages
20 External SMTP: Normal Growth Have had 2 old slow servers for years; this year added a 3 rd to keep up with load. This year 60% growth in message numbers compared to 30% in previous years. Servers no longer keeping up: regular periods when queues build resulting in long delays. Spam episodes result in multi-hour delays because servers so close to the edge. Conclusion: Need to add more horsepower.
21 External SMTP: Inaction Consequence By September will not be able to keep up with normal number of messages from other post offices. – Queues which keep growing in size. – Mail indefinitely delayed. – Complete meltdown of service (messages arrive faster than can be delivered.)
22 Number of Messages per Day* Messages339,00012,000MB Recipients429,00018,000MB Internal SMTP Server UTORmail to UTORmail 96,000 5,000MB UTORmail to external109,000 6,000MB External SMTP Server external to UTORmail223,000 7,000MB * Stats for May 27, 2003
23 Internal SMTP: Normal Growth Have used the same old slow 2 servers for years. Currently at capacity. Before September, must increase number of Internal SMTP servers; intend to use other UTORmail servers coming out of production. May need to put in faster SMTP servers later in the year.
24 Internal SMTP: Inaction Consequence By September, long delays when the send button is pushed by customers. By mid-fall, delays in delivering messages. Depending on continued internal growth in usage, possible meltdown (messages arriving faster than they can be delivered, resulting in service collapse.)
31 Spam Update TechKnowFile.03 Opportunity for feedback Spam problem seems more pervasive Selected four areas to attempt quantification Dentistry, Law, SIS, AMS to be surveyed Asked to measure spam for a week to quantify and examine impact on users
32 Dentistry Users reported spending 5-10 min/day on Spam Worse for those in field using slow WebMail 10 Min becomes 30min/day in Internet Café at $$ Disincentive to publish contact info for collaboration, as it attracts Spam May impair growth of digital publishing
37 AMS G.Kemp monitored all his mail for May 1,237 emails received in Total 584, or >47% were Spam If deletion took10 secs each, amounts to 97 min. Some have to be opened to verify Spammness, so even more time consumed AMS staff results to be presented
38 Nature of Complaints Increasing frustration with junk: Viagra, cheap mortgages, get rich quick schemes, free digital cable, penis/breast enlargement pills Sexually explicit offers becoming much more offensive Some complain it amounts to sexual harassement Could lead to unsafe workplace problems Growing ANGER, not just nuisance
39 More Complaints 150 Span messages after 4 day absence!!! I dont care if youre too stupid to fix it, just let me KNOW if you are!! Some offices expect mail from outside strangers, so many messages must be opened in order to verify nature Now some viruses generate Spam Increasing # users are demanding action
40 Costs to Remedy Spam Assassin to mark suspected Spam for user to file separately or just delete if trusted PureMessage License cost is C$5K/cpu oto Anti-Spam Defns + Support = C$3.2k/yr/cpu CPUs approx $8k oto,so $13Koto + $3.2K/yr/cpu 6 CPU = $78K + $19.2K/yr + 1/4 FTE ($20K)
41 Spam Filtering Proposed costs for UTORmail load Deptl mail systems doing own thing No knowledge of deptl mail systems or load Technically feasible to funnel all mail through Spam filter and then back to deptl systems Deploy on UTORmail, incrementally add Deptl until need more processors
42 PureMessage Testimonials U.Wash has 120k mailboxes, 800k msgs/day Run 13 servers, 5 out, rest anti-span and virus 40% of inbound mail is Spam Tag messages as Spam Stanford has 22k user accounts, 500K msgs/day Run 7 Dual-cpu SUN Netra 20 servers, Solaris Spam not blocked but tagged
43 PureMessage Testimonials Alabama ISP has 11k Users, 1M msgs/day Claim Incredible Accuracy Cuts 40% of their traffic by blocking IndianaU CS get 14K emails/day, 3.5K Spam Run PM on 1 Dual 1.4GHz Athlon, 1GB RAM Identify close to 100% of Spam
44 Pause for Discussion (Also end of Presentation)