Deploying Office 365 At UC Merced Nick Dugan and Todd Van Zandt UC Merced Information Technology UCCSC 2013

Introductions Nick Dugan, Senior Systems Administrator ndugan@ucmerced.edu Todd Van Zandt,  Director of Academic Technology and User Services tvanzandt@ucmerced.edu

Agenda Background Selection and Pilot Process Infrastructure Deployment Migration User Support Next Steps and Conclusions

Background UC Merced Opened 2005 Population in Fall 2012 5,800 students 140 faculty 1,000 staff 10,000 students by 2020

Background Technology ~8,000 active accounts ~10,000 alumni / former student accounts Lots of infrastructure from ~2005 Sun Messaging Server Oracle Calendar SPARC Hardware

Background Sun -> Oracle == $$$ Oracle Calendar EOL SPARC hardware refreshes Maintenance and license costs Oracle Calendar EOL Heavy use of Outlook with Oracle Calendar connector Incompatible with Outlook 2010 Client software not supported in Mac OS X Mountain Lion Email Growing Pains Quotas, Compromised accounts, SMTP blacklisting Aging webmail client

Background Minimally Utilized Windows Infrastructure Active Directory Identity Management: Sun Waveset LDAP: Sun Directory Server Active Directory Fed from IDM Lab support Printing NAS Authentication Minimal account attributes populated

Selection and Pilot Process Selection Committee Convened November 2011, charged with developing functional requirements and recommending a solution Representatives from faculty, staff, and student populations Locally hosted options ruled out Microsoft vs. Google Work completed March 2012

Selection and Pilot Process Microsoft vs. Google No clear consensus among committee members Subjective and objective scoring gave a slight advantage to Office 365 Both better than current solution High level of satisfaction with both High satisfaction with Lync among staff Final decision left to IT after consideration of committee report and analysis of costs

Selection and Pilot Process IT Review March – May 2012 Microsoft reduced price of A2 plan to $0 Evaluated technical requirements for implementation Reviewed committee input Spoke with other UC CIOs and IT organizations Final recommendation for Office 365 delivered to Chancellor and Provost

Infrastructure Deployment Consultant Engagement Decision made to employ professional services for tenant deployment and migration support Interviewed and vetted approx. 10 companies in July 2012 Selected Cloudbearing, initial engagement Sept. 2012

Infrastructure Deployment Active Directory Enhancements Domain Controllers in four physical locations (Merced x2, Berkeley, Fresno) ADFS Servers 2x ADFS, 2x ADFS proxy, load balanced across two physical locations DirSync Server HA deployment not possible

Infrastructure Deployment Why ADFS and DirSync? Apprehension about pushing passwords to the cloud IDM Already fed Active Directory Infrastructure vs. custom development trade-offs made sense

Infrastructure Deployment Lesson – Time is everything ADFS is sensitive to clock skew ADFS Proxies are not domain-joined by design No automatic clock sync with DCs Small time discrepancies lead to authentication outages Proxies need reliable NTP

Infrastructure Deployment Identity Management Integration Sun Waveset (Java) on Solaris EOL, Replacement in 2014 Office 365 == PowerShell IDM provision/deprovision events now trigger licensing actions on a remote PowerShell server Lots of waiting built into process due to (un)timeliness of events in o365 cloud Account provisioning went from real-time to “an hour or two”

Infrastructure Deployment Shibboleth / SSO integration Goal: Never see the ADFS sign-on page, OWA integration into campus portal

Infrastructure Deployment Shibboleth configured as a relying party in ADFS ADFS Home Realm Discovery page modified to bypass IdP selection box ADFS Logout handler modified to ignore SAML errors and redirect to global logout page We will help office365@ucdavis.edu

Infrastructure Deployment Dynamic Mailing Lists Sun Messaging does this well Office 365 does this… differently. Limited to default attributes We have dynamic lists based on lots of attributes we don’t push to AD Interim Solution: maintain Sun Messaging Server at lists.ucmerced.edu Distribution lists in o365 forward to @lists for expansion and ultimate delivery back to o365 Long-term Solution: Next-Gen IDM with proper group management

Migration - Email Approach and timing Early adopters, opt-ins (early-mid December) Faculty and Staff by affiliation/geographic location (Jan 2-20) Students alphabetically (Jan 14-25) Mailing lists (Feb 2013) Alumni (March-April 2013)

Migration – Email Technical considerations Respecting user settings Forwarding Vacation messages (didn’t do) Messages that won’t migrate (>25 MB) Identify and notify

Migration - Email Technical Process Set DirSync flag in AD so accounts are pushed to Cloud Assign licenses using PowerShell (not too early) wait Copy forwarding settings using PowerShell Change LDAP routing address user@merced.onmicrosoft.com Set migration password Migrate mail Remove migration password

Migration – Email Tool #1: Office 365 IMAP Migration Tool Pros: Cons: Free Fast (not subject to the same bandwidth throttling as non-MS tools) Easy Cons: ZERO reporting/logging. Only provides a count of “skipped” messages for each account – not which messages or why they were skipped.

Migration - Email Lesson: Never trust a tool that has no logging. Half-way through faculty/staff migration, number of skipped messages went through the roof Hundreds or thousands on some accounts, zero on others All attempts at resolution failed Session/resource issue with local IMAP server? – No Too many concurrent migrations? – No Microsoft able to provide an explanation or resolution? – No Panic? – Yes

Migration – Email Migration Tool #2 – MigrationWiz Contract with Cloudbearing provided up to 300 mailbox licenses Pros: Excellent Reporting Capabilities Excellent Support Easy to incorporate into migration workflow Cons: $11/mailbox == $$$$ Subject to Microsoft inbound traffic throttling

Migration - Email Lesson: Ask what other people are doing first. office365@ucdavis.edu

Migration - Email Migration Tool #3: imapsync http://imapsync.lamiral.info/ Pros: $50 (might as well be free) Excellent support Great reference from UCSB Cons: Required a bit of work to incorporate into our migration workflow Subject to Microsoft inbound traffic throttling

Migration - Email Schedule, bandwidth, timings “Just in time” mailbox provisioning was difficult Start data migration by 10pm, run until done Largest migration batch was ~1,100 accounts 40 simultaneous connections Generally finished by 8:00am

Migration - Email Lesson: Your carefully constructed process will experience its worst breakdown the day you migrate the Chancellor. Story

Migration – Email Jan 3, 2013 – Migration of Chancellor’s office 8:00am – Departmental account identified that did not provision successfully (sAMAccountName >20 chars) 10:00am – Sysadmin attempts to rectify problem by deleting and then re-provisioning account to Active Directory Sysadmin accidentally deletes ou=People instead of individual account

Migration – Email Recovery First thought – pull the (virtual) network plug on one of the DCs before changes propagate Too late Second thought – shut down DirSync server NOW Preserved accounts and mailboxes in cloud tenant ADFS non-functional, so as auth tokens expired, users lost access Third thought – Reprovision from IDM? No IDM group management, share permissions would be lost

Migration - Email Recovery Decision made to restore from tape and rebuild AD Backups were good, but VM restore process was failing 3:00pm – Support case with Symantec Overnight – work case with Symantec, upgrade server and client software Jan 4 morning – single DC restored Windows sysadmins rebuild AD forest Jan 5 9am – Full functionality restored

Migration – Email Lessons: AD Recycle Bin would have made this a non-issue Requires 2008 R2 domain functional level UCM was at 2003 Verify backups (AND restores) Prepare for the worst when migrating campus executives

Migration - Calendar Cloudbearing contract included migration of calendar data through third party (CalMover) All calendar data migrated to Office 365 at the same time Thursday, Jan 17 5:00pm – Oracle Calendar database extracted and uploaded to CalMover Friday, Jan 18 – receive PSTs from CalMover Fri-Sun – import PSTs to Office 365 Monday Jan 22 – Office 365 official UCM Calednar

Migration - Calendar Issues, side effects Modification or cancelation of migrated meetings would not notify attendees Contacts, tasks, and other non-calendar data was excluded from import Resource delegates, sharing and booking permissions did not migrate

Migration – Calendar Timing, speed, historical events Cloudbearing recommended limiting historical data to 12 months PST import could be very slow, might not finish over weekend window Individuals needing complete historical data would be accommodated through other means of import/export Application/service updates made the import much quicker than anticipated Allocated 3 days, completed in 6 hours

User Support Campus Communication Pilot Support Migration Support Documentation Training Client Issues

User Support - Communication Campuswide announcements News articles MSOs Info sessions Record and make available Inform about Benefits Cons / Changes (more important) Process , expectations Direct emails to users at the time of migration

User Support - Communication Website Large Banner to draw attention O365 project section included Migration schedule Configuration Guides List by device, not just service or client Training & Getting Started Guides Explanation of what is coming (large mailboxes, mobile support) , but also limitations and differences Update all old information pages, aside from Config Guides

User Support - Pilot Dedicated a technician to assist with installs Did not provide a lot of help Asked pilot group to review draft documentation IT Student Employees Engaged Resident Assistants (additional support) Lessons Learned Needed to have created formal testing plans Needed to have been more engaged in support

User Support - Migration Get Extra Help! We used temps, technically savvy dept staff, and IT students and staff Dedicated 2 (of 5) Desktop Support technicians to this effort once migrations started 2 DSS departed as we were starting Hired 4 temps to help with migration appointments First used them to create and finalize documentation for various client set ups Followed up on problems, after migrations completed for the day Identify technical leads in departments and target for early adoption Student Technology Consultants Interested IT staff

User Support - Migration Temps: The Bad: Not as engaged, less reliable, had to replace two The Good: Easy to hire, able to release when not needed, trained for the future, targeted at O365 tasks Suggestions: Interview more so you have extras “on-hand”; Have projects lined up for them in downtime; Keep an eye on them

User Support - Migration Coordinated with Depts (opt-in) Took a LOT of effort Difficulties with scheduling 60-120 per group; Only so many depts on a day Plus additional one off early adopters Coordinating postponements for out of office users Allowed us to test our processes and improve documentation

User Support - Migration Informed Depts for mass moves Chose by location; focused support to an area Tables in the lobbies (rarely used) Self-help documentation was good Morning appts for high profile groups Most done within 4-6hrs Everyone wanted help immediately OWA was an option Many users (faculty and lecturers) not present

User Support - Migration Staff Very interested; needed help and assistance Many Windows Outlook users Remove Outlook Oracle Connector Reattach Personal Folders and Archives Import Address Books and Autocomplete lists Upgrade to Office 2010 (became too lengthy) Faculty Many were out of office Many self-configured their clients

User Support - Migration Students Mostly web access users so just adapted to using OWA Or still forwarded their email to Gmail, etc. Switched webmail in Campus Portal if migrated Did not read email to understand process; if went to direct to MyMail link Migration of lots of students took longer for data to copy, new mailbox looked empty except for new Offered trainings, did not attend More critical for Grad students (instruction and research) Finally moved them en masse; should’ve planned that from start

User Support - Documentation Configuration of Clients Outlook, Thunderbird, MacMail, Mac Outlook What about calendar for Oracle Calendar Client users What about Lightning, Mac Calendar? Configuration of Mobile Devices Use of new systems (OWA)

User Support - Documentation Didn’t do well documenting for Outlook 2007  2010 Didn’t do well documenting Outlook calendar for old Oracle Calendar client users Documenting the use of Oracle Calendar web client for early adopters who were Outlook users

User Support - Training Pre-migration What to expect, limitations and differences, overview of OWA and Mac/PC Outlook Post-migration OWA Calendar Lync iOS Dept or Individual Request Specific questions

User Support – Client Issues Calendar Mac Outlook & Outlook 2007 Viewing & Sharing calendars Acting on calendars in Mac Outlook (selected vs. checked) Client differences for viewing multiple calendars Overlay, Side-by-side, Both Setting delegates; only Outlook client or powershell Needed to have done more testing with delegates and room calendars to understand needs and functionality The wonderful subculture of Exec Administrative Assistants Breaking bad processes established by deficiencies in previous Calendar Adding Holidays

User Support – Client Issues Email Forwarding to multiple users Away Messages when Forwarding is set Creation of Distribution Groups (limiting) Attachment / Message Size limits Limits on Mass Mailing to unique accounts MacMail – sync stops Thunderbird – archive issue iOS – disappearing messages Outlook – Unread Email and Sync logs & Modification Resolution Recovering deleted emails and calendar appointments

User Support – Client Issues OWA Different on mobile browsers Settings are limited on mobile Lync Mac client Login issues Features not as robust Considerations for default presence; All visible or All offline? All visible – encourages use of enterprise IM All offline – Faculty and Administrators not public Can’t separate students easily if all on the same tenant

What’s Next “Wave 15” Service Upgrade Back-end upgrade to 2013 Office products Significant web UI changes Just in time to make our new documentation obsolete Some time between now and end of year Self-service web portal for resource management and calendar delegation Changing booking options for resources Additional means of managing calendar permissions for non-Outlook users

Conclusions What have we gained? Modern messaging and calendaring platform 25x quota increase Enterprise IM Proper mobile device support +1 critical enterprise service (ADFS)

Conclusions Dollars and cents 1 FTE freed to work on other projects +6 Windows servers (VM) -3 SPARC servers $17,000 annual hardware/software maintenance Money we won’t spend Hardware refreshes (servers, SAN) Ongoing (increasing) Oracle licensing costs

Conclusions What have we lost? Control Upgrades and changes (announced and otherwise) happen when they happen Outages

Questions? ndugan@ucmerced.edu tvanzandt@ucmerced.edu office365@ucdavis.edu