1. Deploying Office 365 At UC Merced
Nick Dugan and Todd Van Zandt
UC Merced Information Technology
UCCSC 2013

2. Introductions Nick Dugan, Senior Systems Administrator
Todd Van Zandt,  Director of Academic Technology and User Services

3. Agenda Background Selection and Pilot Process
Infrastructure Deployment
Migration
User Support
Next Steps and Conclusions

4. Background UC Merced Opened 2005 Population in Fall 2012
5,800 students
140 faculty
1,000 staff
10,000 students by 2020

5. Background Technology ~8,000 active accounts
~10,000 alumni / former student accounts
Lots of infrastructure from ~2005
Sun Messaging Server
Oracle Calendar
SPARC Hardware

6. Background Sun -> Oracle == $$$ Oracle Calendar EOL
SPARC hardware refreshes
Maintenance and license costs
Oracle Calendar EOL
Heavy use of Outlook with Oracle Calendar connector
Incompatible with Outlook 2010
Client software not supported in Mac OS X Mountain Lion
Growing Pains
Quotas, Compromised accounts, SMTP blacklisting
Aging webmail client

7. Background Minimally Utilized Windows Infrastructure Active Directory
Identity Management: Sun Waveset
LDAP: Sun Directory Server
Active Directory
Fed from IDM
Lab support
Printing
NAS Authentication
Minimal account attributes populated

8. Selection and Pilot Process
Selection Committee
Convened November 2011, charged with developing functional requirements and recommending a solution
Representatives from faculty, staff, and student populations
Locally hosted options ruled out
Microsoft vs. Google
Work completed March 2012

9. Selection and Pilot Process
Microsoft vs. Google
No clear consensus among committee members
Subjective and objective scoring gave a slight advantage to Office 365
Both better than current solution
High level of satisfaction with both
High satisfaction with Lync among staff
Final decision left to IT after consideration of committee report and analysis of costs

10. Selection and Pilot Process
IT Review
March – May 2012
Microsoft reduced price of A2 plan to $0
Evaluated technical requirements for implementation
Reviewed committee input
Spoke with other UC CIOs and IT organizations
Final recommendation for Office 365 delivered to Chancellor and Provost

11. Infrastructure Deployment
Consultant Engagement
Decision made to employ professional services for tenant deployment and migration support
Interviewed and vetted approx. 10 companies in July 2012
Selected Cloudbearing, initial engagement Sept. 2012

12. Infrastructure Deployment
Active Directory Enhancements
Domain Controllers in four physical locations
(Merced x2, Berkeley, Fresno)
ADFS Servers
2x ADFS, 2x ADFS proxy, load balanced across two physical locations
DirSync Server
HA deployment not possible

13. Infrastructure Deployment
Why ADFS and DirSync?
Apprehension about pushing passwords to the cloud
IDM Already fed Active Directory
Infrastructure vs. custom development trade-offs made sense

16. Infrastructure Deployment
Lesson – Time is everything
ADFS is sensitive to clock skew
ADFS Proxies are not domain-joined by design
No automatic clock sync with DCs
Small time discrepancies lead to authentication outages
Proxies need reliable NTP

17. Infrastructure Deployment
Identity Management Integration
Sun Waveset (Java) on Solaris
EOL, Replacement in 2014
Office 365 == PowerShell
IDM provision/deprovision events now trigger licensing actions on a remote PowerShell server
Lots of waiting built into process due to (un)timeliness of events in o365 cloud
Account provisioning went from real-time to “an hour or two”

18. Infrastructure Deployment
Shibboleth / SSO integration
Goal: Never see the ADFS sign-on page, OWA integration into campus portal

19. Infrastructure Deployment
Shibboleth configured as a relying party in ADFS
ADFS Home Realm Discovery page modified to bypass IdP selection box
ADFS Logout handler modified to ignore SAML errors and redirect to global logout page
We will help

20. Infrastructure Deployment
Dynamic Mailing Lists
Sun Messaging does this well
Office 365 does this… differently.
Limited to default attributes
We have dynamic lists based on lots of attributes we don’t push to AD
Interim Solution: maintain Sun Messaging Server at lists.ucmerced.edu
Distribution lists in o365 forward for expansion and ultimate delivery back to o365
Long-term Solution: Next-Gen IDM with proper group management

21. Migration - Email Approach and timing
Early adopters, opt-ins (early-mid December)
Faculty and Staff by affiliation/geographic location (Jan 2-20)
Students alphabetically (Jan 14-25)
Mailing lists (Feb 2013)
Alumni (March-April 2013)

22. Migration – Email Technical considerations Respecting user settings
Forwarding
Vacation messages (didn’t do)
Messages that won’t migrate (>25 MB)
Identify and notify

23. Migration - Email Technical Process
Set DirSync flag in AD so accounts are pushed to Cloud
Assign licenses using PowerShell (not too early)
wait
Copy forwarding settings using PowerShell
Change LDAP routing address
Set migration password
Migrate mail
Remove migration password

24. Migration – Email Tool #1: Office 365 IMAP Migration Tool Pros: Cons:
Free
Fast (not subject to the same bandwidth throttling as non-MS tools)
Easy
Cons:
ZERO reporting/logging.
Only provides a count of “skipped” messages for each account – not which messages or why they were skipped.

25. Migration - Email Lesson: Never trust a tool that has no logging.
Half-way through faculty/staff migration, number of skipped messages went through the roof
Hundreds or thousands on some accounts, zero on others
All attempts at resolution failed
Session/resource issue with local IMAP server? – No
Too many concurrent migrations? – No
Microsoft able to provide an explanation or resolution? – No
Panic? – Yes

26. Migration – Email Migration Tool #2 – MigrationWiz
Contract with Cloudbearing provided up to 300 mailbox licenses
Pros:
Excellent Reporting Capabilities
Excellent Support
Easy to incorporate into migration workflow
Cons:
$11/mailbox == $$$$
Subject to Microsoft inbound traffic throttling

27. Migration - Email Lesson: Ask what other people are doing first.

28. Migration -
Migration Tool #3: imapsync
Pros:
$50 (might as well be free)
Excellent support
Great reference from UCSB
Cons:
Required a bit of work to incorporate into our migration workflow
Subject to Microsoft inbound traffic throttling

29. Migration - Email Schedule, bandwidth, timings
“Just in time” mailbox provisioning was difficult
Start data migration by 10pm, run until done
Largest migration batch was ~1,100 accounts
40 simultaneous connections
Generally finished by 8:00am

30. Migration -
Lesson: Your carefully constructed process will experience its worst breakdown the day you migrate the Chancellor. Story

31. Migration – Email Jan 3, 2013 – Migration of Chancellor’s office
8:00am – Departmental account identified that did not provision successfully (sAMAccountName >20 chars)
10:00am – Sysadmin attempts to rectify problem by deleting and then re-provisioning account to Active Directory
Sysadmin accidentally deletes ou=People instead of individual account

34. Migration – Email Recovery
First thought – pull the (virtual) network plug on one of the DCs before changes propagate
Too late
Second thought – shut down DirSync server NOW
Preserved accounts and mailboxes in cloud tenant
ADFS non-functional, so as auth tokens expired, users lost access
Third thought – Reprovision from IDM?
No IDM group management, share permissions would be lost

35. Migration - Email Recovery
Decision made to restore from tape and rebuild AD
Backups were good, but VM restore process was failing
3:00pm – Support case with Symantec
Overnight – work case with Symantec, upgrade server and client software
Jan 4 morning – single DC restored
Windows sysadmins rebuild AD forest
Jan 5 9am – Full functionality restored

36. Migration – Email Lessons:
AD Recycle Bin would have made this a non-issue
Requires 2008 R2 domain functional level
UCM was at 2003
Verify backups (AND restores)
Prepare for the worst when migrating campus executives

37. Migration - Calendar
Cloudbearing contract included migration of calendar data through third party (CalMover)
All calendar data migrated to Office 365 at the same time
Thursday, Jan 17 5:00pm – Oracle Calendar database extracted and uploaded to CalMover
Friday, Jan 18 – receive PSTs from CalMover
Fri-Sun – import PSTs to Office 365
Monday Jan 22 – Office 365 official UCM Calednar

38. Migration - Calendar Issues, side effects
Modification or cancelation of migrated meetings would not notify attendees
Contacts, tasks, and other non-calendar data was excluded from import
Resource delegates, sharing and booking permissions did not migrate

39. Migration – Calendar Timing, speed, historical events
Cloudbearing recommended limiting historical data to 12 months
PST import could be very slow, might not finish over weekend window
Individuals needing complete historical data would be accommodated through other means of import/export
Application/service updates made the import much quicker than anticipated
Allocated 3 days, completed in 6 hours

40. User Support Campus Communication Pilot Support Migration Support
Documentation
Training
Client Issues

41. User Support - Communication
Campuswide announcements
News articles
MSOs
Info sessions
Record and make available
Inform about
Benefits
Cons / Changes (more important)
Process , expectations
Direct s to users at the time of migration

42. User Support - Communication
Website
Large Banner to draw attention
O365 project section included
Migration schedule
Configuration Guides
List by device, not just service or client
Training & Getting Started Guides
Explanation of what is coming (large mailboxes, mobile support) , but also limitations and differences
Update all old information pages, aside from Config Guides

43. User Support - Pilot Dedicated a technician to assist with installs
Did not provide a lot of help
Asked pilot group to review draft documentation
IT Student Employees
Engaged Resident Assistants (additional support)
Lessons Learned
Needed to have created formal testing plans
Needed to have been more engaged in support

44. User Support - Migration
Get Extra Help!
We used temps, technically savvy dept staff, and IT students and staff
Dedicated 2 (of 5) Desktop Support technicians to this effort once migrations started
2 DSS departed as we were starting
Hired 4 temps to help with migration appointments
First used them to create and finalize documentation for various client set ups
Followed up on problems, after migrations completed for the day
Identify technical leads in departments and target for early adoption
Student Technology Consultants
Interested IT staff

45. User Support - Migration
Temps:
The Bad: Not as engaged, less reliable, had to replace two
The Good: Easy to hire, able to release when not needed, trained for the future, targeted at O365 tasks
Suggestions: Interview more so you have extras “on-hand”; Have projects lined up for them in downtime; Keep an eye on them

46. User Support - Migration
Coordinated with Depts (opt-in)
Took a LOT of effort
Difficulties with scheduling
per group;
Only so many depts on a day
Plus additional one off early adopters
Coordinating postponements for out of office users
Allowed us to test our processes and improve documentation

47. User Support - Migration
Informed Depts for mass moves
Chose by location; focused support to an area
Tables in the lobbies (rarely used)
Self-help documentation was good
Morning appts for high profile groups
Most done within 4-6hrs
Everyone wanted help immediately
OWA was an option
Many users (faculty and lecturers) not present

48. User Support - Migration
Staff
Very interested; needed help and assistance
Many Windows Outlook users
Remove Outlook Oracle Connector
Reattach Personal Folders and Archives
Import Address Books and Autocomplete lists
Upgrade to Office 2010 (became too lengthy)
Faculty
Many were out of office
Many self-configured their clients

49. User Support - Migration
Students
Mostly web access users so just adapted to using OWA
Or still forwarded their to Gmail, etc.
Switched webmail in Campus Portal if migrated
Did not read to understand process; if went to direct to MyMail link
Migration of lots of students took longer for data to copy, new mailbox looked empty except for new
Offered trainings, did not attend
More critical for Grad students (instruction and research)
Finally moved them en masse; should’ve planned that from start

50. User Support - Documentation
Configuration of Clients
Outlook, Thunderbird, MacMail, Mac Outlook
What about calendar for Oracle Calendar Client users
What about Lightning, Mac Calendar?
Configuration of Mobile Devices
Use of new systems (OWA)

51. User Support - Documentation
Didn’t do well documenting for Outlook 2007  2010
Didn’t do well documenting Outlook calendar for old Oracle Calendar client users
Documenting the use of Oracle Calendar web client for early adopters who were Outlook users

52. User Support - Training
Pre-migration
What to expect, limitations and differences, overview of OWA and Mac/PC Outlook
Post-migration
OWA
Calendar
Lync
iOS
Dept or Individual Request
Specific questions

53. User Support – Client Issues
Calendar
Mac Outlook & Outlook 2007
Viewing & Sharing calendars
Acting on calendars in Mac Outlook (selected vs. checked)
Client differences for viewing multiple calendars
Overlay, Side-by-side, Both
Setting delegates; only Outlook client or powershell
Needed to have done more testing with delegates and room calendars to understand needs and functionality
The wonderful subculture of Exec Administrative Assistants
Breaking bad processes established by deficiencies in previous Calendar
Adding Holidays

54. User Support – Client Issues
Forwarding to multiple users
Away Messages when Forwarding is set
Creation of Distribution Groups (limiting)
Attachment / Message Size limits
Limits on Mass Mailing to unique accounts
MacMail – sync stops
Thunderbird – archive issue
iOS – disappearing messages
Outlook – Unread and Sync logs & Modification Resolution
Recovering deleted s and calendar appointments

55. User Support – Client Issues
OWA
Different on mobile browsers
Settings are limited on mobile
Lync
Mac client
Login issues
Features not as robust
Considerations for default presence; All visible or All offline?
All visible – encourages use of enterprise IM
All offline – Faculty and Administrators not public
Can’t separate students easily if all on the same tenant

56. What’s Next “Wave 15” Service Upgrade
Back-end upgrade to 2013 Office products
Significant web UI changes
Just in time to make our new documentation obsolete
Some time between now and end of year
Self-service web portal for resource management and calendar delegation
Changing booking options for resources
Additional means of managing calendar permissions for non-Outlook users

57. Conclusions What have we gained?
Modern messaging and calendaring platform
25x quota increase
Enterprise IM
Proper mobile device support
+1 critical enterprise service (ADFS)

58. Conclusions Dollars and cents 1 FTE freed to work on other projects
+6 Windows servers (VM)
-3 SPARC servers
$17,000 annual hardware/software maintenance
Money we won’t spend
Hardware refreshes (servers, SAN)
Ongoing (increasing) Oracle licensing costs

59. Conclusions What have we lost? Control
Upgrades and changes (announced and otherwise) happen when they happen
Outages

60. Questions? ndugan@ucmerced.edu tvanzandt@ucmerced.edu