Server Upgrades CPTE 433 John Beckett

Tool #1: The Worksheet A blank/lined piece of paper. Go through it thrice. Work through each step yourself Work through.. With techs Work through each step with the stakeholders. –Don’t just ask them “Is this OK?”. –Tell them how this will affect their operation. –Tell them what’s expected of them. –Note what they expect of you.

Tool #2: Service Checklist What services are provided by the service? Who are the customers of each service? What software package provides which service? What data sources are required for each service? Who/what depends on each service

For Each Service… Verify that the software package will work with the new OS or plan an upgrade path. Develop a test that will verify that it is working. –Part of the definition of “done”. –May require user participation. –E.g.

Plan the Transition Write a back-out plan. –Multiple stages along the path? Select a maintenance window. Announce the upgrade as appropriate. –Over-communicate because they don’t get it the first time or the same time. Execute the tests developed early to make sure they are correct.

Do the Upgrade Do the upgrade with a mentor watching/helping. Repeat all the tests developed earlier. Use the back-out plan if necessary –Re-test to make sure it is working Communicate the completion/back-out to the customers. (“Thank you”)

Service Checklist This drives the entire process. A server exists to serve. Spreadsheet on the Web –People can get the latest version of your plan. –They might not notice changes you’ve made Send notifications with an attached copy of spreadsheet –Indicate changes made since the last version. –A change log (last changes show at the top) is a good idea.

Customer Dependency Check Meeting includes customers plus their support techs. Resolve techie arguments beforehand. Focus on the service and implications to operations.

What If You Don’t Know? If you don’t know what services are on a machine… Look at where software might be installed. Scan for open ports. Ask people. If you’re still not sure: –Try a short test-down period to see what goes down when that server is offline. Beware of cacheing issues.

Software Compatibility Issues RTFM Bring up a test server –Good reason to have a spare server. –The same server could be used in the event of a hardware failure. –Beware of naming conflicts. What if the new version of the software requires a new version of the OS? –Test that combination with your config & data!

Verification Tests Emphasis on designing verification tests ahead of time. Weak points: –“What didn’t we get working right away after the last upgrade?” –“What’s the most important thing that doesn’t always work?” –Beware of over-emphasizing something you have really solved, and letting something else creep up on you.

Maintenance Window When? Ask your customers! How long? –Backing up –Making the changeover –Installing s/w & data –Testing –Backout –“Fudge Factor” / backout time

“Monday Night Carte Blanche” Excellent idea in the past. Still a good idea for changes to the system But how much is actually testing? –Do “space-division multiplexing” – spare server – to conserve on downtime. Users are like cats… –They prefer predictability even if it isn’t exactly what they want.

What Does The Mentor Do? One person does the upgrade. The other person records what happened.

Communicate the Change What you were attempting to do That it was completed within time-frame –No need to brag that it went faster – that simply increases expectations. –If over, be brief about your tribulations. –They don’t care how hard you worked. They do care what happened to them. Current status Additional things you discovered along the way

Re-Use Your Tests Baseline for whether things work You look really stupid if a previous problem re-surfaces on a subsequent upgrade –But that happens all too often! –…especially on changes made in the response to problems. –Re-read your wrap-up communiqué from last time to make sure you’ve dealt with last time’s problems.

Dress Rehearsal For some upgrades you may wish to do a “dress rehearsal”. Separate host: ideal –Except that it does not affect other systems the same Same host: –Same environment –Save the hard drive and restore it Beware of transactions processed