TOI - Refresh Upgrades in Cisco Unity Connection 8.6 Saumya Saxena

What is a Refresh Upgrade? New upgrade implementation starting in Cisco Unity Connection 8.6 Upgrades from older versions of RHEL cannot be accomplished using the standard L2 upgrade process in Cisco Unity Connection Performs a “clean install” into the inactive partition, rather than installing to the inactive partition while the old version remains running Commonly referred to as an “RU” (or Refresh Upgrade) Mention that CUCM is implementing Refresh Upgrades, and that Connection is essentially plugging-in to that framework so that we can perform a Refresh Upgrade as well. The third bullet is meant to just emphasize that with L2 upgrades the system was functional during the install, but not for the switch-version. But with RU, the system will instead actually be mostly non-functional for the duration of the upgrade.

How is this different than previous upgrades? Requires a COP file to be installed prior to beginning the RU (without the COP file, attempts to upgrade to 8.6 will be blocked from the start) Looks like a “clean install” when viewing local console of machine No switch-version (that the administrator will know about) Post-upgrade CLI / GUI question is changed (be careful of the default): CLI – Do not automatically switch back to the previous version after the upgrade completes Web – Do not reboot to the previous partition after the upgrade completes “Reconfiguration and Upgrade Guide for Cisco Unity Connection 8.x” will have more detailed information For the first bullet, the point that’s important is that if you don’t install the COP file prior to starting an RU, if you try to upgrade to 8.6 the upgrade won’t start at all, it will say that 8.6 is an unsupported upgrade version. So while not installing the COP file will prevent an RU from succeeding, it won’t be a reason why an RU might fail either (since the upgrade will not have even started yet if you didn’t install the COP file at all). For the second bullet, I mention this because if you are viewing the local console of the machine during the RU process, it will look like the system is being clean installed, and that is expected behavior; the previous partition isn’t being overwritten (unless you have a 7825-H3 or 7828-H3). For the third bullet, I mention this because in L2 upgrades, it was a two step process (install, then switch-version). With RU, it’s all taken care of in the one process, it’s not separated anymore. For the fourth bullet, mention that the Web and CLI will have a new question to ask before starting the upgrade. Previously, you would be asked if you want to switch- versions immediately after the upgrade completes (CLI), or reboot to the new version immediately after the upgrade completes (Web). However, now the question will ask if you want to automatically go back to the previous version after the upgrade completes (the web and cli word the question differently, as noted above). If you answer this incorrectly, you will not be able to execute a switch- version and get to 8.6, that is not supported. Fifth bullet is a promotion point for our docs that will go into much more detail than this presentation.

Three phases of a refresh upgrade Export phase – exports all Informix databases, and some platform files, for use later on in the RU process (normally in /common on the hard drive, except for 7825-H3 and 7828-H3 platforms). This phase happens at the beginning of the upgrade process, before any reboots take place. Install phase – After the export phase completes, the machine reboots into the other partition, and begins to install the new version of Cisco Unity Connection (including RHEL, etc.). This phase also may contain multiple reboots during its process. Import phase – After everything is installed, the previously-exported data will be imported back to the new install (DB content into Informix, platform files to the proper locations on the hard drive). Each bullet here is meant to summarize the three main phases of the RU from Connection’s Point-of-view.

Notable differences in functionality during upgrade The following applies to standalone RU only: Most services will be offline for the entire duration of the RU All administrative interfaces will be offline All phone access (including voice message access) will be offline Plan on the server being offline for however long an L2 upgrade + switch- version would take on the system previously (as a rough estimate) Emphasize that this slide for now is only relevant to standalone Connection servers, and that for clustered installations (discussed a few slides later), this content changes slightly. On the first bullet here, mention that while we only list the two services here (admin, phone), that there are many other impacted services and functionality on the Connection server, and that if you want a complete list you can refer to our official documentation for more detail. Too much to list here.

7825-H3 and 7828-H3 refresh upgrades RAID driver changing for these platforms in Cisco Unity Connection 8.6 (current HW RAID driver in existing versions is not supported in RHEL 5.5, with Connection 8.6) Part of RU for these platforms is a switch to a new software-based RAID driver that is supported in RHEL 5.5 To switch the RAID driver, the existing HD content must be formatted External USB Drive Requirement: minimum 128GB (during Export Phase, data will be put on this USB drive prior to the machine’s hard drives being formatted to use the new RAID driver). A couple of points to include for the fourth bullet for the external USB hard drive – make sure the USB drive doesn’t contain any data that needs to be preserved because the RU process will format the external USB drive before performing the export, and that you should leave the USB drive plugged into the machine for the entire duration of the RU to ensure it will safely complete.

7825-H3 and 7828-H3 refresh upgrades (cont.) It is very important to make sure DRS backups are taken with these platforms prior to beginning any upgrades Reverting to older version after RU completes is not allowed (once the hard drive is formatted, there is no old version to revert back to anymore) Instead of revert, you can re-install old version, then restore from previous DRS backup In the event of an RU failure on these platforms, the machine may not be able to automatically revert back to the previous version (esp. if the hard drive is already been formatted) Emphasize that DRS backups will be the only backup plan incase of an RU failure for these platforms. The only way to go back will be to re-install the old version on the machine again, then restore the DRS backup. You can also mention that our documentation also tries to raise this caution as well.

Connection cluster refresh upgrades Subscriber must be online and running prior to beginning RU on publisher (otherwise publisher RU will fail) Publisher must be completely upgraded and running prior to beginning RU on subscriber (otherwise subscriber RU will fail) Duration of a cluster RU (both publisher, and subscriber) will be notably longer than previous L2 cluster upgrades (roughly equivalent to two RU back-to-back) Subscriber will be expected to be online and functional for much longer while publisher is upgrading, and vice versa The first two bullets are important to note for clusters, because these points were not entirely true for L2 upgrade conditions.

Cluster refresh upgrade functionality differences While the publisher is being upgraded: All admin interfaces (and other services) on both publisher and subscriber are disabled Phone interface on publisher disabled, still enabled on subscriber Users can still access messages on subscriber through phone, no state changes are preserved! Any phone messages left on subscriber during this process will not be delivered until subscriber RU is started While the subscriber is being upgraded: Publisher is fully-functional, able to take messages and allow users to retrieve messages on all interfaces Publisher will also deliver any messages from the subscriber while the publisher was being upgraded Subscriber will pull down database from publisher, overwriting any local changes that took place while the publisher was upgrading (ex. changes to message read state) When the subscriber finishes its RU, the cluster should be restored, and the publisher will have primary role As with the similar slide earlier for standalone configurations, you can also mention that this is discussed in much more detail in our official documentation. Biggest two points to emphasize here: While the pub is being upgraded, users can still call into the TUI on the subscriber and check messages, but if they change any attributes on the messages (new->saved, new->deleted, saved->deleted, etc.) that those attribute changes will not be preserved Any messages left over the TUI on the sub will stay on the sub, even after the pub has finished its RU. The trigger to get those messages back over to the pub to be delivered, is to start the RU on the sub, not for the pub to finish its RU.

Reverting from Connection 8.6 to earlier versions No voice messages left after upgrading to 8.6 will be migrated back to the previous version Any changes to greetings or voice names after upgrading to 8.6 will also not be migrated back to the previous version As before, any administrative changes (user adds, changes, deletes, etc.) after upgrading to 8.6 are also not migrated back to the previous version Only the first two bullet points are different for reverts from 8.6 to earlier versions, but they are important to mention.

What logs to gather for troubleshooting All logs on this slide will be located in /var/log/install on the machine system-history.log: contains previous version and new version information. Also indicates whether upgrade succeeded or failed. install.log: platform install log file (grab the one with the latest timestamp in the file name). This is the main log documenting the upgrade procedure. cuc-export.log: contains the logged information from the execution of the Connection Export Phase of the RU cuc-install.log: contains the logged information from the execution of the Connection Install Phase of the RU cuc-import.log: contains the logged information from the execution of the Connection Import Phase of the RU

What logs to gather for troubleshooting (cont.) /var/log/active/cuc/informix.log: contains any logging from Informix to capture database-related changes during the upgrade When troubleshooting RU for a clustered install, be sure to gather the log sets from both the publisher and the subscriber Be sure to point out that the informix.log file is not in /var/log/install/ this log file will be in /var/log/active/cuc, separate from all of the install log files. Also note that when dealing with cluster RU configurations, it’s often helpful to have the log sets from both the publisher and the subscriber, often the troubleshooting will have to correlate between what is happening on both machines.

Sample system-history.log 03/21/2011 07:57:06 | root: Cisco Option Install ciscocm.refresh_upgrade_v0.8.cop Start 03/21/2011 07:59:09 | root: Cisco Option Install ciscocm.refresh_upgrade_v0.8.cop Success 03/21/2011 08:26:35 | root: Upgrade (refresh) 8.6.0.98000-54 Start 03/21/2011 08:38:24 | root: Upgrade (refresh) 8.6.0.98000-54 Start-Install 03/21/2011 08:49:37 | root: Boot 8.6.0.98000-54 Start 03/21/2011 09:49:03 | root: Upgrade (refresh) 8.6.0.98000-54 Failure 03/21/2011 09:51:03 | root: Boot 7.1.3.30000-1 Start Highlighted sections (red) indicate: When the RU is started (along with the version # we are trying to upgrade to) Any # of intermediary reboots in between When the RU is finished, including if it was successful or not (above example shows a failure so TAC can see what a failure will look like). If it was successful, the “Failure” word would be replaced with “Success” instead. Also shows that if an RU fails, we will automatically try to go back to the previous version, shown in the last line that we are booting back to the older version #

Sample install.log 03/23/2011 16:34:57 component_install|Execute "/common/refresh_upgrade/Cisco/connection/scripts/cuc-export RU Export 8.6.0.99091-9 8.5.1.10000-26 /common/component/connection /usr/local/cm/ /common/log/install/capture.txt"|<LVL::Debug> 03/23/2011 16:57:11 component_install|(CAPTURE) Launching export|<LVL::Debug> 03/23/2011 16:57:11 component_install|File:/common/download/8.6.0.99091- 9/Cisco/bin/component_install:798, Function: execute_shell_cmd(), /common/refresh_upgrade/Cisco/connection/scripts/cuc-export RU Export 8.6.0.99091-9 8.5.1.10000-26 /common/component/connection /usr/local/cm/ /common/log/install/capture.txt failed (1)|<LVL::Error> 03/23/2011 16:57:11 refresh_upgrade|Exiting with result 1|<LVL::Info> This log shows that the cuc-export script is being launched, and all of the arguments being passed to the cuc-export script. Third line shows that cuc-export failed, and returned a non-zero (failure) error code back to the RU scripts Last line shows that the RU is exiting as a failure. Takeaway from this log is that from here, we can see when our scripts (cuc-export, cuc-import, etc.) are being called, and if we returned success (0) or failure (1), but that this log will NOT indicate the nature of that failure. You’d have to consult the appropriate scripts separate log file to find more detail why those scripts failed specifically.

Sample cuc-export.log 2011-03-30 19:35:48 + echo 'Attempt to contact the subscriber server failed. Skipping restarting subscriber server services.' 2011-03-30 19:35:48 Attempt to contact the subscriber server failed. Skipping restarting subscriber server services. 2011-03-30 19:35:48 + echo 'Restoration of local server (publisher) to secondary server state completed.' 2011-03-30 19:35:48 Restoration of local server (publisher) to secondary server state completed. 2011-03-30 19:35:48 + recover_was_run=0 2011-03-30 19:35:48 + echo 'Recovery procedure from failed CUC data export done.' 2011-03-30 19:35:48 Recovery procedure from failed CUC data export done. 2011-03-30 19:35:48 + main_exit 2011-03-30 19:35:48 + rc=0 2011-03-30 19:35:48 + [[ 0 == 0 ]] 2011-03-30 19:35:48 + '[' 0 -ne 0 ']' 2011-03-30 19:35:48 + '[' 0 == 0 ']' 2011-03-30 19:35:48 + rc=1 2011-03-30 19:35:48 + '[' 1 '!=' 0 ']' 2011-03-30 19:35:48 ++ basename /common/refresh_upgrade/Cisco/connection/scripts/cuc-export 2011-03-30 19:35:48 + echo 'cuc-export failed.' 2011-03-30 19:35:48 cuc-export failed. 2011-03-30 19:35:48 ++ basename /common/refresh_upgrade/Cisco/connection/scripts/cuc-export 2011-03-30 19:35:48 ++ date '+%m/%d/%y %H:%M:%S' 2011-03-30 19:35:48 + echo 'cuc-export: The CUC export script ran at 03/30/11 19:35:48' 2011-03-30 19:35:48 cuc-export: The CUC export script ran at 03/30/11 19:35:48 2011-03-30 19:35:48 + exit 1 This is a sample cuc-export.log file from a publisher attempting to perform an RU, while the subscriber server is offline. Highlighted text shows the failure to contact the subscriber server, logs that the publisher tries to recover to its original state, then exits with failure (exit 1). Also mention that interpreting cuc-import.log will be very similar, even though a sample cuc-import.log file is not provided in this TOI.

Helpful Resources TOI presentations from IPCBU for RU in CUCM: http://wwwin- eng.cisco.com/Eng/VTG/IPCBU/CUCM/CallManager_MontBlanc/Presentations/Mont Blanc-Refresh-Upgrade-IR1-TOI.pptx http://wwwin- eng.cisco.com/Eng/VTG/IPCBU/CUCM/CallManager_MontBlanc/Presentations/Mont Blanc-Refresh-Upgrade-IR2-TOI.pptx Here are some reference links to include that were made by IPCBU engineers detailing RU from a Callmanager point-of-view