© 2005 Mt Xia Technical Consulting Group - All Rights Reserved. HACMP – High Availability Testing and Updates November, 2005

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Introduction Scope HACMP – Testing Scenarios Operating System Updates HACMP Updates Database and Application Updates

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Scope –This presentation covers failover and failback testing in an example HACMP environment. –Also described here are update methods that need to be considered for support and maintenance of the cluster. –This presentation is not designed as a step-by-step procedure for these processes, only as a guideline for development and implementation of update methodologies.

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Updates to the system microcode, operating system, HACMP, applications and databases will require periodic service outages in order to keep the systems up-to-date and current. The time required to perform updates can vary widely depending upon the methodology chosen to perform the updates. The service outages may be classified as follows: –Significant –Moderate –Minimal Service Outages

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Significant service outages to business functions can be experienced by implementing an update methodology as follows: –Stop applications –Stop database –Update operating system –Update system microcode –Update adapter microcode –Update HACMP –Update database –Update Applications –Restart HACMP –Test Failover/Failback Significant Service Outage

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Moderate service outages to business functions can be experienced by implementing an update methodology as follows: –Create alternate clone of rootvg on hot-standby node –Update alternate clone –Reboot hot-standby from alternate clone –Create alternate clone of rootvg on active node –Update alternate clone –Reboot active node from alternate clone –Synchronize HACMP between nodes –Restart HACMP on active node –Restart HACMP on hot-standby node Moderate Service Outage

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Minimal Service Outage Minimal service outages to business functions can be experienced by implementing an update methodology as follows: –Update hot-standby node –Failover to hot-standby node –Update original node –Synchronize HACMP between nodes

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Update Steps Steps to perform during an update/upgrade process: –Update AIX operating system –Update system microcode –Update adapter microcode –Update HACMP –Update database –Update applications –Synchronize HACMP –Test Failover/Failback

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Failover/Failback –Automated –Manual (desired) HACMP – Testing Scenarios

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. smitty hacmp –System Management –Automated Testing Provides automated testing of basic HACMP functions Some automated tests will always fail because of the cluster configuration Results can be difficult to interpret HACMP – Automated Testing

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. Manual Failover/Failback testing –Provides targeted testing of specific environment and/or components –Results are easier to interpret smitty hacmp –System Management –Start HACMP clinfo –Stop HACMP HACMP – Manual Testing

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. HACMP Policies Any changes to the startup environment of a clustered machine requires failover testing Enterprise wide unique identifiers shall be used for all shared resources Enterprise wide unique user name, group name, UID, and GID number will be used Regularly scheduled HACMP verification for each cluster. Regularly scheduled audit of HACMP patch levels

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. HACMP Guidelines Oracle database's will normally be implemented in a non-concurrent access cluster Application servers will normally be implemented in 2-way cascading clusters In virtualized environment, HACMP will be implemented in a 1- way cascading cluster with failover to a standby node built with minimal resources

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. HACMP Standards Machine Naming IP addresses, names, aliases VG, Major Numbers, LV, JFS Log, and Mount Points Cluster Names Resource Group Names Application Server Names Application start and stop script names User names and UID numbers Group names and GID numbers HACMP filesets

HACMP Update © 2005 Mt Xia Technical Consulting Group - All Rights Reserved. HACMP Procedures SAN Storage requests DNS requests Network cabling TSM backups rootvg backups Disaster Recovery Failover Testing LVM Modifications Change Control User Modifications