Failover and High Availability Stefan Zier Sr. Software Engineer, Server Team May 2006 © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Agenda Introduction High availability set-up explained Software Systems IP addresses Heartbeat networks Installation overview © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Introduction © 2006 ArcSight Confidential

High Availability vs. Disaster Recovery? High availability: Cut down on unscheduled down time Hardware failures Operating system failures Application errors Disaster Recovery: Stay operational during catastrophes Natural disasters Major infrastructure issues © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Technical Approaches High availability Second system connected through multiple communication paths Shared storage between the two systems Software that monitors both systems and moves services between them as needed Quick, automatic recovery Disaster recovery Mostly single points of failure in communication paths Replicated storage between the two sites Often manual switch between two sites (may be sufficient and more cost effective) © 2006 ArcSight Confidential

Removing Single Points of Failure Not as simple as it first seems: Many details make implementation complex System only as good as weakest link in the chain Single point of failure is a component whose failure brings down the entire system Examples Single switch that connects all the NICs Single power circuit powering systems… File server used to host system files © 2006 ArcSight Confidential

Current HA/DR Support in ArcSight High availability configurations supported for ArcSight Manager ArcSight Database ArcSight SmartConnectors Disaster recovery configurations deployed at customers Oracle DataGuard: Can only be used without partition archiving at the moment; Oracle is working on the issue SAN block level replication: Requires more bandwidth between sites, but less complex © 2006 ArcSight Confidential

Connector Redirection Connectors can redirect events to another manager Offer access to real time events to cover shorter periods of downtime Development manager set-ups (second manager to develop content before moving to production) Not a full failover solution Resources replication scripts provide incomplete replication Managers will get out of sync (Active Lists, Assets, Rule Engine state, Event IDs, M1s, etc.) Requires duplicate databases © 2006 ArcSight Confidential

Technologies that Don’t Fit ArcSight Profile Load balancing – would compromise correlation capabilities, since they require all events to go through one system Hot-Hot standby – would consume substantial network and CPU resources to keep rich manager state in sync Seamless Console Failover – would cause a similar delay during failover and add some CPU/network cost on the manager © 2006 ArcSight Confidential

© 2006 ArcSight Confidential HA Setup Explained © 2006 ArcSight Confidential

Failover Architecture Public Network Manager 1 Manager 2 Heartbeat Networks Shared Storage Shared Database © 2006 ArcSight Confidential

Failover Management Software (FMS) Software that manages services in a failover cluster Starts and stops Manager/Connector/Oracle Monitors whether Manager/Connector/Oracle is running Migrates a service IP or virtual IP between systems Needs to run on both hosts Tested products EMC AutoStart (Legato AAM) Veritas Cluster Server (VCS) © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Software Public Network Manager 1 Manager 2 Heartbeat Networks Shared Storage Shared Database © 2006 ArcSight Confidential

Why Didn’t ArcSight Invent a Proprietary FMS? ArcSight wants to provide best-of-breed solution Existing FMS are very mature and well-tested Many OS platform combinations supported FMS need to solve many low-level problems that are not ArcSight’s core competency © 2006 ArcSight Confidential

Why Do We Need Identical Systems? Any difference between the two systems presents a risk Failover may fail, leaving the component nonfunctional Standby system not powerful enough to handle load Different OS/patch version on the standby system could prevent component from starting up It is easier to keep systems updated if they are identical Not recommended to use standby system for double duty ! Good practice: When you restart the manager for a configuration change, use the FMS and bring it up on the standby node © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Shared Storage Shared volume to host $ARCSIGHT_HOME or $ORACLE_HOME Needed so all files (rules checkpoints, archived reports, etc.) are reliably in sync between components Needs to be HA (use RAID, multiple I/O channels) Shared SCSI bus cannot be used Simple NFS server or similar would be single point of failure SAN used for shared storage © 2006 ArcSight Confidential

IP Address Transparency Typical set-up has three IP addresses One IP address for each system (system IP) One IP address for the Manager/Connector/Database, also called virtual IP or service IP Clients always talk to service IP: Point DNS and/or hosts files there! Some FMS use IP-based networks for heartbeat networks, these may require additional IPs © 2006 ArcSight Confidential

IP Address Transparency Console or Connector arcsight.customer.com  192.168.10.89 Public Network 192.168.10.89 Manager 1 192.168.10.90 192.168.10.98 Manager 2 Heartbeat Networks Shared Storage Shared Database © 2006 ArcSight Confidential

IP Address Transparency Console or Connector arcsight.customer.com  192.168.10.89 Public Network 192.168.10.89 Manager 1 192.168.10.90 192.168.10.98 Manager 2 Heartbeat Networks Shared Storage Shared Database © 2006 ArcSight Confidential

Multiple Communication Paths Multiple discrete communication channels between systems Use a mix of technologies (serial, disk, ethernet) At least two channels Needed to avoid split brain syndrome © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Split Brain Syndrome Public Network Manager 1 Manager 2 Shared Storage Shared Database © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Split Brain Syndrome Public Network Manager 1 Manager 2 Shared Storage Shared Database © 2006 ArcSight Confidential

What happened to the other server? What happened to the other server? Split Brain Syndrome What happened to the other server? What happened to the other server? Public Network Manager 1 Manager 2 Shared Storage Shared Database © 2006 ArcSight Confidential

Probably broke. Let’s start up. Split Brain Syndrome Let’s just keep going. Probably broke. Let’s start up. Public Network Manager 1 Manager 2 Shared Storage Shared Database © 2006 ArcSight Confidential

Looks pretty inconsistent… Split Brain Syndrome Public Network Manager 1 Manager 2 Looks pretty inconsistent… Shared Storage Shared Database © 2006 ArcSight Confidential

Installation Overview © 2006 ArcSight Confidential

Set-Up Procedure for ArcSight Manager From one system, install Manager on shared disk In managersetup, install the startup scripts, but disable them so that Manager doesn’t auto-start on during boot (option #3) Set the Cluster ID to host name in managersetup On the other host, run managersetup from shared disk, also install startup scripts Note that startup scripts do not reside on shared disk, but on local disk on each system Test Manager on both systems © 2006 ArcSight Confidential

© 2006 ArcSight Confidential FMS Set-Up Install FMS Set-up IP addresses, heartbeat networks Set-up Manager startup, shutdown and monitoring (scripts under utilities/failover) Set-up service IP/virtual IP Test whether the manager can be started/stopped/migrated Group the manager process and the service IP so they move between hosts together © 2006 ArcSight Confidential

© 2006 ArcSight Confidential FMS Options EMC AutoStart (fka. Legato AAM) Communication over IP Unix, Windows Veritas Cluster Server Communication over proprietary protocol UNIX, Windows © 2006 ArcSight Confidential

© 2006 ArcSight Confidential ARCSIGHT_CID ARCSIGHT_CID is the ArcSight Cluster ID Set in /etc/arcsight/arcsight.conf Manually set in .profile for convenience Must be different on each of the hosts Usually, the host name is used Affects where log files get written: logs/$ARCSIGHT_CID/ (default value is default) © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Scripts Startup and shutdown scripts Use /etc/init.d/arcsight_manager start|stop arcsight_manager needs to be executed as root Monitor Use $ARCSIGHT_HOME/bin/arcsight managerup arcsight managerup needs to run as arcsight user If needed, alter the monitor script to pipe the output of managerup into a file so you can monitor output © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Summary ArcSight now supports HA for Connectors The same HA solution can be used for Manager and database EMC AutoStart and Veritas Cluster Server are supported and tested Tech Notes are available on configuration for both products © 2006 ArcSight Confidential

© 2006 ArcSight Confidential Questions and Answers Download Slides https://support.arcsight.com More ArcSight Events http://www.arcsight.com Join the User Forum https://forum.arcsight.com © 2006 ArcSight Confidential

© 2006 ArcSight Confidential