High Availability Linux (HA Linux) Reliable services: other techniques Other techniques for reliable services: High Availability Linux (HA Linux) Hot Standby machines

Reliable services: other techniques Goal: in case a machine providing a service becomes unavailable, another one takes over transparent for the users Unavailabilities may be caused by: scheduled system maintenance OS upgrade intrusive security fixes preventive hardware interventions network interventions unscheduled system failures hardware failures power failures network (eg. switch) failures

Reliable services: other techniques Distinguish two different service types: stateless services (no local state data) State data is e.g. stored in an external data base no local data, another machine can simply take over load balancing may be the better choice Others services: local data representing the state of the service this data needs to be shared somehow Worst case: state information is kept in memory

How to share state data: Reliable services: other techniques How to share state data: keep up-to-date data on external storage keep a copy of the state data on external storage use an appropriate external data base share this external storage between different machines Keeps static data in an external configuration database

External storage solutions: Reliable services: other techniques External storage solutions: Network Attached Storage (NAS) solution: a separate machine hosts the disk export via Network File System (NFS) introduces a new SPOF (single point of failure) Storage Area Network (SAN) solution: use Fibre Channel (or InfiniBand) based external storage may require use of cluster file systems can be designed in a fully redundant way expensive ! May become complex

Reliable services: Linux-HA What is Linux-HA ? It means High Availability Linux Open source product Works on many OS flavours (gentoo, debian, SuSE, MacOS/X...) About 100 downloads of core component per day Typical use cases: Mail servers Firewalls File servers DNS servers DHCP servers Proxy Caching servers See http://www.linux-ha.org/running

Reliable services: Linux-HA How does Linux-HA work ? Main component: Heartbeat Heartbeat needs one or more media paths Death of node detection via heartbeat Master: runs service Slave: checks if master is alive External storage for state data

Reliable services: Linux-HA How does Linux-HA work ? Simple modes of operation: Active – Passive: The master runs all services The slave only checks if the master is alive The slave does nothing else (no load balancing) No service degradation in case the master goes down now

Reliable services: Linux-HA How does Linux-HA work ? Simple modes of operation: Active – Active: Both master and slave run services In case of a failure the alive nodes takes over all services The service becomes degraded in case of a fail over

Reliable services: Linux-HA Linux - HA remarks Box switching Initiated by heartbeat, if the service goes down Or by administrator, in case of scheduled interventions The user sees only one IP adress, it's completely transparent Monitoring constraints can imply that the boxes need to be close Usually on the same switch (which introduces a SPOF)

Reliable services: Linux-HA Examples of Linux - HA in FIO VOMRS Data is kept in Oracle RAC DDB -> external storage MyProxy Static data in Quattor configuration database Other data synchronized via rsync No cluster file systems being used for sharing data.

Reliable services: Hot Standby Hot standby: built-in failover for some services Example: LSF master In Platform LSF any “server” machine is a possible master candidate. There are no network restrictions and no paths which differ from the standard network connection like in HA linux. Failover is initiated by the application itself. Note: each possible master needs access to the most recent event data files (!)

Reliable services: Hot Standby Hot standby for LSF master: definitions LSF server: any machine which runs LSF daemons LSF master: Box which runs master batch and scheduling daemons There can be only one such machine in each LSF instance LSF master candidate Any LSF server which knows the current LSF state at any time Note: it is possible to name possible master candidate nodes. At CERN we have 2

Reliable services: hot standby

Reliable services: hot standby Remarks about the current LSF installation: True redundancy would require (at least) 3 nodes. Too expensive ... The two master nodes have identical hardware. If the NFS server fails, the redundancy is gone, but the system will still work because the master keeps a local copy of the event data The hot standby machine is alway empty. No loadbalancing. A failover requires up to 10 minutes. The hot standby has to fire up additonal daemons and read in the current status files

Reliable services: summary HA Linux: Fast failure detection Independent paths to check master Automatic fail over Independent of application Machines need to be close together (risk of SPOF) Eventually on the same switch (for cabling and networking) Requires an extra IP address Hot standby mechanism: No restrictions on cabling (no SPOF) Depending on application a DNS alias is useful Needs to be supported by the application, not a general solution No independent failure detection possible Requires a hot standby which is not used by other machines No load balancing possible

Reliable services: Questions ?