1. Cluster Management the IBM Way
Background
ScotGRID project
IBM Management Solutions
Technologies
Boot, Installation, Server, IBM x330 server, KVM, Console Server, Raid,Vlan
Procedures
putting the hardware together
installing RedHat
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

2. ScotGRID http://www.scotgrid.ac.uk/ JREI funded
ScotGRID is a £800k prototype two-site Tier 2 centre in Scotland for the analysis of data primarily from the ATLAS and LHCb experiments at the Large Hadron Collider and from other experiments. The centre currently consists of a 118 CPU Monte Carlo production facility run by the Glasgow PPE group and a 5TB datastore and associated high-performance server run by Edinburgh Parallel Computing Centre.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

3. ScotGRID-Glasgow
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

4. IBM Management Suites
Xcat is set of scripts tables and goodies written to help install a Beowulf Cluster the IBM Way
CSM is alternative suite of software from IBM to install and manage a cluster of PC servers. It seems to have its origin in the RS2000/AIX world
IBM seem to have other management software with titles including words like Tivoli, Director.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

5. Booting - PXE
ftp://download.intel.com//labs/manage/wfm/download/pxespec.pdf
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

6. PXE
PXE is Intel’s Pre-eXecution Environment and includes a dhcp/tftp loader in PROM on an Ethernet Card. It serves much the same role as MOP on Vaxstations and Decstations doing diskless booting.
The Xcat solution uses PXE to load PXELINUX - a network variant of SYSLINUX as used on RedHat CDs
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

7. PXE
PXELINUX can load a kernel either over the network or from the local disk according to a tftp-ed configuration file.
IBM suggest a boot order - floppy,CD,net,disk
In Xcat, network loading is mainly used to initiate kickstart installation and the local disk load is used for normal operation.
(Re)installation is just a matter of changing the configuration file.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

8. Automated Linux installation
kickstart
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

9. kickstart RedHat’s scheme to automate installation
installation parameters in file as
file on bootnet floppy
nfs accessible file named with ip number
http …..
possibility of scripts to customise installation after the RedHat code has finished
Xcat calculates kickstart files for compute, head and storage nodes from local configuration tables and locally modified templates.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

10. Servers Redundancy - fileservers have dual power supplies, RAID
error checking - ECC
reduce chances of undetected data corruption
CDF had an incident recently that proved to be undetected corruption of data in one system
mountable in 19 inch rack
Blades (like PCs as CAMAC Modules) are now supported by xcat
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

11. IBM x330 Server X330 server is 1U high
( 1U = 1¾ inch and a full rack is 42U )
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

12. (Remote) Support Processors
Each system has a PowerPC support Processor to power on/off main processor, do monitoring,...
Support processors are daisy chained with RS485 to a card with serial and Ethernet connections
Remote access via Ethernet and serial with snmp, telnet, http and proprietary protocols
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

13. KVM Switch
in a box - ScotGRID has an 8 port KVM switch from Apex (
integrated C2T - the IBM x330 compute nodes do not have conventional Keyboard,Video and Mouse connections - they have part of a KVM switch onboard and a pair of connectors to daisy-chain a rack’s worth of servers into a distributed KVM switch
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

14. KVM Switch box On screen control by pressing “PrintScreen”
>1 port for real K, V and M
Cascadeable in organised manner
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

15. KVM Switch integrated
An adapter cable connects keyboard, video and mouse to first system and short interconnecting cables daisy-chains the rest of the rack’s worth of servers into a distributed KVM switch.
Switching is effected by keyboard shortcuts or a button on the front of the server
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

16. Console Servers conserver package http://www.conserver.com
Terminal Servers in reverse
IBM Supplied 4 Equinox ELS16
accept incoming telnet
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

17. Serial Line access Terminal Server lines connect to COM1
talks to support processor when server unbooted
is /dev/ttyS0 when linux is running
Conserver establishes telnet to all COM1 ports and multiplexes access to multiple linux clients - guess this is important to avoid finding access blocked.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

18. Raid
remote management of the IBM ServeRAID controllers using RaidMan software
configuration
monitoring
the remote agent component seems to provoke crashes of kernel
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

19. Switches
IBM supplied 3 x 48 port Cisco Catalyst Switches and 1 x 8 port Gigabit switch to interconnect the file servers, other switches and the Campus Backbone
The switches divide the ports between an Internet accessible VLAN and a private VLAN for the compute nodes and management units
The management interface is via the serial port, telnet and http
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

20. Cisco Catalyst 3500XL Switch
Sockets on right take GBICs
ScotGRID-Glasgow has 3 of 48 port switches and a single unit with 8 GBIC sockets
Separate VLANs for private and Internet.
( GBIC = GigaBit Interface Converter  Cisco Gigabit Transceiver to various fibres and 1000baseT)
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

21. ScotGRID-Glasgow Masternode and Fileserver nodes on left
Compute Nodes in middle and right with headnodes low down
Daisy chained KVM and Support Proc network
Cascaded starwired Ethernet and serial lines
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

22. Putting it together Masternode Storage Nodes Head Nodes
Campus Backbone
Internet VLAN
VLAN
100 Mbps
1000 Mbps
Compute Nodes
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

23. Compute Nodes COM1 Keyboard/Video/Mouse 2 Pentium III Serial eth0
Console in
Console out
eth0
ethernet
Keyboard/Video/Mouse
2 Pentium III
Support Processor
COM1
Remote Support Adapter
eth1
Serial
RS485 in
RS485 out
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

24. Software Configure VLANs
Install RedHat and xcat manually on masternode
Tabulate topology
Update BIOS
Configure Terminal Servers
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

25. VLANs Cisco boxes use a serial port to start configuration
Use CISCO flavoured RJ45 <-> 9 Pin D type adapter
cu - l /dev/ttyS0 -s 9600
set ip address; enable telnet,...
ongoing management via telnet/tftp/http
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

26. RedHat Install everything off CD + errata
Fail to spot unconfigured mailman rpm - eventually exhausted inodes with 95,671 empty error logs
/etc/syslog.conf
# 1/10/02 Trick learned from IBM - Ctl Alt F12 displays /dev/tty12 *.info;mail.none;news.none;authpriv.none /dev/tty12
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

27. Tables
Cisco3500.tab storage1 cisco1,1 storage2 cisco1,2 storage3 cisco1,3 node01 cisco2,1 node02 cisco2,2 …………………
Tables describe the connections, ip networking parameters, selectable options
# file of Global Settings for ScotGRID cluster 21/5/02 DJM #
# r-series commands pathnames - actually ssh equivalents
rsh /usr/bin/ssh
rcp /usr/bin/scp
# Absolute location of the SSH Global Known Hosts file that contains the ssh pub
lic keys for all the nodes
gkhfile /usr/local/xcat/etc/gkh
# Directory used by the tftp daemon to retrieve 2nd stage boot loaders
………………………….
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

28. BIOS Boot floppy and use CD in every Box Vers 27/11/2018 23:11
HEP SYSMAN 26th Nov 2002

29. Terminal Servers Use serial port to start configuration
Use Equinox flavoured RJ45 <-> 9 Pin D type adapter
cu -l /dev/ttyS0 -s 9600
Setup script over /dev/ttyS0
Ongoing management via telnet
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

30. Harvest MAC Addresses
cause compute nodes to remotely boot kernel and application to send packets out ethernet ports and emit MAC addresses on COM1
collect MAC Addresses from serial lines or via Cisco management interface
using topology tables and MAC addresses to calculate dhcpd.conf
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

31. Configure Remote Support Adapters
The RSA Ethernet <-> RS485 boxes can live in their own box or in a spare PCI slot - as in ScotGRID-Glasgow
boot utility in hosting server and set ip address
xcat script uses tables to locate and configure the RSA over ethernet
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

32. Compute Node Installation
calculate kickstart file
reboot compute node
press button
xcat command : rpower noderange boot
last stage of kickstart post-installation script resets booting to local disk
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

33. Batch OpenPBS http://www.openpbs.org
MAUI scheduler
plugin scheduler for PBS/WIKI/Loadleveler/Sun’s Grid Engine
able to organise parallel jobs using >1 cpu, perhaps slight overkill for us
lots of control - per user, per group, per job length, target fair share, time waiting,……...
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

34. Ongoing Management Snmp psh noderange command xcat rcommands
mainly from support processors via syslog to warning s
psh noderange command
parallel ssh
party trick : psh compute eject
xcat rcommands
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002

35. rvitals Vers 27/11/2018 23:11 HEP SYSMAN 26th Nov 2002
martin]# rvitals node08 all
node08: CPU 1 Temperature = 21.0 C (69.8 F) node08: CPU 2 Temperature = 21.0 C (69.8 F) node08: hard shutdown: 85.0 C (185.0 F) node08: soft shutdown: 80.0 C (176.0 F) node08: warning: 75.0 C (167.0 F) node08: warning reset: 63.0 C (145.4 F) node08: DASD 1 Temperature not available. node08: Ambient Temperature = 16.0 C (60.8 F) node08: System Board 5V: 5.06 node08: System Board 3V: 3.29 node08: System Board 12V: node08: System Board 2.5V: 2.63 node08: VRM1: 1.78 node08: VRM2: 1.78 node08: Fan 1: 70% node08: Fan 2: 72% node08: Fan 3: 78% node08: Fan 4: 78% node08: Fan 5: 76% node08: Fan 6: 75% node08: Power is on. node08: System uptime = 3556 node08: The number of system restarts = 139 node08: System State: Currently booting the OS, or no transition was reported.
Vers 27/11/ :11
HEP SYSMAN 26th Nov 2002