2. Hardening Linux for Enterprise Applications Peter Knaggs & Xiaoping Li Oracle Corporation Sunil Mahale Network Appliance Session id: 40274

3. Agenda  Hardening Linux - Using NIC Failover for HA  Understanding network bonding driver  System Requirements & Configuration  Test Procedure & Observation  Status & Statistics information  Summary  Q & A

4. Hardening Linux – Using NIC Failover for HA  Redundant data paths to networked storage  Ability to tolerate failures of NICs  Active/Active Load balancing or failover  Achieving HA in Oracle environments with NAS

5. Understanding network bonding driver  Linux bonding driver to accomplish NIC failover  Included in 2.4 kernel  Bonds multiple network interfaces  Configured as a loadable kernel module  Understanding functionality of NIC failover in Oracle

6. System Configuration  Hardware – Linux Systems  2 * Intel White Boxes with 4 CPU and 3GB RAM  3 * Intel Pro1000 Gigabit Ethernet NICs per system – Storage  3 * Network Appliance F880 filers  Total of 18 Disk Shelves with 3TB usable storage  Total of 5 * Gigabit Ethernet NICs – Switch  Cisco 6509 Gigabit Ethernet Switch

7. System Requirements  Software – Linux Systems  Red Hat Advanced Server 2.1, kernel 2.4.9, e.12  Intel Pro1000 Ethernet driver (e1000_4412k1)  Oracle 9i Release 2 database – Storage  NetApp Filer F880 running Data ONTAP 6.4.1

8. NIC Fail over environment 8 SERVER Gigabit Ethernet switch DATA1DATA2 LOG1 Redo Log i/o Path Data File i/o Paths bond0 NetApp Filers

9. Setup & Configuration  Servers – Setup the server with Red Hat Advanced Server 2.1, kernel 2.4.9, e.12 – Use the e1000_4412k1 module for the Intel GiGE NICs – Configure the GiGE NICs in a private network – Ensure the GiGE NICs are connected to the Cisco switch

10. Setup & Configuration  Servers (cont…) – Bonding Driver/module  Check if the bonding driver is loaded ( lsmod )  Check to see if there is module to load ( bonding.o )  Load the bonding module into the kernel ( modprobe )

11. Setup & Configuration  Servers (cont…) – Configure two GiGE network interfaces as eth3 and eth4 – Use the e1000_4412k1 module for eth3 and eth4  Bring down all the interfaces using the e1000 module  Unload the default e1000 module ( rmmod e1000 )  Load the new e1000 module ( modprobe e1000_4412k1 )  Bring up all the network interfaces

12. Setup & Configuration  Servers (cont…) – Configuring the bond0 virtual interface  Add the alias for bond0 interface to /etc/modules.conf alias bond0 bonding  Create the configuration file for bond0 interface /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 IPADDR=10.1.3.101 NETMASK=255.255.255.0 NETWORK=10.1.3.0 BROADCAST=10.1.3.255 BOOTPROTO=none ONBOOT=yes GATEWAY=130.35.148.1 USERCTL=no

13. Setup & Configuration  Servers (cont…) – Bring down the eth3 and eth4 interface to be used for bond0 – Unmount any file systems or volumes currently mounted by eth3 and eth4 – Delete the configuration files for eth3 and eth4  Remove the ifcfg-eth3 and ifcfg-eth4 from /etc/sysconfig/network-scripts

14. Setup & Configuration  Servers (cont…) – Create the bond0 virtual interface # modprobe bonding; # ifconfig bond0 netmask 255.255.255.0 broadcast 10.1.3.255; # ifconfig bond0 10.1.3.101; # ifenslave bond0 eth3; # ifenslave bond0 eth4; # ifenslave bond0 up; – Check to see if bond0, eth3 and eth4 have the same MAC address

15. Setup & Configuration  Storage – Configure the 3 NetApp filers  2 Filers are used for storing Oracle datafiles, 1 for Oracle log files, (DATA1, DATA2 and LOG1)  DATA1 and DATA2 each have 2 GiGE NICs configured  Filer LOG1 has 1 GiGE NIC configured  Filer DATA1 and DATA2 each have 4 logical volumes  Filer LOG1 has 1 logical volume  All the GiGE NICs are connected to the Cisco switch

16. Setup & Configuration  Switch – Enable channel trunking or port trunking  Interface eth3 and eth4 from the server are connected to 2 ports of the switch  Create a port channel for these ports Console> (enable) set port channel 4/1-2 on Where: eth3 & eth4 are connected to port 4/1-2  Enable portfast for the ports (spantree portfast)

17. Test Procedure & Observation  Non Database Tests  Oracle Database Tests

18. Test Procedure & Observation  Non Database Tests – Copy of large file over the bond0 interface to the NetApp filer – Simulate NIC failure  Down the eth3 interface of bond0 ifconfig eth3 down  Bring up eth3 interface ifconfig eth3 up  Pulling out network cables on the enslaved interface, eth3 – Observations  IO load was distributed over the eth3 and eth4 of bond0  I/O load switched to the remaining interface, eth4

19. Test Procedure & Observation  Database Tests – Create a very large database  Create a large Oracle 9i OLTP database (1TB) on Filers  Run the OLTP workload with 55 users, around 6500 tpmC  The workload was run for about 30min  Simulated NIC failure by pulling network cable – Observation  Average load on the bond0 interface was about 10MB/s  The network traffic on eth3 and eth4 were evenly spread  The effect of simulated NIC failure on thruput was < 10%

20. Test Procedure & Observation  Testing with new bonding driver – The new bonding driver at HP’s website http://h18007.www1.hp.com/support/files/networking/nics – Has been running in Oracle data centers with good stability – Download the RPMs, build and install the driver – Remove the default module and load the new one

21. Test Procedure & Observation  Testing with active/passive mode with new bonding driver – Load the new module with “mode=1” modprobe bonding mode=1 – The I/O load will be only on first slave NIC – The other slave will act as a backup – When the active slave fails, the backup will take over – You must have “portfast” enabled on the switch for the ports

22. Status & Statistics information  Advantages of the new bonding driver – Clear status information in the proc file system  cat /proc/net/bond0/info Bonding Mode: active-backup Currently Active Slave: eth3 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth4 MII Status: up Link Failure Count: 7 Slave Interface: eth3 MII Status: up Link Failure Count: 8

23. Status & Statistics information  Advantages of the new bonding driver – Clear status information from the dmesg log file # modprobe bonding miimon=100; # dmesg bonding.c:v1.0.1-2HP bond0 registered with MII link monitoring set to 100 ms, in bonding mode. bond0 registered without ARP monitoring

24. Status & Statistics information  Advantages of the new bonding driver – Clear status information from the sar report – I/O load on the bond interface bond0 is consistent with its slaves – In load balancing mode, I/O activity shown on bond0 is sum of its slaves

25. Status & Statistics information  sar activity report 11:07:33 AM IFACE rxpck/s txpck/s rxbyt/s 11:07:36 AM eth3 5935.88 2853.82 8454566.78 11:07:36 AM eth4 4564.45 2835.22 6491304.32 11:07:36 AM bond0 10500.33 5689.04 14945871.10

26. Status & Statistics information  Advantages of the new bonding driver – Clear status information in the rpm database rpm -qil bonding – Useful man pages

27. Summary  The bonding driver can be used for NIC failover  Provides redundant data paths for networked storage  The default bonding driver only supports load balancing  The new driver, supports Active/Passive or load balancing  The effect of simulated NIC failures on thruput was < 10%  Achieve HA in Oracle environment with NAS

28. A Q & Q U E S T I O N S A N S W E R S