1. Setup of NetApp gFiler Gateway Series

2. Installation Steps List of what the customer must provide Physical hardware & cabling setup Issues for clustering or stand alone setup Configuration of storage subsystem – 9970V Setup Console for initial Gateway access Boot Gateway from flash compactdisk Initial boot up of Gateway Set environmental variables Re-boot Gateway to initialize Luns or Lun x/Vol0 Special boot menu (called Floppy boot menu) Configure Network interfaces on Gateway Add Licenses for NFS Install On-Tap OS 6.4.1 from Unix Admin (NFS) (or) Install On-Tap OS 6.4.1 from Windows Admin (CIFS) Configure NTP Configure Time Zone to match network service Confirm installation (some handy commands) <

3. Racking and Cabling Unpack –Save license papers! Rack (3 man job!) Cable up fibre to Array (cable all paths) Power –Both supplies & cluster ps Ethernet –Management port –Gig-e Ports Console port (for initial config & possible recovery)

4. What the customer must provide 2 IP addresses (Access & Management) –Gateway address –Netmasks –DNS servers 3 power feeds (110/220) CIFS (Windows file server) –Domain Name –WINS/DC ip address –Membership in Domain NFS –NIS server? –UNIX admin host IP (trusted host for administration)

5. Cluster Configuration Make sure both gFilers are cabled together and that they are both powered up. Set the environmental variables FIRST on both gFilers first before proceedings to install the Data ONTAP –setenv LUN-ownership low/high –setenv oprah-style true –setenv java? true –setenv jit-enable? true

6. Configure storage If direct connect –FCAL –Fabric Off –No LUN security If through a switch –Fabric On –Use LUN Security & Zoning (Each NetApp HBA should see only one copy of each LUN. See Appendix A for gFiler/Storage configurations) Present consecutive LUNs in order starting at 0 (Will not detect high numbered LUNs without lower ones for no- clustered configuration)

7. Boot Flash Upgrade You may need to upgrade the Boot Flash if you are unable to discover LUNs from an HDS storage subsystem. See Appendix B for detailed instructions on how to upgrade the Boot Flash.

8. Configure NetApp ROM Attach to the NetApp console using the provided serial cable with an application like HyperTerm or SecureCRT –Settings Bits per Second: 9600 Data Bits: 8 Parity: None Stop Bits: 1 Flow Control: No flow control Expect that the first boot will fail and yield an “OK>” prompt

9. Boot messages Intel Open Firmware by FirmWorks Copyright 1995-2003 FirmWorks, Network Appliance. All Rights Reserved. Firmware release 4.2_i2 Press Del to abort boot, Esc to skip POST Memory size is 4096 MB Testing SIO Testing LCD Probing devices Memory size: 4096MB Testing 512MB Complete Finding image... Loading /pc-card:\x86\kernel\primary.krn Starting Press CTRL-C for special boot menu................................................................................................................................................ MTL_MODULE(1): Initializing IB interconnect. NetApp Release 6.4X11: Fri Jan 31 08:02:30 PST 2003 Copyright (c) 1992-2003 Network Appliance, Inc. Starting boot on Sun Feb 16 00:27:37 GMT 2003 Boot Post Messages (Make sure firmware is 4.2.i2 or later) Loading kernel Image from boot flash Special Boot menu (not used for first config) Successful load of flash image

10. Boot messages Found 3 LUNs on device 4a.0L0 Found 3 LUNs on device 4b.0L0 Found 3 LUNs on device 8a.0L0 Found 3 LUNs on device 8b.0L0 DBG: OFW LUN_ownership not set Sun Feb 16 00:27:45 GMT [config.noPartnerLUNs:CRITICAL]: No LUNs were detected or the partner; this node will be unable to takeover correctly PANIC: OFW LUN_ownership not set in process dmgr_thread on release NetApp Releas e 6.4X11 on Sun Feb 16 00:27:45 GMT 2003 version: NetApp Release 6.4X11: Fri Jan 31 08:02:30 PST 2003 cc flags: 3O DUMPCORE: START DUMPCORE: END -- coredump *NOT* written. halt after panic during system initialization Program terminated ok We found 3 LUNS On each path However the NetApp Wants to know who owns These LUNs? Since there are no LUNs with a bootable OS & no ownership is set we cant boot So since we can’t boot a core dump occurs and yields an OK> prompt

11. Configure NetApp Prom Did it see any storage? –Link lights –Messages showing storage discovery “Found X LUNs on device AA.bbb” Set enviornmental variables –OK> setenv oprah-style true –If it is the first cluster (or unclustered) ok> setenv LUN-ownership low –If it is the second member of the cluster: ok> setenv LUN- ownership high –OK> setenv java? true OK> setenv jit-enable? true Reboot –Ok> boot

12. Initializing/zeroing out disks If there is data already on the disks the NetApp will see the drives as unavailable Need to enter special boot menu and zero out disks If the NetApp sees the disks as already zeroed out then the boot menu will skip to the first time configuration questions.

13. Example Using special boot menu to zero out disks Intel Open Firmware by FirmWorks Copyright 1995-2003 FirmWorks, Network Appliance. All Rights Reserved. Firmware release 4.2_i2 Press Del to abort boot, Esc to skip POST Memory size is 4096 MB Testing SIO Testing LCD Probing devices Memory size: 4096MB Testing 512MB Complete Finding image... Loading /pc-card:\x86\kernel\primary.krn Starting Press CTRL-C for special boot menu................................................................................................................................................ Special boot options menu will be available. MTL_MODULE(1): Initializing IB interconnect. Press CTRL-C to go to special boot menu CTRL-C recognized Will boot into menu

14. Example Using special boot menu to zero out disks NetApp Release 6.4X11: Fri Jan 31 08:02:30 PST 2003 Copyright (c) 1992-2003 Network Appliance, Inc. Starting boot on Sun Feb 16 15:51:28 GMT 2003 Found 3 LUNs on device 4a.0L0 LUN Ownership using low range Sun Feb 16 15:52:00 GMT [config.noPartnerLUNs:CRITICAL]: No LUNs were detected for the partner; this node will be unable to takeover correctly This boot is of OS version: NetApp Release 6.4X11. The last time this filer booted, it used OS version:. The WAFL/RAID versions of the previously booted OS are unknown. If you choose a boot option other than Maintenance mode or Initialize disks, the file system of your filer might be upgraded to a new version of the OS. If you do not want to risk having your file system upgraded, choose Maintenance mode or reboot using the correct OS version. (1) Normal boot. (2) Boot without /etc/rc. (3) Change password. (4) Initialize all disks. (5) Maintenance mode boot. We want the NetApp to Initialize the disks that it can see. So we select 4

15. Example Using special boot menu to zero out disks Selection (1-5)? 4 Install a new file system? y This will erase all the data on the disks, are you sure? yes Sun Feb 16 15:52:27 GMT [raid.vol.disk.add.done:notice]: Addition of disk 4a.0L2 (S/N 045150060002) to volume vol0 has completed successfully Sun Feb 16 15:52:28 GMT [config_async_0:notice]: Volume vol0 has been added to the system.

16. Booting the OS The gFiler will boot up and prompt you to answer questions to configure it. If you make a mistake it can be corrected afterwards (options setting ) On the first boot the NetApp will attempt to DHCP e0 (the onboard Ethernet). We will change the IP to the customer specified value.

17. First Boot (Configuring System Parameters) ifconfig e0 mediatype auto Configuring onboard ethernet e0. Contacting DHCP server. Ctrl-C to skip DHCP search... Sun Feb 16 23:15:22 GMT [rc:info]: Contacting DHCP server Sun Feb 16 23:15:23 GMT [10/100-IV/e0:info]: Ethernet e0: Link up. Configuring e0 using IP 10.3.21.238 netmask 255.255.254.0. ifconfig e0 up mediatype auto netmask 255.255.254.0 10.3.21.238 Sun Feb 16 23:15:34 GMT [rc:info]: DHCP config successful NetApp Release 6.4X11: Fri Jan 31 08:02:30 PST 2003 System ID: 0050390937 (); partner ID: () System Serial Number: 1034194 () System Rev: C2 Backplane Part Number: 104-00013 Backplane Rev: C2 Backplane Serial Number: 1034194 slot 0: System Board Processors: 1 Memory Size: 3072 MB slot 0: 10/100 Ethernet Controller IV e0 MAC Address: 00:a0:98:01:12:15 (auto-100tx-fd-up) slot 0: NetApp ATA/IDE Adapter ata0a (1f0) ata0a.0 30MB slot 1: Remote Management Controller slot 4: FC Host Adapter 4a 3 Disks: 90.3GB slot 4: FC Host Adapter 4b 3 Disks: 90.3GB slot 5: Dual 10/100/1000 Ethernet Controller V e5a MAC Address: 00:07:e9:15:08:d0 (auto-unknown-cfg_down) e5b MAC Address: 00:07:e9:15:08:d1 (auto-unknown-cfg_down) slot 6: Interconnect HBA slot 7: NVRAM Memory Size: 256 MB slot 8: FC Host Adapter 8a 3 Disks: 90.3GB slot 8: FC Host Adapter 8b 3 Disks: 90.3GB slot 9: Gigabit Ethernet Controller IV e9 MAC Address: 00:02:b3:c1:e2:63 (auto-unknown-cfg_down)

18. First Boot (Configuring System Parameters) (auto-unknown-cfg_down) Please enter the new hostname []: dexter Do you want to configure virtual network interfaces? [n]: n Please enter the IP address for Network Interface e0 [10.3.21.238]: 10.3.20.134 Please enter the netmask for Network Interface e0 [255.255.254.0]: Should interface e0 take over a partner IP address during failover? [n]: Please enter media type for e0 {100tx-fd, tp-fd, 100tx, tp, auto} [auto]: 100tx- fd Please enter the IP address for Network Interface e5a []: Should interface e5a take over a partner IP address during failover? [n]: Please enter the IP address for Network Interface e5b []: Should interface e5b take over a partner IP address during failover? [n]: Please enter the IP address for Network Interface e9 []: Should interface e9 take over a partner IP address during failover? [n]: Please enter the IP address for Network Interface e10a []: Should interface e10a take over a partner IP address during failover? [n]: Please enter the IP address for Network Interface e10b []: Should interface e10b take over a partner IP address during failover? [n]: Please enter the IP address for Network Interface e11 []: Should interface e11 take over a partner IP address during failover? [n]: Would you like to continue setup through the web interface? [n]: Please enter the name or IP address of the default gateway: 10.3.20.1 The administration host is given root access to the filer's /etc files for system administration. To allow /etc root access to all NFS clients enter RETURN below. Please enter the name or IP address of the administration host: 10.3.20.91 Where is the filer located? []: HDS TRC San Diego Lab (Row 2, Rack0) Do you want to run DNS resolver? [n]: y Please enter DNS domain name [hds.com]: trc.hds.com You may enter up to 3 nameservers Please enter the IP address for first nameserver [10.1.7.34]: 10.3.20.215 Do you want another nameserver? [y]: y Please enter the IP address for alternate nameserver [10.1.7.33]: 140.243.186.22 Do you want another nameserver? [n]: Do you want to run NIS client? [n]:

19. First Boot (Configuring System Parameters) This system will send event messages and weekly reports to Network Appliance Technical Support. To disable this feature, enter "options autosupport.support.enable off" within 24 hours. Enabling Autosupport can significantly speed problem determination and resolution should a problem occur on your system. For further information on Autosupport, please see:. http://now.netapp.com/autosupport/ Press the return key to continue. The remote management controller (RMC) extends autosupport by sending additional system event alerts. Your autosupport settings are used for sending these alerts via email over the RMC LAN interface. Would you like to configure the RMC LAN interface [y]: y The MAC address for the dedicated RMC LAN interface is 00:30:d3:06:29:7e. Would you like to enable DHCP on the RMC LAN interface [y]: n Please enter the IP address for the RMC []: 10.3.20.135 Please enter the netmask for the RMC []: 255.255.254.0 Please enter the IP address for the RMC gateway []: 10.3.20.1 Please enter media type for the RMC {100tx-fd, tp-fd, 100tx, tp, auto} [auto]: 1 00tx-fd The mail host is required by your system to send RMC alerts and local autosupport email. Please enter the name or IP address of the mail host: 10.3.20.91 You may use the autosupport options to configure alert destinations. Now unpack the tar file provided on the tar floppies into the root directory of the filer. When you are done, type "download" to install the boot image on the hard disk and "reboot" to begin using the new image. Proceed to the administration host you specified earlier in the setup

20. First Boot (Configuring System Parameters) Sun Feb 16 23:20:24 GMT [init_java:warning]: Java disabled: Missing /etc/java/r t131.jar. Sun Feb 16 23:20:24 GMT [init_zapi:warning]: Zapi RPC is disabled: java not enabled Sun Feb 16 23:20:24 GMT [httpd_servlet:warning]: Java Virtual Machine not accessible Sun Feb 16 23:20:24 GMT [ems.engine.startReplay:info]: Replay begins for EMS boot-time cache Sun Feb 16 23:20:24 GMT [ems.engine.endReplay:info]: Replay ends for EMS boot-time cache Sun Feb 16 23:20:24 GMT [dyn_dev_qual_admin:info]: Firmware is up-to-date on all disk drives Sun Feb 16 23:20:26 GMT [10/100-IV/e0:info]: Ethernet e0: Link being reconfigured. add net default: gateway 10.3.20.1 There are 5 spare disks; you may want to use the vol command to create new volumes or add disks to the existing volume. Sun Feb 16 23:20:27 GMT [savecore_admin:info]: saving 102M to core.0.nz ("raid: there are no data or parity disks in process rc on release NetApp Release 6.4X11") Sun Feb 16 23:20:27 GMT [mgr.boot.floppy_done:info]: NetApp Release 6.4X11 floppy boot complete. Sun Feb 16 23:20:27 GMT [cf.fm.unexpectedAdapter:warning]: Warning: clustering is not licensed yet an interconnect adapter was found. NVRAM will be divided into two parts until adapter is removed Sun Feb 16 23:20:27 GMT [mgr.boot.reason_ok:notice]: System rebooted. Sun Feb 16 23:20:27 GMT [asup_main:warning]: minimal Autosupports unavailable. Could not read /etc/asup_content.conf Sun Feb 16 23:20:27 GMT [asup_main:warning]: minimal Autosupports unavailable. Could not read /etc/asup_content.conf Sun Feb 16 23:20:28 GMT [sysconfig.sysconfigtab.openFailed:notice]: sysconfig: table of valid configurations (/etc/sysconfigtab) is missing. dexter> Sun Feb 16 23:20:28 GMT [console_login_mgr:info]: root logged in from console Sun Feb 16 23:20:29 GMT [sysconfig.sysconfigtab.openFailed:notice]: sysconfig: table of valid configurations (/etc/sysconfigtab) is missing. Sun Feb 16 23:20:31 GMT [savecore_admin:info]: core.0.nz processing completed. Sun Feb 16 23:20:31 GMT [savecore_admin:info]: Generating /etc/crash/mini-core.0 Sun Feb 16 23:20:31 GMT [10/100-IV/e0:info]: Ethernet e0: Link up. Sun Feb 16 23:20:31 GMT [savecore_admin:info]: (/etc/crash/mini-core.0) generati on completed. Sun Feb 16 23:20:41 GMT [rmc.connect:warning]: The remote management controller in slot 1 is missing its LAN cable. Please connect it as soon as possible. dexter>

21. Add Licenses In order to install the Data ONTAP software you will need to activate either CIFS, NFS or HTTP (depending on the administration host) which requires a license. dexter> license add QAPXI A nfs license has been installed. nfs enabled. dexter> Sun Feb 16 23:21:54 GMT [rc:notice]: nfs licensed dexter> license add HHJNL A http license has been installed. Use "options httpd.enable" to enable http. dexter> Sun Feb 16 23:22:13 GMT [rc:notice]: http licensed

22. Update Data ONTAP software Even if it is already at 6.4 update the filer (no java) Download code Download release notes Download install docs

23. Installing/Updating Data ONTAP software from UNIX Mount filer Copy code to filer root Extract tar file Run install script Unmount filer From filer console run “download” command Reboot filer ADMINHOST /># mount filer:/ /mnt ADMINHOST /># cp 64_sysfiles_I.tar /mnt ADMINHOST /># cd /mnt ADMINHOST /mnt># tar –xvf 64_sysfiles_I.tar ADMINHOST /mnt>#./install_netapp /mnt (completes sucessfully) ADMINHOST /mnt># cd / ADMINHOST /># umount /mnt

24. Installing/Updating Data ONTAP software from Windows 2K After successfully installing the CIFS license, map the C$ drive from the gFiler to a drive letter on the administration host. Load the NetApp Data ONTAP CD that came with the gFiler on the administration host and change directories on the CD to “\ONTAP\6_4_1\X86”. Execute the “setup.exe” command making sure you are extracting the files into the “/etc” directory under \\ \C$.\\ \C$ From the gFiler console run the command “download”. Wait until the “download” command finished then reboot the gFiler. Note: If you are unable or are having problems adding the gFiler into the Windows domain choose “Windows Workgroup authentication using the filer’s user accounts”, option 2 in “cifs setup”. You can rerun “cifs setup” again to add the gFiler to the domain at a later date.

25. Configure NTP Time synchronization is Important! 2 options NTP or rdate. NTP is preferred, but it is important that Windows Doman Controllers, NFS clients, and Filer all be synchronized to a common time source. From filer console: >options set timed.proto ntp >options set timed.servers 10.3.20.1 >options set timed.enable on

26. Set Time Zone Easily set by console (or web) >timezone America/Los_Angeles Or for the rest of the US (Not the whole list) America/Phoenix America/Indiana America/Indianapolis America/Chicago America/Los_Angeles America/Denver America/Detroit America/Mexico_City America/Montreal America/Vancouver America/New_York

27. Configure CIFS deedee*> cifs setup Enable CIFS access to the filer by a Windows(tm) PC Your filer is currently only visible to PCs on the same net. Do you wish to make the system visible via WINS? [no]: This filer is currently configured as a Multiprotocol filer. Do you want to configure this filer as a NTFS-only filer? [no]: The default name of this filer will be 'DEEDEE'. Do you want to modify this name? [no]: CIFS supports three types of user authentication: 1. Windows Domain authentication. 2. Windows Workgroup authentication using the filer's user accounts. 3. /etc/passwd and/or NIS/LDAP based authentication. What type of authentication will this filer use? [1]: 1 The filer will use Windows Domain authentication.

28. Configure CIFS Enter the Windows Domain for the filer []:trc.com TRC.COM is a Windows 2000(tm) domain. In order to create this filer's domain account, you must supply the name and password of an administrator account with sufficient privilege to add the filer to the TRC.COM domain. Please enter the Windows 2000 user [Administrator@TRC.COM]: administrator@trc.co m Password for administrator@trc.com: CIFS - Logged in as administrator@trc.com. CIFS - Updating filer account 'cn=deedee,cn=computers,dc=trc,dc=com' CIFS - Connecting to domain controller. Sun Feb 23 09:53:53 PST [rc:info]: Starting DC address discovery for TRC. Sun Feb 23 09:53:58 PST [rc:info]: Found 2 addresses using DNS site query (Defau lt-First-Site-Name). Sun Feb 23 09:53:58 PST [rc:info]: Found 3 addresses using generic DNS query. Sun Feb 23 09:53:58 PST [rc:info]: DC address discovery for TRC complete. 2 uniq ue addresses found. Sun Feb 23 09:53:58 PST [rc:info]: Connection with \\W2KDC01 established Welcome to the TRC (TRC.COM) Windows 2000(tm) domain.

29. Configure CIFS CIFS local server is running. deedee*> Sun Feb 23 09:5 4:06 PST [nbns_timer:info]: All CIFS name registrations complete for local server

30. Confirm Available storage dexter> df Filesystem kbytes used avail capacity Mounted on /vol/vol0/ 22185864 105868 22079996 0% /vol/vol0/ /vol/vol0/.snapshot 5546464 0 5546464 0% /vol/vol0/.snapshot dexter> storage show Slot: 8a Description: Fibre Channel Host Adapter 8a (Dual-channel, QLogic 2312 rev. 2) Firmware Rev: 3.1.15 PCI Bus Width: 64-bit PCI Clock Speed: 33 MHz FC Node Name: 2:000:00e08b:09911d Cacheline Size: 16 FC Packet Size: 2048 Link Data Rate: 1 Gbit SRAM Parity: Yes External GBIC: No State: Enabled In Use: Yes Redundant: Yes Slot: 8b Description: Fibre Channel Host Adapter 8b (Dual-channel, QLogic 2312 rev. 2) Firmware Rev: 3.1.15 PCI Bus Width: 64-bit PCI Clock Speed: 33 MHz FC Node Name: 2:001:00e08b:29911d Cacheline Size: 16 FC Packet Size: 2048 Link Data Rate: 1 Gbit SRAM Parity: Yes External GBIC: No State: Enabled In Use: Yes Redundant: Yes

31. Confirm Available storage Slot: 4a Description: Fibre Channel Host Adapter 4a (Dual-channel, QLogic 2312 rev. 2) Firmware Rev: 3.1.15 PCI Bus Width: 64-bit PCI Clock Speed: 33 MHz FC Node Name: 2:000:00e08b:097d03 Cacheline Size: 16 FC Packet Size: 2048 Link Data Rate: 1 Gbit SRAM Parity: Yes External GBIC: No State: Enabled In Use: Yes Redundant: Yes Slot: 4b Description: Fibre Channel Host Adapter 4b (Dual-channel, QLogic 2312 rev. 2) Firmware Rev: 3.1.15 PCI Bus Width: 64-bit PCI Clock Speed: 33 MHz FC Node Name: 2:001:00e08b:297d03 Cacheline Size: 16 FC Packet Size: 2048 Link Data Rate: 1 Gbit SRAM Parity: Yes External GBIC: No State: Enabled In Use: Yes Redundant: Yes DISK SHELF BAY SERIAL VENDOR MODEL REV --------------------- --------- ---------------- -------- ---------- ---- 4a.0L0 - - 045150060000 HITACHI OPEN-L 2103 8a.0L1 - - 045150060001 HITACHI OPEN-L 2103 8a.0L2 - - 045150060002 HITACHI OPEN-L 2103 8b.0L0 - - 045150060003 HITACHI OPEN-L 2103 4b.0L1 - - 045150060004 HITACHI OPEN-L 2103 4b.0L2 - - 045150060005 HITACHI OPEN-L 2103

32. Appendix B – gFiler/Storage Connectivity Cluster / Direct-Attach 9570V configuration 0A 0B Filer1 1A 2B Filer2 1A 2B 1A 1B Up to 84/168 LUNs on each port (84 per node) LUN_ownership = highLUN_ownership = low Host Group 00 Port 0A Host Group 00 Port 0B Host Group 00 Port 1A Host Group 00 Port 1B Controller 0 Controller 1 9570V High Low

33. Appendix B – Upgrading the Boot Flash Upgrading the Boot Flash image for NetApp gFilers Gateways 1)Hitachi Data Systems gets kernel code via FTP from Network Appliance: a)ftp ftp.netapp.comftp.netapp.com b)Login:anonymous c)Passwd:[your email address] d)bin (change to binary mode) e)cd frm-ntap/HDS f)get netapp_6.4.1-x86 g)Save it to some easily-remembered directory on your PC 2)Remove Compact Flash card from gFiler 3)Insert the Compact Flash card into a Compact Flash reader/writer. This piece of equipment can be purchased for about $20.00 from a computer supply store like Fry’s, Circuit City or Best Buy.

34. Appendix B (continued) 4)CD to "X86" folder on the Compact Flash card 5)CD to "KERNEL" folder 6)Remove file called "PRIMARY.KRN" 7)Copy "netapp_6.4.1-x86" from wherever you stored it on your PC to the KERNEL folder and rename it "PRIMARY.KRN" 8)Insert the Compact Flash card into gFiler, button it up, and reboot.