1. Unix Administration Guntis Barzdins

2. SYS ADMIN TASKS Linux System Administration Setting the Run Level System Services User Management Network Settings Scheduling Jobs Quota Management Backup and Restore Adding and Removing software/packages Setting a Printer Monitoring the system (general, logs) Monitoring any specific services running. Eg. DNS, DHCP, Web, NIS, NPT, Proxy etc.

3. Process Manipulation  Once you run a program (e.g. vi, myprog,...), that program will suspend the terminal you called it in (the terminal will not be receiving input from you). You can start the program in the background to avoid this: myprog & You can suspend a program that is running and send it to background, if you already started it: Ctrl-z (to suspend) bg (sends the suspended program to the background)  ps (show running processes)  top (monitor running processes)  kill (kill processes)  & (send process to background)  bg (send process to background)  fg (get process from background)  Ctrl+c (terminate process)  Ctrl+z (suspend process)

4. Intrusion Detection System (IDS) Open Source Tripwire – is a file integrity- checking program for UNIX/Linux operating systems Host-based Software that alerts you when important files change Tripwire keeps a hash value for each designated file When a file is altered/deleted, tripwire will have a new hash value that is different than the original Replaced by more advanced HIDS: OSSEC, Samhain, AIDE Client/Server mode etc.

5. Tripwire tutorial in a slide  Initial setup download / build / install it generate policy file # twadmin –create-polfile /etc/tripwire/twpol.txt modify policy file (e.g. remove unnecessary files) # vi /etc/tripwire/twpol.txt build initial database # tripwire –init  check periodically # tripwire –check reconcile differences (e.g. software installation) # tripwire –update –accept-all –twrfile report_file

6. LINUX Firewall Linux Security

7. SELinux Linux Security Originally created by NSA to meet US DoD MAC Malicious or broken software can have root-level access to the entire system by running as a root process. SELinux (Security Enhanced Linux) provides enhanced security. Through SELinux policies, a process can be granted just the permissions it needs to be functional, thus reducing the risk SELINUX can take one of three values enforcing - SELinux security policy is enforced. permissive - SELinux prints warnings instead of enforcing. disabled - SELinux is fully disabled.

8. SELinux Configuration Linux Security

9. AppArmor Less complex and less secure Popular in user oriented distributions (Ubuntu, SUSE), enabled for some potentially vulnerable services by default Bundle software packages with AppArmor profiles Can create profile file by launching application in learning mode, can make secure enough profile if application not already compromised Capabilities: FS open/read/write different modes, networking (all/tcp/udp), executability etc.

10. Log files  On linux, you can go to /var/log  Depends on the application  Information shown in log files depend on the debug level you defined

11. Configuring Disk Quotas Linux System Administration To implement disk quotas, use the following steps: Enable quotas per file system by modifying /etc/fstab Remount the file system(s) Create the quota files and generate the disk usage table Assign quotas

12. Configuring Disk Quotas Linux System Administration Enabling Quotas: Edit fstab to enable usrquota LABEL=/1 / ext3 defaults 1 1 LABEL=/boot /boot ext3 defaults 1 2 LABEL=/users /users ext3 exec,dev,suid,rw,usrquota 1 2 LABEL=/var /var ext3 defaults 1 2 LABEL=SWAP-sda5 swap swap defaults 0 0

13. Configuring Disk Quotas Linux System Administration Remounting the File Systems: Issue the umount command followed by the mount command to remount the file system in which quota has been implemented (umount /users;mount /users) Creating the Quota Database Files: Use quotacheck command to create quota.user file quotacheck -cu /users Assigning Quotas per User: assigning the disk quotas with the edquota command (edquota ) Disk quotas for user web_cc (uid 524): Filesystem blocks soft hard inodes soft hard /dev/sdb1 988612 1024000 1075200 7862 0 0

14. Linux Filesystem Management Linux Commands df Shows the disk free space on one or more filesystems. (df -k, df -H) du Shows how much disk space a directory and all its files contain. (du, du –sk, du – sh ) Find out which users use most space etc. $ du /home -d 1 | sort fsck Filesystem check. Must not be run on a mounted file system. (fsck ) badblocks Used to search a disk or partition for badblocks. (badblocks )

15. Linux Filesystem Management Linux Commands sync Synchronize data on disk with memory. `sync' writes (suggests to write) any data buffered in memory out to disk. mount Used to mount a filesystem. Complement is umount. (mount, mount –a) umount Unmounts a filesystem. Complement is mount. (umount )

16. Native UNIX Backup Utilities  UNIX Systems include 3 core utilities that allow you to backup files to tape or disk. tar (very simple to use) cpio (a bit more complex) dump (most complex of the three)

17. Using the tar Utility for Backup  tar usage: tar [x|c]vf [tape device] [files/directory]  where: x = extract from a tape c = compress onto tape j = use bzip compression z = use gzip compression (just like when we tar and untar regular.tar files)

18. Other UNIX Backup Utilities  cpio – has the ability to detect I/O errors during backup that tar cannot detect  Has the ability to do things like specify wildcard patterns during restore  dump – very fast, detects I/O errors, allows you to perform incremental backups  Used to backup filesytems  Operates below filesystem abstractions – on blocks

19. TARCPIODUMP Simplicity of Invocation Very simple (tar c )Needs find to specify file namesSimple. few options Recover from I/O errors?None. Write your own utilityResync option on HP-UX will cause some data lossAutomatically skips over bad sections Backup special filesLater RevisionsYes Multi-volume backupLater RevisionsYes Backup across network?Using rsh only Yes Append files to backupYes, (tar –r)No Multiple Independent Backups on Single TapeYes Ease of listing files on the volumeDifficult, Must search entire backup ( tar –t ) Difficult, Must search entire backup ( cpio –it ) Simple, Index at front ( restore –t ) Ease and speed of finding a particular fileDifficult, No wildcards, Must search entire volumeModerate, Wildcards, Must search entire volumeInteractive. Very easy with commands like cd, ls Incremental backup NoMust use find to locate new/modified filesIncremental of whole filesystem only, Mult. Levels List files as they are being backed uptar cvf 2>logfilecpio –v 2>logfileOnly after backup with restore –t >logfile (Dump can show % complete, though.) Backup based on other criteriaNoFind can use multiple criteriaNo Restore absolute path names to relative locationOnly by using chrootLimited with cpio -IAlways relative to current working directory Interactive decision on restoreYes or No possible with tar –wCan specify new path or name on each fileSpecify individual files in interactive mode CompatibilityMultiple platformMultiple platform with ASCII header, not always portable Readable between some platforms, but cannot be relied on Primary usefulness Individual user backup, transfer files between filesystems System backup, transfer files between filesystemsSystem backup Volume efficiencyMedium, usually limited to 10k block sizeMedium, usually only 5K block size, but can specify larger size on some OSs High, can usually specify up to maximum block size of device Wildcards on restoreNoYesOnly in interactive mode Simplicity of selecting files for backup from numerous directories Low, must specify each independent directory, subdirectories included Medium, find optionsNone, will backup one and only one filesystem Specifying directory on restore get files in that directory YesNo, must use "path/*"Yes Stop reading tape after a restored file is foundNo Will stop reading tape as soon as last file is found Track deleted filesNo If you restore with –r, files deleted before last incremental dump will be deleted. Filesystem efficiency BetterWorst (files get a stat from both find and cpio)Best Limit on path length (Tests done with Solaris native utils 7/99.) 155 characters. Complains "prefix is greater than 155 characters." Gtar has slight workaround. 255 characters. Doesn’t complain. Just truncates pathname to 255 char’s. 1056 characters. Likelihood that file exists in TOC but not in archiveLow Medium (since TOC is made first)

20. rsync  Over network and filesystem  Secure through SSH Both ends require rsync executable, no services or daemons required  Incremental backup  Delta encoding Only changed parts of files transmitted  Example rsync -avz root@192.168.1.2:/home /backups/server1  Many options

21. Linux Services Linux System Administration There are 113 daemons, Out of them, the following are most widely used: apmd : Power Management autofs : Automount services crond : Periodic Command Scheduler cups : Common Unix Printing System dhcpd : The DHCP server dovecot : IMAP (Internet Message Access Protocol) and POP3 (Post Office Protocol) server gpm : Mouse httpd : Apache Web server

22. Linux Services Linux System Administration iptables : Kernel based Packet Filtering firewall kudzu: Finds new Hardware mysqld : MySQL server named : BIND server network : Networking nfs : Network File Share nfslock : NFS file locking ntpd : NTP (Network Time Protocol) server portmap : RPC (Remote Procedure Call) support postgresql : The Postgresql Database Engine

23. Linux Services Linux System Administration sendmail : Sendmail Mail Server smb : Samba Network Services snmpd : Simple Network Management Protocol squid : Squid Proxy Server sshd : Open SSH and SFTP server syslog : System Logging xinetd : Provides support for telnet, ftp, talk, tftp etc. ypbind : NIS Server

24. Automating Unix Administration  You don’t want to spend the whole day making sure that all servers/workstations and their services are fine  Use monitoring tools that can alert you for any problem in the network mon, nagios, cacti, angel Zabbix – Latvian product  Create scripts to check the status of servers/services and use cron to run it periodically Mail the result to admin

25. Example script #!/bin/sh machine="sunfire" down= i=0 while [ $i -le 15 ] do sun=$machine"$i" /usr/sbin/ping $sun > /dev/null if [ $? -ne 0 ] then down="$down:$sun" fi i=`echo "$i+1" | bc -l` done if [ -n "$down" ] then echo $down | tr : '\012' | /usr/ucb/mail -s "DOWN machines" admin@ccse.kfupm.edu.sa fi exit 0

26. Lost Root Passwd  If you have LILO installed, type LILO: linux init 1 Booting into single-user mode Change the root passwd, reboot again  If you have installed GRUB Type ‘e’ to go to edit mode, add init 1 argument at the end  Boot with LiveCD (default Ubuntu etc.) Mount the root disk chroot into mounted disk passwd Reboot and remove CD

27. Network File System (NFS) Originally developed by Sun in 1984 A distributed file system protocol Uses a network protocol instead of block level access Builds on the Open Network Computing Remote Procedure Call system (ONC RPC) Originally developed by Sun as part of the NFS project Allows files to be accessed using the same interfaces and semantics as local files: mounting/unmounting, listing directories, read/write at byte boundaries, system's native permission model etc. First widely used IP-based network file system Other notable network file systems are Andrew File System (AFS), Apple Filing Protocol (AFP), and Server Message Block (SMB; also known as Common Internet File System, CIFS) Transparent mobility of files, e.g. user machines mount home directory from one spacey central server

28. ONC RPC A request–response protocol An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied parameters The remote server sends a response to the client, and the application continues its process Client waits until the server has finished processing before resuming execution (unless the client sends an asynchronous request to the server) The programmer writes essentially the same code whether the subroutine is local or remote First popular implementation of RPC on Unix Serializes data (External Data Representation) so that it can be transferred between different OS and transport layers Access to RPC services is provided via a port mapper that listens on a well-known port (111) over UDP and TCP

29. NFS Architecture  VFS layer hides differences between OS’s It doesn’t matter what OS the client or server implements, UNIX or Windows. As long as the file systems are compliant with the file system model offered by NFS.  Operations on VFS are either passed to local FS or to NFS Client, which handles files at the remote server.  All client-server communication is done through RPCs, with client and server stubs. Implemented with either UDP or TCP.

30. NFS Architecture

31. NFS (Network File System) RPC requestActionIdempotent GETATTRGet file attributeYES SETATTRSet file attributeYES LOOKUPFile name searchYES ACCESSCheck accessYES READLINKRead from symbolic linkYES READRead fileYES WRITEWrite to the fileYES COMMITFix server cache data to the diskYES CREATECreate fileNO REMOVERemove fileNO RENAMERename fileNO

32. NFS (Network File System) RPC requestActionIdempotent LINKCreate hard linkNO SYMLINKCreate symbolic linkNO MKNODCreate special nodeNO MKDIRCrate directoryNO RMDIRRemove directoryNO READDIRRead directoryYES READDIRPLUSExtended directory readYES FSSTATGet FS dynamic attributeYES FSINFOGet FS static attributeYES PATHCONFGet POSIX informationYES VFS interface allows for a modular implementation, reflected in a simple protocol (initially). NFS translates VFS requests into RPCs to server – instead of translating them into disk accesses.

33. NFS Versions Version 1 was used only for in-house purposes at Sun Version 2 (1989) operated only over UDP: stateless server side, with locking implemented outside of the core protocol Version 3 (1995) added support for 64-bit file sizes and offsets, to handle files larger than 2 gigabytes (GB); support for asynchronous writes on the server, to improve write performance; etc. Using TCP as a transport made using NFS over a WAN more feasible Version 4 (2000), influenced by AFS and CIFS, includes performance improvements, mandates stronger security, and introduces a stateful protocol

34. Stateless vs. Stateful

35. Stateless Client side caching for speed Problem with caching: global consistency NFS 3: let the client deal with consistency Client pings back to check state of file: Local cache is current  continue Local cache is old  invalidate Simplicity but no consistency guaranty! Locking implemented in the user space by rpc.lockd Write-through: write is done synchronously both to the cache and to the backing store. Fault tolerance: easy recovery – not much loss because of write-through

36. Stateful Local file systems have state NFS 4 maintains a state of all open files Open/Close calls give the server information: read/write mode, number of clients, versions of files Version number: during open, refresh local cache only if current version is old Server can issue calls to clients for sake of consistency Guaranteed consistency through call-backs and version checking Eliminates useless write-through: unless write shared, no write-through Challenge: recovery from crash/disconnect

37. Leases for cache synchronization FreeBSD extension to NFS3 (NQNFS) There are 3 types of leases: Non-cache lease – define that all file system operations should be taken synchronously with server Read cache lease – let client cache data, do not allow to change the file Write cache lease – let client cache write operations for lease time. If client cache write data, then this data will not be written to the server synchronously. When lease time coming to the end, client will try to get another lease, but if it’s not possible, then data have to be written to the server. Adopted for use in NFS4

38. Read cache lease Client A Client BServer Read req. + lease Answer Read sys. call Read sys. Call (from cache) Read req. (cache miss) Answer Lease timeout Read sys. call Lease expired Read lease req. Answer with same ctime ctime the same - cache valid Read sys. Call (from cache) Lease timeout Read req. (cache miss) Answer Read cache lease for client A Read req. + leaseRead sys. call Lease timeout Answer Client B added to lease Read req. (cache miss) Answer Time

39. Write cache lease Server Client B Write system call Write cached lease for client B Answer (write cache lease) Write system call (cached leaved records) Write cached lease req. before previous lease expired. Get record lease Lease update Answer (write cache lease) Lease timeout System call Lease expiredrecord answer record answer Lease expiration Stopped for a moment because of records Write_slack seconds After last records Time

40. Non-cache lease Client AServer Client B Time Read sys. call req. Read req. + lease Read cache lease for A client answerRead req. (from cache) Read req. (miss cache) answer Lease timeout Read sys. call req. Lease request Answer (non-cache lease) Read sys. call req. (non-cache lease mode) Read req. Read data Lease expired Write sys. call req. Get write cache lease Answer (non-cache lease) Write sys. call (async write cached) Cleanup req. Write cached data to server record answer record Release msg. Write sys. call req.Get write cache lease record answer Synchronous Writes wihout cache

41. Starting up NFS (3)  There are three key things you need to start on Linux to make NFS work. /usr/sbin/rpc.portmap /usr/sbin/rpc.mountd /usr/sbin/rpc.nfsd  These things should start up automatically at boot time. The file that makes this happen is "/etc/rc.d/rc.inet2" rpcinfo -p localhost program vers proto port 100000 2 tcp 111 portmapper 100000 2 udp 111 portmapper 100005 1 udp 679 mountd 100005 1 tcp 681 mountd 100003 2 udp 2049 nfs 100003 2 tcp 2049 nfs

42. 42 Exporting File System  To make parts of your file system accessible over the network to other systems The /etc/exports file must be set up to define which of the local directories will be available to remote users and how each is used # sample /etc/exports file /home/yourname 192.168.12.1(rw) /master(rw) trusty(rw,no_root_squash) /projects proj*.local.domain(rw) /usr *.local.domain(ro) @trusted(rw) /home/joe pc001(rw,all_squash,anonuid=150,anongid=100) /pub (ro,insecure,all_squash) /pub/private (noaccess) stop and restart the server # etc/rc.d/init.d/nfs stop # etc/rc.s/init.d/nfs start

43. /etc/exports  Contains information about the directory paths and partitions that are sharable and hosts they can be shared with. i.e. “Any host from.rutgers.edu can access the /home/documents directory on my server”  Entry format: /dir/to/export client1(permissions) client2 (permissions) Sample entry: /tmp iti.rutgers.edu(rw) 185.14.237.4(ro)  Need to run exportfs to inform NFS server process about changes in /etc/exports: > /usr/sbin/exportfs –a (exports all entries)

44. The NFS Server  Started though rc script: /etc/rc.d/init.d/nfs Must be started after: /etc/rc.d/init.d/portmap  Uses these RPC daemons in /usr/sbin : rpc.nfsd – main component of NFS system rcp.mountd – handles mount requests rpc.quotad – allows for quota enforcement via NFS. All of which are started in the nfs rc script when the system starts  /etc/exports – the main server configuration file  Above utilities are part of knfsd package.rpm package on Linux.

45. The NFS Client  Requires knfsd-clients.rpm package on Linux.  Necessary services started from: /etc/rc.d/init.d/nfslock  RPC daemons in /sbin handle file locking between client and server: rpc.locked rpc.statd All are started from the nfslock rc script automatically  Allows clients to mount remote file systems either using the mount command or by placing an entry in the /etc/fstab file.

46. Local and remote file systems accessible on an NFS client mount –t nfs Server1:/export/people /usr/students mount –t nfs Server2:/nfs/users /usr/staff

47. SMB  SMB is Microsoft’s protocol to share files and printers Also renamed CIFS (Common Internet File System) Client/Server, no location transparency Not the same as Samba : an open source implementation of SMB primarily found on UNIX systems (Linux) SMB usually runs on NetBIOS (naming + sessions + datagram)  NetBIOS + SMB developed for LAN use  A number of other services run on top of SMB In particular MS-RPC, a modified variant of DCE-RPC Authentication for SMB handled by the NT Domains suite of protocols, running on top of MS-RPC To know more: Timothy D Evans, NetBIOS, NetBEUI, NBF, NBT, NBIPX, SMB, CIFS Networking TCP/IP NetBIOS SMB MS-RPC NT-Domain

48. SMB Protocol  Request/response.  Runs atop TCP/IP.  E.g., file and print operations. Open, close, read, write, delete, etc. Queuing/dequeing files in printer spool.

49. Samba Services  File sharing.  Printer sharing.  Client authentication.  Unix server, Windows clients

50. FUSE (Filesystem in Userspace) Lets non-privileged users create their own file systems without editing kernel code.

51. FUSE  Allows to implement anything with file write and read operations and provide it as file system  Encryption – EncFS, TrueCrypt, etc.  Network protocols – SSH, FTP, SFTP, etc.  Cloud storage – Dropbox and every other kind  RAM disk

52. Network Booting  No need for hard disk (or hard disk with Linux) on every host  Boot server and boot client  High level work flow The system boots up, may be with floppy (could be with hard disk also) Sends dhcp request for IP number, gets one Mounts the root file system over NFS

53. Requirements for Network Booting  Setup a LAN infrastructure  Need to setup nfs server  Need to setup dhcp server  Build a kernel image for network booting

54. Setup a LAN infrastructure Ethernet Cable Hub Ethernet Cable Your m/c to be booted Your host, NFS server and DHCP server should be on same LAN NFS server

55. Setup nfs server Edit /etc/exports file before starting the nfs server. / 10.114.7.115(rw,no_root_squash) This will export all files with root r/w to host 10.114.7.115 Save your exports file and from the prompt execute exportfs command Start the nfs server (nfs daemon) E.g. /etc/rc.d/inid.d/nfs start

56. Setup dhcp server  Add in your /etc/dhcpd.conf before starting the dhcp server.  Set the correct MAC address in /etc/dhcpd.conf as follows: subnet netmask 255.255.255.0 { } subnet 10.10.10.0 netmask 255.255.255.0 { host master { hardware Ethernet ; fixed-address ; option root-path ”; }  Save your /etc/dhcpd.conf file  start the dhcpd dameon by “/etc/rc.d/init.d/dhcpd start” command

57. Build a kernel image for network booting  Linux Kernel compilation steps: Assumptions: machine x86 (i386); boot loader lilo. Get plain vanilla kernel from www.kernel.orgwww.kernel.org Explode it into a directory (better if can do it in /usr/src/) => tar -zxvf linux-2.x.xx.tar.gz Optional: create a symbolic link ln -s linux-2.x.xx linux cd to linux directory cd /usr/src/linux or cd /usr/src/linux-2.x.xx Select the components support by make menuconfig or make xconfig - save the configuration Select IP:BOOTP support from Networking options In File system -> Network File System -> Select  NFS File system support and  Root file system on NFS Do Make dep bzImage Make modules modules_install  Copy the /usr/src/linux/arch/i386/boot/bzImage to /boot  Do mkbootdisk with new kernel as argument