1. www.ischool.drexel.edu INFO 320 Server Technology I Week 9 Unix/Linux tools 1INFO 320 week 9

2. www.ischool.drexel.edu Overview This wraps up the course with a handful of other tools and technologies of interest –Linux Utilities –RAID 2INFO 320 week 9

3. www.ischool.drexel.edu Linux Utilities 3INFO 320 week 9 From notes by Dr. Randy M. Kaplan, and (Petersen, 2009)

4. www.ischool.drexel.edu Linux Utilities We’ll look at a few more Unix applications of common interest for server administration –uniq, diff –lpr, cups –which, tar –ntp, cron –emacs 4INFO 320 week 9

5. www.ischool.drexel.edu uniq uniq removes duplicate lines in a fileuniq uniq filename Example –uniq lastnames –Any duplicate names in the file named lastnames will be eliminated –Make sure to sort the file first! 5INFO 320 week 9

6. www.ischool.drexel.edu diff diff compares the contents of one file with another filediff –diff filename1 filename2 Example –diff colorsa colorsb –Compares the contents of colorsa with the contents of colorb 6INFO 320 week 9

7. www.ischool.drexel.edu diff The diff utility bears some further examination Let’s assume that colorsa contains – –Red –Blue –Green –Yellow Let’s assume that colorsb contains – –Red –Blue –Green 7INFO 320 week 9

8. www.ischool.drexel.edu diff The diff utility operates under the assumption the first file is to be converted into the second file diff output is a series of lines that define how the first file can be modified so that its content matches the second file –The output for colorsa and colorsb would be 4d3 which specifies that the 4 th line of the colorsa file should be deleted in order that the colorsa file match colorsb 8INFO 320 week 9

9. www.ischool.drexel.edu lpr lpr prints a filelpr lpr filename Example –lpr areport –lpr –PofficePrinter areport (prints the file areport on the printer named officePrinter ) 9INFO 320 week 9

10. www.ischool.drexel.edu cups The Common Unix Printing System (CUPS) is a GNU print server applicationCUPS Cups is managed using a web-based tool at http://localhost:631http://localhost:631 –There you can see print jobs, manage printers, etc. –But first you need to define the printers to be managed 10INFO 320 week 9

11. www.ischool.drexel.edu cups It config file is /etc/cups/printers.conf /etc/cups/printers.conf –That contains the device file for each printer and configuration entries for it –Devices can be identified by parallel name (lp0, lp1, …), serial name (tty0, tty1), or URI Each device gets a print spool directory under /var/spool/cups/devicename The cups daemon is cupsdcupsd 11INFO 320 week 9

12. www.ischool.drexel.edu which which shows the location of the file related to the command (like whereis)whichwhereis which command –Example which info /usr/local/bin/info Useful when a command is not doing what is expected 12INFO 320 week 9

13. www.ischool.drexel.edu tar tar is short for ‘tape archive’tar –Original purpose was to create and read magnetic tape archives –Today it’s used to extract a directory with subdirectories and files beneath it tar files are also frequently compressed –The extension of a tar file that has been compressed will be.tar.gz or.tgz 13INFO 320 week 9

14. www.ischool.drexel.edu tar tar syntax is –tar options filename Example –tar –xvzf myTarfile.tgz –Decompress and restore (extract) all files in the archive named myTarfile.tgz 14INFO 320 week 9

15. www.ischool.drexel.edu ntp The Network Time Protocol (ntp) coordinates time settings across your network It and its documentation (ntp-doc) are installed with apt-get or aptitude like most packages –Docs are likely under /usr/share/doc/ntp-doc/html/index.html 15INFO 320 week 9

16. www.ischool.drexel.edu ntp The ntp config file is /etc/ntp.conf Start ntp with –sudo /etc/init.d/ntp start Check status of ntp with –ntpq –p Or use an external ntp server with the ntpdate command (add to startup or cron) –sudo ntpdate ntp.ubuntu.com 16INFO 320 week 9

17. www.ischool.drexel.edu cron Addressed in more detail in connection with backups, cron allows you to schedule scripts to execute cron cron is started automatically from /etc/init.d on entering multi-user runlevels –crontab files can be in its spool area (/var/spool/cron/crontabs) and more important (/etc/crontab)  this is where you define cron jobs! 17INFO 320 week 9

18. www.ischool.drexel.edu cron cron logs its action to the syslog facility ’cron’, and logging may be controlled using the standard syslogd facilitysyslogd The heart of setting a service to run, is defining a crontab entry for it –You can specify when a file runs by minute, hour, day of month, month, or day of week –Each of those five parameters correspond to numbers or *s at the start of each entry 18INFO 320 week 9

19. www.ischool.drexel.edu cron # use /bin/sh to run commands, no matter what /etc/passwd says/bin/sh /etc/passwd SHELL=/bin/sh/bin/sh # mail any output to ‘bruce@example.com’, no matter whose crontab this isbruce@example.com MAILTO=bruce@example.combruce@example.com # run five minutes after midnight, every day 5 0 * * * $HOME/bin/daily.job >> $HOME/tmp/out 2>&1 19INFO 320 week 9

20. www.ischool.drexel.edu cron # # run at 2:15pm on the first of every month -- output mailed to bruce (above) 15 14 1 * * $HOME/bin/monthly 23 0-23/2 * * * echo "run 23 minutes after midn, 2am, 4am..., everyday" 5 4 * * sun echo "run at 5 after 4 every sunday" 20INFO 320 week 9

21. www.ischool.drexel.edu Editors Before there were word processors there were programs called editors Like a word processor, an editor is used to create text files and manipulate them The original use for a text editor was to create program files and modify them –A venerable text editor, designed for Lisp programming, is Emacs 21INFO 320 week 9

22. www.ischool.drexel.edu Emacs Not developed in the UNIX environment –Therefore – not a special purpose utility More than a text editor –Built-in programming language, LispLisp –Designed to do everything Emacs is a language sensitive editor with specific features for editing programming languages 22INFO 320 week 9

23. www.ischool.drexel.edu The big pattern! Notice the pattern for most Unix applications outside of the kernel –App names are all lower case, e.g. appname –Most apps can be installed with apt-get or aptitude, e.g. sudo apt-get install appname –Every app has a man page; man appname –Most apps have a text configuration file, often something like /etc/appname.conf 23INFO 320 week 9

24. www.ischool.drexel.edu The big pattern! –Many apps can be started with sudo /etc/init.d/appname start –If the app needs a spool directory, it’s under /var/spool/appname –The daemon for an app is usually appnamed –If there’s separate documentation for the app, it’s probably under /usr/share/doc/appname See? UNIX is easy! 24INFO 320 week 9

25. www.ischool.drexel.edu RAID 25INFO 320 week 9 Mostly from (Bytepile, 2009) and (Rankin, 2009) Above image from www.fordcortina.net/pix/raid.jpg

26. www.ischool.drexel.edu RAID RAID (was Redundant Array of Inexpensive Disks, now often Independent) was developed circa 1988 by Patterson, Gibson & Katz; Berkeley CA –A RAID array is a collection of drives which collectively act as a single storage system, which can tolerate the failure of a drive without losing data, and which can operate independently of each other. 26INFO 320 week 9

27. www.ischool.drexel.edu RAID RAID is defined by various levels a set of disks (a RAID array) is designed to achieve – RAID 0, RAID 5, etc. They differ mainly in terms of –Cost –Speed –Redundancy, and efficiency to do so –How many disks are needed at a minimum 27INFO 320 week 9

28. www.ischool.drexel.edu RAID RAID can be implemented in software alone, or with dedicated hardware (e.g. cards or built into a motherboard) –Hardware-based RAID is faster and more expensive RAID typically requires all drives in an array to be identical (same make and model, not just same size and capacity) 28INFO 320 week 9

29. www.ischool.drexel.edu RAID 0 RAID 0 is known as striping (watch spelling, it’s not stripping!) It offers no redundancy, but improves performance (speed) by reading alternately between disks –No speed improvement for writing data! Two or more drives can do RAID 0 29INFO 320 week 9

30. www.ischool.drexel.edu RAID 1 RAID 1 is called mirroring It does nothing for performance, but provides redundancy by maintaining a duplicate copy of data –Hence there is 100% duplication of data, the least efficient possible RAID –Using at least two disks in matched pairs, it simply makes two copies of everything on separate disks 30INFO 320 week 9

31. www.ischool.drexel.edu RAID 0+1 So if we do both mirroring and striping at once, we have RAID 0+1 –RAID 0+1 is implemented as a mirrored array whose segments are RAID 0 (striped) arrays Hence we need at least four disks, and get the read speed boost of striping, and the redundancy of mirroring 31INFO 320 week 9

32. www.ischool.drexel.edu RAID 3 RAID 3 does striping, plus adds a parity drive; 3 drive minimum –The last drive has a parity bit from all of the other drives, used to reconstruct one drive if it fails Very high read and write speeds, high efficiency, good for multimedia data, but expensive and complex 32INFO 320 week 9

33. www.ischool.drexel.edu RAID 5 RAID 5 is similar to RAID 3, but the parity information is distributed across all the drives; 3 drive minimum –Hence if any one drive fails, it can be recreated from the other drives High read rate and efficiency, medium write rate –Very commonly used for dB, web, file, and other servers 33INFO 320 week 9

34. www.ischool.drexel.edu RAID 10 Notice that RAID 10 is not the same as RAID 0+1 RAID 10 is a striped array whose segments are RAID 1 arrays (mirrors) instead of single drives –Minimum of 4 disks needed –Good when you need higher performance than RAID 1; but very expensive 34INFO 320 week 9

35. www.ischool.drexel.edu RAID 10 RAID 10 can handle multiple drive failures, as long as you don’t lose both halves of a mirrored pair 35INFO 320 week 9

36. www.ischool.drexel.edu RAID 50 RAID 50 or 05 is striping two sets of RAID 5 arrays –Minimum of six disks! –Better redundancy, faster reads 36INFO 320 week 9

37. www.ischool.drexel.edu RAID and Ubuntu A RAID 5 array can not be used as the boot partition in Ubuntu – GRUB can’t read it –A boot partition can be RAID 1 RAID is configured during the partitioning process of installation –Each partition is formatted to type ‘K raid’, and they are joined into an array as a multidisk (MD) device, then partitioned to /, /boot, swap 37INFO 320 week 9

38. www.ischool.drexel.edu RAID and Ubuntu The main tools for administering RAID in Linux are /proc/mdstat and mdadm mdadm The former is a file with your current arrays –cat /proc/mdstat Linux recognizes RAID 0, 1, 4, 5, 6 and 10 –This is software-only RAID 38INFO 320 week 9

39. www.ischool.drexel.edu RAID Capacity The usable capacity of common RAID levels is –RAID 0 = # Drives x Cap Drive –RAID 1 = (# Drives / 2) x Cap Drive –RAID 3 or 5 = (# Drives -1) x Cap Drive RAID 3 and 5 ‘lose’ one drive’s capacity for parity information from the other drives, so larger arrays are more efficient 39INFO 320 week 9 From http://www.kintronics.com/raidwpaper.htmhttp://www.kintronics.com/raidwpaper.htm

40. www.ischool.drexel.edu Other RAIDs There are other RAID approaches (RAID 2, 4, 6, 7, 53, 1E, etc.) but combinations of RAID 0, 1, and 5 are most common Other storage technologies include –JBOD –DAS vs. NAS –SAN 40INFO 320 week 9

41. www.ischool.drexel.edu JBOD JBOD is Just a Bunch Of Disks –RAID systems require identical drives –JBOD allows combining physical drives into large logical volumes without requiring identical drives –Cheap, easy, but no redundancy 41INFO 320 week 9

42. www.ischool.drexel.edu DAS vs. NAS DAS is direct-attached storage, a fancy name for disks which are part of a server NAS (network-attached storage) is 100% dedicated to serving files over a network –It’s one or more hard drives, with enough computer attached to let it communicate directly over the network –Simple and cheap 42INFO 320 week 9

43. www.ischool.drexel.edu SAN A Storage Area Network is a dedicated, high performance storage network –Transfers data among servers and storage devices, separate from the local area network –Used when high speed large transfers are common, often using fiber optic connections –Often does load balancing and has high reliability –Often used for transaction processing 43INFO 320 week 9

44. www.ischool.drexel.edu SAN 44INFO 320 week 9 From http://www.rad-direct.com/Product-Whitepaper-IP-SAN-3.htmhttp://www.rad-direct.com/Product-Whitepaper-IP-SAN-3.htm

45. www.ischool.drexel.edu Hot-swappable Finally, storage systems can have hot- swappable drives or not –Hot-swappable means they can be safely removed from the system while it’s running Most RAID systems are hot-swappable, so that failed drives can be removed without reducing availability 45INFO 320 week 9

46. www.ischool.drexel.edu References Bytepile, 2009. “RAID Types - Classifications ” from http://bytepile.com/raid_class.php http://bytepile.com/raid_class.php The RAID Advisory Board (RAB) does not seem to exist any longer 46INFO 320 week 9