1. Introduction to the Linux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARS October, 2014

2. Roadmap Linux overview The command shell Navigating the filesystem Basic commands & wildcarding Shell redirection & pipelining Editing text files PermissionsProcesses 10/14cja 20142

3. Poll How many here to learn the command line For HPC, specifically Flux, work? For other computing cluster work? For desktops/other Linux/Unix/Mac OS X? Other? 10/14cja 20143

4. Course Text William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No Starch Press, January 2012. Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcomma nd/TLCL/13.07/TLCL-13.07.pdf. http://downloads.sourceforge.net/project/linuxcomma nd/TLCL/13.07/TLCL-13.07.pdfhttp://downloads.sourceforge.net/project/linuxcomma nd/TLCL/13.07/TLCL-13.07.pdf 10/14cja 20144

5. Linux Overview 10/14cja 20145

6. A compute node 12 Intel cores 48 GB RAM Local disk Network 10/14cja 20146

7. 10/14 http://linuxforums.org.uk/index.php?action=media;sa=item;in=262 cja 20147

8. The command shell 10/14cja 20148

9. The command line A basic way of interacting with a Linux system Execute commands Create files and directories Edit file content Access the web Copy files to and from other hosts Run HPC jobs … do things you can’t do from the conventional point- and-click Graphical User Interface (GUI) 10/14cja 20149

10. Why command line? 1.Linux was designed for the command line 2.You can create new Linux commands using the command line, without programming 3.Many systems provide only the command line, or poorly support a GUI interface Such as most HPC systems Such as most HPC systems 4.Many things can be accomplished only through the command line Much systems administration & troubleshooting Much systems administration & troubleshooting 5.You want to be cool 10/14cja 201410

11. The command shell The command shell is an application that reads command lines from the keyboard and passes them to the Linux operating system to be executed When you login to a remote Linux system using a tool like ssh, you will automatically be connected to a shell On your desktop, laptop, or tablet, you may have to find and execute a terminal emulator application to bring up a shell in a window 10/14cja 201411

12. The command shell 10/14 http://www.livefirelabs.com/208_global/images/unix-operating-system.gif cja 201412

13. Types of shells Standard Linux shell (used in this course) bash“Bourne-again” shell Other shells shBourne shell, original AT&T Unix shell csh“C shell”, common on older Unix systems tcsh“Enhanced C shell”, better version of csh What shell am I using? echo $SHELL 10/14cja 201413

14. Connecting via ssh Terminal emulators Linux and Mac OS X Start Terminal Use ssh command Windows U-M PuTTY/WinSCP (U-M Blue Disc) https://www.itcs.umich.edu/bluedisc/ https://www.itcs.umich.edu/bluedisc/ PuTTY http://www.chiark.greenend.org.uk/~sgtatham/putty/ http://www.chiark.greenend.org.uk/~sgtatham/putty/ SSH Secure Shell (deprecated) 10/14cja 201414

15. Lab 0 Task: Start a local shell on Mac OS X 1.Login to Mac OS X with your uniqname and Kerberos password 2.If there is a Terminal icon in the Dock, double-click it Otherwise: a.Bring up a Finder window: Click on the desktop, type Command-N b.Start the Terminal Application: In the Finder window, click on Applications on the left, scroll down on the right until you find the Utilities folder, double-click on the Utilities folder, scroll down on the right until you find the Terminal application, double-click it 3.This creates a Terminal window with a shell running inside it 4.From a Terminal window, Command-N will start a new Terminal 10/14cja 201415

16. Lab 0 10/14cja 201416

17. The shell prompt The “~$ “ is the shell prompt This means the shell is waiting for you to type something Format can vary, usually ends with “$”, “%” or “#” If $ or %, you have a normal shell This shell has your privileges If #, you have a so-called “root shell” This shell has administrator privileges You can do a great deal of irreversible damage You can do a great deal of irreversible damage 10/14cja 201417

18. Typing into the shell Basic input line editing commands Backspace erases previous character Left and right arrow move insertion point on the line Control-U erases the line so you can start over Enter executes the line you typed Control-C interrupts whatever command you started and returns you to the shell prompt (usually) Up and down arrow will access your command history Type “ exit ” without the quotes to exit the shell Click the red "close" icon at the top of the Terminal window to close it 10/14cja 201418

19. Lab 1 Task: Enter some basic commands ~$ date ~$ id ~$ ps ~$ df -kh ~$ who ~$ top# type Control-C to exit 10/14cja 201419

20. Interlude: Flux 10/14cja 201420

21. Flux Flux is a university-wide shared computational discovery / high-performance computing service. Interdisciplinary Provided by Advanced Research Computing at U-M (ARC) Operated by CAEN HPC Hardware procurement, software licensing, billing support by U-M ITS Used across campus Collaborative since 2010 Advanced Research Computing at U-M (ARC) College of Engineering’s IT Group (CAEN) Information and Technology Services Medical School College of Literature, Science, and the Arts School of Information 10/14 http://arc.research.umich.edu/resources-services/flux/ cja 201421

22. The Flux cluster … 10/14cja 201422

23. A Flux node 12 Intel cores 48 GB RAM Local disk Network 10/14cja 201423

24. Logging in to Flux ssh flux-login.engin.umich.edu MToken (or Software Token) required You will be randomly connected a Flux login node Currently flux-login1 or flux-login2 Firewalls restrict access to flux-login. To connect successfully, either Physically connect your ssh client platform to the U-M campus wired or MWireless network, or Use VPN software on your client platform, or Use ssh to login to an ITS login node (login.itd.umich.edu), and ssh to flux-login from there 10/14cja 201424

25. A shell on a login node Now you have a shell on the login node Your shell prompt has changed to something like [flux-login1 ~]$ Try those simple commands here! Flux is for HPC work The login nodes are reserved for copying data, editing and compiling programs and scripts, and short test runs For HPC computation, you should run on the compute nodes A batch job will always run on the compute nodes An interactive batch job will get you a shell on a compute node 10/14cja 201425

26. A shell on a compute node Getting a shell on a compute node Enter this command on the login node: [flux-login1 ~] $ ~cja/hpc100/getshell After a short delay, you’ll get an interactive shell: qsub: waiting for job 11147004.nyx.engin.umich.edu to start qsub: job 11147004.nyx.engin.umich.edu ready [nyx5509 ~]$ You can type commands into this shell for all of our in-class experiments Type “exit” to exit this shell After two hours, your interactive shell will be killed 10/14cja 201426

27. Navigating the filesystem 10/14cja 201427

28. Linux Filesystem Concepts Files are stored in a directory Directories may contain other directories as well as files A hierarchy of these directories is called a directory tree A directory tree (a connected graph with no cycles) has a single topmost root directory A directory tree rooted at the system root directory “ / ” is called a filesystem 10/14cja 201428

29. 10/14 http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html cja 201429

30. Linux Filesystem Concepts A file is accessed using its path name Absolute path name /dir1/dir2/…/dirn/filename/usr/X11R6/bin Relative path name current-working-directory/filenamebin Every shell maintains a notion of a current working directory Initialized at login to your home directory Changed via cd command Two special directories. refers to the current directory.. refers to the current directory’s parent directory 10/14cja 201430

31. Basic commands 10/14cja 201431

32. Prerequisites Some fundamental commands: ~$ file file# what kind of file is file? ~$ cat file# display contents of text file ~$ less file# paginate text file ~$ man command# get info about command Exercise: figure out how to make the date command display the date in Coordinated Universal Time (UTC) 10/14cja 201432

33. Lab 2 Task: navigate the file system Commands: ~$ cd # make your home directory the current working directory ~$ pwd# print working directory ~$ mkdir foo# create directory foo ~$ cd foo# cd to the foo directory ~$ mkdir bar# create directory bar ~$ cd..# cd back to the foo directory ~$ tree# display foo’s directory tree 10/14cja 201433

34. Navigating the filesystem Some fundamental commands: ~$ pwd# print working directory ~$ cd dir# make dir the current working directory ~$ cd # cd to your home dir ~$ cd ~cja# cd to cja’s home dir ~$ mkdir dir# create directory dir ~$ rmdir dir# remove (empty) directory dir ~$ tree# display dir tree 10/14cja 201434

35. Listing info on files ls – list information about files ~$ ls # list contents of cur dir ~$ ls dir# list contents of dir ~$ ls –l# list details of files in cur dir including access, owner & group, size, and last-modified time ~$ ls –t# list newest files first ~$ ls –R dir# list all files in tree dir ~$ ls –lt dir# options can be combined 10/14cja 201435

36. Working with files These commands manipulate files ~$ mv foo bar# rename file foo to bar ~$ cp foo bar# copy file foo to bar ~$ cp –r dir1 dir2# copy dir tree dir1 to dir2 ~$ cp f1 f2 dir# copy f1 and f2 to directory dir ~$ mkdir dir# create empty directory dir ~$ rmdir dir# remove empty directory dir ~$ rm file# remove file file ~$ rm –r dir# remove directory tree dir 10/14cja 201436

37. Lab 3 Exercise: Create a directory named tutorial in your home directory. In that directory, create a directory named sample and a directory named test. Create a file named msg in directory test that contains a copy of the file /etc/motd. Extra credit: Make the last-modified time of your copy identical to that of /etc/motd. Hint: look at the options of the copy command 10/14cja 201437

38. Compressing and archiving These commands compress and archive files ~$ gzip foo# compress foo to foo.gz ~$ gunzip foo# uncompress foo.gz to foo ~$ bzip2 foo# better compress foo to foo.bz2 ~$ bunzip2 foo# uncompress foo.bz2 to foo ~$ tar cf foo.tar bar# archive subtree bar in file foo.tar ~$ tar xf foo.tar# restore archive from file foo.tar ~$ tar zcf foo.tgz bar # archive and compress ~$ tar jcf foo.tjz bar # archive and compress better 10/14cja 201438

39. Wildcards The shell accepts wildcarded arguments This is also called "shell globbing" Wildcards: ?Matches a single character *Matches zero or more characters [chars]Matches any of the chars [c 1 -c 2 ]Matches chars ‘c 1 ’ through ‘c 2 ’ [^chars]Matches any but the chars ~$ ls foo.?# match files named foo.x, where x is any character ~$ echo *.[cs]# echo files that end in.c or.s ~$ mv [o-z]* save# move files starting with o through z to directory save ~$ echo [^A-Z]?# ??? 10/14cja 201439

40. Shell redirection & pipelining 10/14cja 201440

41. Shell redirection A Linux command can have its inputs and outputs redirected ~$ ls >foo# put list of files in current directory into file foo ~$ ls >>bar# add list of files in current directory to end of file foo ~$ sort <bar# sort lines from file bar ~$ sort <<EOF # sort lines entered at keyboard whiskey bravo tango EOF ~$ wc -l ~/mycounts # count number of lines from file /etc/motd and put result in file mycounts in my home directory 10/14cja 201441

42. More Linux commands More useful Linux tool commands ~$ grep string # show lines of input containing string ~$ tail# show last few lines of input ~$ head# show first few lines of input ~$ sort# sort the input 10/14cja 201442

43. Shell pipelining A Linux command can have its output connected to the input of another Linux command ~$ ls | wc –l# count files in current directory ~$ last | grep reboot# when did we reboot? Exercise: How many people are running MATLAB on your compute node? 10/14cja 201443

44. Editing text files 10/14cja 201444

45. Editing text files Simple editor nano or pico "What you see is what you get” editor Simple to learn if you want to get started quickly Supported editors vi or vim emacs Powerful but more complex If you have time and inclination to become proficient, spend time here 10/14cja 201445

46. Text files Watch out for source code or data files written on Windows systems Use these tools to analyze and convert source files to Linux format file dos2unix 10/14cja 201446

47. Permissions 10/14cja 201447

48. File Permissions Three permission bits, aka mode bits Files: Read, Write, Execute Directories: List, Modify, Search Three user classes User (File Owner), File Group, Other man chmod 10/14cja 201448

49. File Permissions, examples -rw------- cja lsait 40 Oct 1 12:03 foo.bz2 file read and write rights for the owner, no access for anyone else chmod u=rw,g=r,o= file -rw-r----- cja lsait 40 Oct 1 12:03 foo.bz2 file read and write rights for the owner, read for members of the lsait group, no access and for others drwxr-x--x cja lsait 4096 Oct 1 12:15 bar list, modify, and search for the owner, list and search for group, and execute only for others 10/14cja 201449

50. Processes 10/14cja 201450

51. Processes On Linux, every program runs in a process You can examine these processes man ps ps ps ax top 10/14cja 201451

52. Processes You can signal a running process To stop it, or "kill" it man kill 10/14cja 201452

53. Any Questions? Charles J. Antonelli LSAIT Advocacy and Research Support cja@umich.edu http://www.umich.edu/~cja 734 763 0607 cja@umich.edu http://www.umich.edu/~cja cja@umich.edu http://www.umich.edu/~cja 10/14cja 201453

54. References 1.http://en.wikipedia.org/wiki/History_of_Linux http://en.wikipedia.org/wiki/History_of_Linux 2.http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html 3.William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No Starch Press, January 2012. Download Creative Commons Licensed version athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf. Licensed http://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdfLicensed http://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf 10/14cja 201454

55. Extra Task: compile and execute simple programs on the Flux compute node Copy sample code to your login directory: cd cp ~brockp/cac-intro-code.tar.gz. tar -xvzf cac-intro-code.tar.gz cd./cac-intro-code Examine, compile & execute helloworld.f90: ifort -O3 -ipo -no-prec-div -xHost -o f90hello helloworld.f90./f90hello Examine, compile & execute helloworld.c: icc -O3 -ipo -no-prec-div -xHost -o chello helloworld.c./chello Examine, compile & execute MPI parallel code: mpicc -O3 -ipo -no-prec-div -xHost -o c_ex01 c_ex01.c mpirun -np 2./c_ex01 10/14cja 201455