1. 1 Applied CyberInfrastructure Concepts 1 Nirav Merchant (nirav@email.arizona.edu) Bio Computing & iPlant Collaborative Eric Lyons (ericlyons@email.arizona.edu) Plant Sciences & iPlant Collaborative University of Arizona http://goo.gl/p4j3mhttp://goo.gl/p4j3m or https://sites.google.com/site/appliedciconcepts/ Will Computers Crash Genomics? Science Vol 331 Feb 2011

2. Tasks for today  Managing your VM (basic LINUX concepts)  Add user, permission, security considerations etc.  Understanding where the files are  Terminal, editors etc.  Shell and shell scripting  Start building your “Data Science ToolBox”

3. Additional reading/reference  Linux Command Line and Shell Scripting Bible, 3rd Edition  The Linux command line: a complete introduction (Both available ebook via UA library) Many online resources:  http://www.freeos.com/guides/lsst/ http://www.freeos.com/guides/lsst/  Goal is to give you enough competence (and confidence) to control your OS !

4. Step #1 for Big Data Toolkit  Command line competency  Really we need command line proficiency ?  Remember: Computational thinking (Abstraction, Automation)

5. What is included ?  Why is LINUX OS the choice for scientific computing ?  What does it mean to have “batteries included” ?  The UNIX Philosophy summarized: A Quarter-Century of Unix (1994) by Peter H. Salus * – Write programs that do one thing and do it well. – Write programs to work together. – Write programs to handle text streams, because that is a universal interface. * Doug McIlroy, E. N. Pinson, B. A. Tague (8 July 1978). "Unix Time-Sharing System Forward". The Bell System Technical Journal. Bell Laboratories. pp. 1902–1903.Doug McIlroy"Unix Time-Sharing System Forward"

6. How much LINUX do I need to know ?  Convergence of OS behavior  Convergence of shells (more later)  Why am I torturing you with useless things ?

7. Work along with me  Open your VM and get to the terminal prompt  Mac and linux users you can open a regular terminal  We will be using commands like: env, ls, who, last, wc etc.  First some basic topics for data and security

8. Permissions  Why do you need them ?  What is a ACL (Access Control List) ?  The UNIX model of permissions (next slides are from Greg Wilson at http://software- carpentry.org)http://software- carpentry.org  Path statement and finding things

9. Has unique user name and user ID user

10. Has unique user name and user ID User name is text: "imhotep", "larry", "vlad", … user

11. Has unique user name and user ID User name is text: "imhotep", "larry", "vlad", … User ID is numeric (easier for computer to store)

12. usergroup

13. usergroup Has unique group name and group ID

14. usergroup Has unique group name and group ID User can belongs to zero or more groups

15. usergroup Has unique group name and group ID User can belongs to zero or more groups List is usually stored in /etc/group

16. usergroupall

17. usergroupall Everyone else

18. usergroupall Has user and group IDs

19. usergroupall read

20. usergroupall read write

21. usergroupall read write execute

22. usergroupall read ✔✔✗ write ✔✗✗ execute ✗✗✗

23. usergroupall read ✔✔✗ write ✔✗✗ execute ✗✗✗ File's owner can read and write it

24. usergroupall read ✔✔✗ write ✔✗✗ execute ✗✗✗ File's owner can read and write it Others in group can read

25. usergroupall read ✔✔✗ write ✔✗✗ execute ✗✗✗ File's can read and write it Others in group can read That's all

26. Where are my files ? Understanding layout of data –Home –Root –Tmp Permissions Storage space and planning for it Managing runaway items (more in next class)

27. Security considerations Update your OS (how can you do that ?) Why you should NEVER run as root (how do I add a user ?) Password and keys (and dual factor) Ssh foo

28. What is Shell? Shell is –Command Interpreter that turns text that you type (at the command line) in to actions: –User Interface: take the command from user Programming Shell can do –Customization of a Unix session –Scripting –Many Many automation steps

29. What is Shell? Shell is –Command Interpreter that turns text that you type (at the command line) in to actions: –User Interface: take the command from user Programming Shell can do –Customization of a Unix session –Scripting –Many Many automation steps

30. Customization of a Session Each shell supports some customization. –User prompt –Where to find mail –Shortcuts (alias) The customization takes place in startup files –Startup files are read by the shell when it starts up –The Startup files can differ for different shell

31. Popular Shells  sh Bourne Shell  ksh Korn Shell  csh,tcsh C Shell (for this course)  bash Bourne-Again Shell

32. Flavors of Unix Shells Two main flavors of Unix Shells –Bourne (or Standard Shell): sh, ksh, bash, zsh Fast $ for command prompt –C shell : csh, tcsh better for user customization and scripting %, > for command prompt To check shell: –% echo $SHELL (shell is a pre-defined variable) To switch shell: –% exec shellname (e.g., % exec bash)

33. Startup files and why you should care  bash: /etc/profile (out-of-the-box login shell settings) /etc/bash.bashrc (out-of-box non-login settings) /etc/bash.bashrc.local (global non-login settings) ~/.bash_profile (login shell user customization) ~/.bashrc(non-login shell user customization) ~/.bash_logout (user exits from interactive login shell) http://cli.learncodethehardway.org/bash_cheat_sheet.pdf

34. Some Special Keys How do you invoke tcsh ? Ctrl-U = Delete everything on the command- line Ctrl-A = Move cursor to the front Ctrl-E = Move cursor to the end Ctrl-P = Set the current command-line to the previous command Ctrl-N = Set the current command-line to the next command TAB = Filename completion

35. Preview pieces of toolbox http://datascienceatthecommandline.com/ We will work though Step 5 and go straight to commands We will wrestle and wrangle with CSV things

36. Next class  Preparing to play with your data set – Can you download a piece of it ?  Learn about space and process management  Introduction to shell scripting and automation  Start building your Big Data command line tool kit (document that on the wiki and git hub for others to emulate)