1. CIS 191: Linux and Unix Class 4 September 30th, 2015

2. Check ‘who’ on eniac and see your friends pennkeys? Time for trolling You can do write pennkey, e.g write xiuruo and then type your messages

3. What we have talked about Basic / intermediate commands Permissions Scripting

4. How can we use these Compile and run a C program 100 times and then store the output Run a complicated python script with many flags and arguments Start a server with complicated configurations Analyze relevant information from text files (diff, grep)

5. What we will cover Scheduled tasks and processes (Today!) Regex Version control Makefile Cloud computing Or your suggestions!

6. Feedback is appreciated!

7. Outline Scheduled Jobs Processes Concurrent Operation and Scheduling Unix Process Management Devices

8. Daemons Daemons are just background processes They’re typically used to provide services that must always run – sshd = ssh server process – udevd = hardware daemon – syslogd = logging daemon – (Names end in “d” by convention) Normal users may never come to realize they exist

9. Running tasks periodically Sometimes you want to run a program on a schedule – Generally for administrative tasks “Water my lawn at noon every other day in the summer” In UNIX systems, your tasks might be more along the lines of clearing an error log or taking a backup of your thesis

10. Cron The cron daemon handles periodic tasks – So if you want a task to run periodically, you must submit jobs to the cron daemon…. – It wakes up once a minute and services whatever tasks it has been assigned (after checking to make sure a particular task should run in that minute, of course) Many system tasks use cron logging backups updates

11. Using cron Don’t use cron From the man page: – “The cron utility is launched by launchd when it sees the existence of /etc/crontab or files in /usr/lib/cron/tabs. There should be no need to start it manually.” cron will start by itself when there is a crontab file which specifies jobs that should be run! – So, if you want to interact with cron, you need to edit either the system crontab file or your personal crontab file.

12. crontabs In general, you don’t want to edit the system crontab – Unless you really know what you’re doing! If you want to edit your user-specific crontab, access it by running – $ crontab –e – This will launch your favorite editor, as specified in the shell’s VISUAL or EDITOR environment variable. export EDITOR=vim export VISUAL=vim – set vim as your default EDITOR for command line programs This should just work in Ubuntu, as with many things. – As for OSX…

13. Editing crontabs with vim (OSX edition) OSX has its own ideas about editing crontabs! – It requires that crontab files be edited in-place – nano does this (but I don’t know what buttons to press...) Making vim work with crontab –e in OSX – In.bashrc (set editor to be vim and set a variable in vim session) export EDITOR=vim export VISUAL=vim alias crontab=“env VIM_CRONTAB=true crontab” – In.vimrc (if vim session has variable as true, then change mode) if $VIM_CRONTAB == “true” set nobackup set nowritebackup endif

14. crontab format Each line in a crontab file represents some task you would like the cron daemon to execute MIN HR DOM MN DOW CMD – MIN – minute of the hour – HR – hour of the day – DOM – day of the month – MN – month of the year – DOW – day of the week (0=Sunday,1=Monday,2=Tuesday,…) – CMD – your command – -- whatever arguments you’d like to pass to your command If both DOM and DOW are non-restricted, the command will be run when either field is satisfied

15. crontab syntax Again, recall the * wildcard (yes, it is important) – Wildcard: 30 08 10 06 * /path-to/my_sweet_backup.sh / Run backup at 8:30 on June 10 th, regardless of what weekday it is – Comma: 00 11,16,21 * * * /path-to/my_spam_script.sh Spam peeps at 11:00AM, 4:00PM, and 9:00PM every day – Hyphen: 00 09-18 * * 1-5 /path-to/check_db_status.sh Check database status every hour from 9AM to 6PM, M-F – * * * * * /path-to/tick.sh Run a ticker script every minute (uses every available time frame to cron, and the cron daemon wakes up every minute) – */10 * * * * /path-to/ten_minutes.sh Runs ten_minutes.sh every ten minutes every day See this awesome link for more information!this awesome link

16. Scheduling a one-off task with at Suppose you want to manage submissions right before and right after lecture, but you’re lecturing at that time. The at utility is perfect for this – Allows you to specify a time, and then enter commands to run at the time you’ve specified Perfect for, say, disabling hw2 submission at 1:30, enabling hw2-late submission at 1:30, and enabling hw3 submission at 3:00.

17. at syntax at 1:30PM sep 17 >project –d hw2 >project –e hw2-late >^D job 2 at Wed Sep 17 13:30:00 2014

18. Viewing and Manipulating the atq at –q (or atq) will list the jobs you’ve submitted to at which have not been completed yet at –c will cat the job text to the terminal for you to view at –r will remove a job from the atq

19. Running at In Ubuntu, you’ll probably need to install at – sudo apt-get install at – It should just work after this… In OSX, at relies on the atrun daemon to manage its jobs – See man atrun for how to enable this daemon

20. Outline Scheduled Jobs Processes Concurrent Operation and Scheduling Unix Process Management Devices

21. Warning

22. What is a Program? We spend a lot of our time writing programs, but what are they, really?

23. What is a Program? We spend a lot of our time writing programs, but what are they, really? – Bits of code that run when we tell them to? – A collection of instructions to the machine? – Compiled machine code?

24. What is a Program? We spend a lot of our time writing programs, but what are they, really? – Bits of code that run when we tell them to? – A collection of instructions to the machine? – Compiled machine code? How do we run programs…?

25. Processes Processes are “programs in execution” – They have associated context (data and execution) – The execution must progress sequentially

26. Processes Processes are “programs in execution” – They have associated context (data and execution) – The execution must progress sequentially Processes are not programs or applications – We could run a program any number of times Each instance of a program is run as a process! Each process has its own address space and contexts

27. Why Processes? Operating systems (including Linux versions) have to execute a lot of different programs – And these programs have to run concurrently… Concurrent Processes

28. Why Processes? Operating systems (including Linux versions) have to execute a lot of different programs – And these programs have to run concurrently… Concurrent Processes – Word processor, browser, email all running at once – Internally, memory management processes – Device serving processes…

29. Processes are Separated Each process has its own context in which it executes In fact, each process believes it has access to the entire machine! – This isn’t true, of course – that would lead to disaster – We aren’t going to go into the details of how this is implemented in this class, but you’ll get a full dose of it in both CIS 380 (OS) and CIS 371 (computer organization) Yes, it’s sufficiently complicated to be fully covered in two classes

30. Process Contexts PC – Program Counter – address of next instruction Stack – temporary data – function parameters, return addresses, locals Heap – dynamically allocated memory – objects, strings Static Data – globals Code – text that’s executed

31. Process Creation Parent processes create children processes – This forms a tree of processes A Process is identified and managed through its process identifier (pid) – View processes on your machine with the ps command ps -a will allows you to see all processes (including the os’s)

32. Unix Process Hierarchy A Unix process is spawned by a parent process A Unix process can spawn any amounts of children The “root” process is called init, and it is created when the operating system is booted – So every process can be traced back to init Each process has a unique process ID number

33. Unix Shell and Processes When you type a command, the shell clones itself The child process then calls exec, which causes it to stop executing the shell and start executing your command The parent shell waits for the child to terminate

34. Recall the unix shell and subshells Call a command in parenthesis ( ) to execute the command in a subshell Call a command in curly braces { } to group them but not spawn a subshell Call a command ending with the & symbol to execute the command in a subshell in the background – Commands in the background can’t write to stdout until they terminate (finish)

35. Keeping track of Processes The kernel’s job A data structure, usually called the process table, keeps track of each active process Entry in this table are called process control blocks (PCBs) So a process’ context is described by its PCB! – The kernel can stop and restart processes – In other words, the state of execution of the process is stored

36. Outline Scheduled Jobs Processes Concurrent Operation and Scheduling Unix Process Management Devices

37. Process states Running – The process is currently executing. Only one process can be executing at a time on a single core. Blocked – The process is waiting – Perhaps for I/O? Ready – The process would like to run

38. How to schedule? We need to schedule processes so that the system runs “well” But what does “well” mean?

39. How to schedule? We need to schedule processes so that the system runs “well” But what does “well” mean? It doesn’t necessarily mean what’s best for any particular application – If you’re an application writer, you should consider the operating system to be a malicious adversary… – It makes it easier to write robust programs

40. We need to balance between… Efficiency – Spend as much time in user processes as we can Fairness – Avoid starving processes for compute time Priority Handling – Need to allow more important processes better service Real-time constraints – Need to have a guaranteed level of service Hardware constraints – How much time do we waste switching processes?

41. Take One : FCFS First Come First Serve – When a job arrives, place it on the end of the service queue – The dispatcher selects the first job and runs it to completion of a CPU burst It’s a nice, simple algorithm But it’s inappropriate for interactive systems… – Why?

42. Take two : Shortest Job First Ready queue is thought of as a priority queue, with the shortest job sorted to be first Then run like in FCFS This would have a provably optimal waiting time But starvation is possible And you can’t actually implement it on real computers – Would have to approximate based on CPU bursts

43. Take three : Round Robin First come first serve, but preemptive – So a process can be interrupted against its will We have a circular ready queue, similar to in FCFS Arriving jobs are placed at the end The dispatcher selects the first job in the queue and runs it to the completion of a CPU burst, or until the time quantum expires If the quantum expires, then the job is placed at the end

44. Take three : Round Robin This is a simple, low overhead approach that works for interactive systems But this relies on the time quantum value – If it’s too large (longer than a CPU burst), we approach FCFS – If it’s too small, we spend too much time in context switching – A typical value is around 20-40 ms A good rule of thumb is to choose a quantum so that the large majority of jobs finish their CPU burst in one quantum Round robin makes the assumption that all processes are equally important

45. Task priority But priority can be very important… – We want user-facing processes to be addressed first – Keyboard input, text processors How can we select/compute the priorities?

46. Multi-Level Feedback Queues Use the round robin technique, but have some levels of queues The time quantum gets bigger as you go “down” the queues; priority decreases as you go down Processes start in the highest level queue If a process relinquishes control, it leaves the queue until it’s ready and re-insert to the end of the same queue If a process is not complete at the end of a time quantum, it is put at the bottom of the next queue down A process in a given queue is not scheduled until all the higher queues are empty

47. What happens here? Processes that do a lot of I/O congregate in the higher level queues – Then don’t have to spend much time doing computation – Most of their time is spend waiting for devices Processes that do a lot of processing sink into lower queues, where they’ll have more time to execute (when they are scheduled) This will result in a more responsive system overall – Why?

48. Completely Fair Scheduler In practice, today most Linux distributions make use of the “Completely Fair Scheduler”, or CFS.“Completely Fair Scheduler”, or CFS This makes use of a red-black tree to sort processes which would like to execute based on the amount of time they spent in execution so far The process which has spent the least time in execution is what’s selected next For more information, check out the wikipedia page! (Linked above)

49. Outline Processes Concurrent Operation and Scheduling Unix Process Management Devices

50. Scheduling in Unix Based on the idea of multi-level feedback queues Priorities from -64 to 63 (lower = higher priority) Negative numbers are reserved for processes waiting to service devices Time quantum is.1 seconds – Empirically found to be the longest quantum which doesn’t reduce perceived responsiveness for editors, etc – But the longer quantum means less context switch Adjust priorities to reflect resource requirement (waiting for event) and resource consumption (CPU time)

51. Unix CPU Scheduler There are two values (stored in PCB/process table entry) – p_cpu: estimate of recent CPU usage – p_nice: a user/OS settable weighting factor (-20 -> 20) default is 0, negative is higher priority, positive is lower priority Each process’ priority is calculated periodically – priority = base + p_cpu + p_nice p_cpu is incremented each time the system clock ticks and the process is found to be executing

52. Manually managing processes You can view which processes are running by using the ps command – This can be piped to grep to find a particular process You can then stop a running process by sending signals with the kill command (it does more than kill…) – Really it sends a signal, based on what you pass it – kill -9 Sends the kill signal! This can’t be handled or interrupted. – kill -2 Sends SIGINT (same as ctrl-C) – And other things! Check out the man page for more details.

53. Viewing live process update feed You can run the top command to see a running update of the processes running on your machine Like running ps over and over again… You can also install htop, which is a tricked out version of top, through the apt-get.

54. Outline Processes Concurrent Operation and Scheduling Unix Process Management Devices

55. Remember /dev? /dev contains system device files Linux systems accomplish tons of magic by treating devices as special files – And by pretending that certain non-device objects are files… – Linux employs devices to provide lightweight “system services” The contents of /dev have odd permissions if you check with ls -l

56. Device Files are “Pseudofiles” When you read from or write to a device “file”, the operating system silently intercepts your request and feeds it to a driver The driver knows how to interact with the device Drivers are typically written in C – Can anyone tell me why that might be?

57. Device File Permissions If you took the liberty of running ls –l on the files, you might see something like this The “b” means that this is a “block” device You could also see a “c”, which would mean it is a “character” device The size field has been replaced by a csv field where – The first value represents the major device number – The second value represents the minor device number

58. Types of Devices Character Devices – Denoted by “c” character at start of permissions – Provide direct, unbuffered access to the hardware – Examples: Serial ports, keyboards, sound cards, modems Block Devices – Denoted by “b” character at start of permissions – Provide buffered access to the hardware – Always allowed to read/write any sized block – Buffered => We don’t know when the data will be written Pseudo Devices – Fake devices that are useful

59. Some Pseudo Devices /dev/null – Accepts all data written to it and does nothing – http://devnull-as-a-service.com/ http://devnull-as-a-service.com/ /dev/full – Always full; returns a continuous stream of NULL bytes when read and returns “disk full” error when written to /dev/zero – Produces endless string of zero bits when read /dev/random and /dev/urandom – Contains a stream of random bytes

60. Hard Disk Partitions Each computer may have several hard drives attached, and each hard drive can be divided into several partitions – Windows assigns each partition its own drive letter (like C:) – Linux allows you to specify where the data on a given partition should appear in the filesystem Every hard drive is assigned a name in /dev – Like /dev/sda for the first drive, or /dev/sdb for the second – Naming starts at sd followed by a letter The n th partition of the drive sdb is sdb(n) – So sdb3 is the third partition on the second hard disk

61. Mounting and Unmounting To use a partition, you can use the mount command – The usage is mount device location – For example, mount /dev/sda2 /media/windows The mounted filesystems and devices are tracked in /etc/mtab. You’ll probably need to be root to access it. umount unmounts a directory – Note the absence of the ‘n’ in umount

62. fstab Non-changing filesystem information is written in /etc/fstab – According to the man page, “it is the duty of the system administrator to properly create and maintain this file” – At boot, fstab tells the system which filesystems should be loaded – Afterwards, fstab is used (by mount/umount) to describe how to mount and unmount filesystems fstab entries contain the filesystem location, the mount point, the file system type, options, and information about core dumping and checking the filesystem