Class 3 CS 202 04 February 2020 On the board ------------ 1. Last time 2. Process/OS control transfers 3. Git/lab setup 4. Process birth 5. The shell, part I 6. File descriptors 7. The shell, part II --------------------------------------------------------------------------- 1. last time: - CPU - process's view of memory and registers - stack frames * demystifying functional scope * demystifying pointers - system calls * how programmers get work out of the system clarification on stack frames: - recall we had main() --> f() --> g(). Some of you asked how, right after g() returns to f(), the frame pointer is in the right place for f. This is the purpose of the prologue/epilogue in g(). The prologue/epilogue are inverse operations (save old frame pointer, restore old one). So the result is that from the perspective of f: after g returns, _the frame pointer hasn't moved_ (even though of course it did). While the frame pointer has a special role in compiled code, this aspect of the frame pointer -- a called function saves the old value at the beginning of the function and restores that old value at the end -- is shared with other registers, known as _call-preserved_ registers. (Also known as "callee-saved"). 2. Process/OS control transfers - To the C compiler (or the assembly programmer) and the machine as a whole, a system call has some key differences versus function calls (even though both are transfers of control): (i) there is a small difference in calling conventions -- a process knows that when it invokes "syscall", ALL registers (except RAX) are call-preserved. That means that the callee (in this case the kernel) is required to save and restore all registers (except RAX, which is the exception because that is where return values go). (ii) Rather than using the "call" instruction, the process uses a different instruction (helpfully called the `syscall` instruction). This causes privilege levels to switch. The picture looks like this: user-level application | (open) v user-level --------------------------- ^ | kernel-level | |____> [table] open() | ..... | iret --------- - Vocabulary: when a user-level program invokes the kernel via a syscall, it is called *trapping* to the kernel Key distinction: privileged versus unprivileged mode --the difference between these modes is something that the *hardware* understands and enforces --the OS runs in privileged mode --can mess with the hardware configuration --users' tasks run in unprivileged mode --cannot mess with the hardware configuration --the hardware knows the difference between privileged and unprivileged mode (on the x86, these are called ring 0 and ring 3. The middle rings aren't used in the classical setup, but they are used in some approaches to virtualization.) - Overall, there are three ways that the OS (also known as the kernel) is invoked: A. system calls, covered above. B. interrupts. An _interrupt_ is a hardware event; it allows a device, whether peripheral (like a disk) or built-in (like a timer) to notify the kernel that it needs attention. (As we will see later, timers are essential for ensuring that processes don't hog the CPU.) Interrupts are **implicit**: in most cases, the application that was running at the time of the interrupt _has no idea that an interrupt even triggered_, despite the fact that handling the interrupt requires these high-level steps: - process stops running - CPU invokes interrupt handler - interrupt handler is part of kernel code, so kernel starts running - kernel handles interrupt - kernel returns control In other words, from the process's viewpoint, it executed continuously, but an omniscient observer would know perfectly well that the process was in fact _interrupted_ (hence the term). In order to preserve this illusion, the processor (CPU) and kernel have to be designed very carefully to save _all_ process state on an interrupt, and restore all of it. We will discuss the underlying mechanisms for these control transfers later in the course. C. exceptions An _exception_ means that the CPU cannot execute the instruction issued by the processor. Classically (and for this part of the course), you can think of this as "the process did something erroneous" (a software bug): dividing by 0, accessing a bogus memory address, or attempting to execute an unknown instruction. But there are non-erroneous causes of exceptions (an example is demand paging, as we will see in the virtual memory unit). When an exception happens, the processor (the CPU) knows about it immediately. The CPU then invokes an _exception handler_ (code implemented by the kernel). The kernel can handle exceptions in a variety of ways: - kill the process (this is the default, and what is happening when you see a segfault in one of your programs). - signal to your process (this is how runtimes like Java generate null-pointer exceptions; processes _register_ to catch signals). - silently handle the exception (this is how the kernel handles certain memory exceptions, as in the demand paging case). The mechanisms here relate to those for interrupts. 3. Git/lab setup [draw a picture] [your computer has git; use that to get base-image (technically vagrantfile] vagrant invokes virtualbox; virtualbox runs Debian Buster. git inside of there fork remote labs, get your labs your origin: that fork your upstream: our "labs" [different use of the word fork from later in this class] 5 repos in all 4. Process birth How does a process come into being? --answer: a system call! --in Unix, it is fork() --fork() creates an exact copy (almost; the return value is different). --thus, what happens if a system had two important users, and one of them runs a process that executes this code: for (i = 0; i < 10; i++) { fork(); } while (1) {} [answer: one of the users gets a LOT more of the CPU than another] --what behavior do you want? [this actually corresponds to research.... difficult on Linux-like systems to impose true resource containers.] 5. The shell, part I --a program that creates processes --the human's interface to the computer --GUIs (graphical user interfaces) are another kind of shell. A. How does the shell start programs? --example: $ ls [see panel 1 on handout; go line-by-line] --calls fork(), which creates a copy of the shell. now there are two copies of the shell running --then calls exec(), which loads the new program's instructions into memory and begins executing them. --(exec invokes the loader) while (1) { write(1, "$ ", 2); readcommand(command, args); // parse input if ((pid = fork()) == 0) // child? execve(command, args, 0); else if (pid > 0) // parent? wait(0); //wait for child else perror("failed to fork"); } --how can shell wait for the end of a process? --with wait() or waitpid() system calls --background process. example: $ sleep 10 vs $ sleep 10 & --QUESTION: why is fork different from exec? What the heck? * We will come back to this. B. Redirection, motivation What does this do? $ ./first3 abcd efgh > foo or say we wanted to extract all of your GitHub ids...how would you do that without pipelines? download html from https://github.com/nyu-cs202 then $ cat blob | grep -o labs-[a-zA-Z0-9\-]* | sort -f | uniq > students.txt How are these things implemented? Remember, the programmer of first3 or cat or grep is long gone, and their output is winding up somewhere that the original program never specified. 6. File descriptors --every process can usually expect to begin life with three file descriptors already open: 0: represents the input to the process (e.g., tied to terminal) 1: represents the output 2: represents the error output these are sometimes known as stdin, stdout, stderr --NOTE: Unix hides for processes the difference between a device and a file. this is a very powerful hiding (or abstraction), as we will see soon 7. The shell, part II - Back to $ ./first3 abcd efgh > foo How is that implemented? Answer: just before exec, shell does: close(1) open("/tmp/foo", O_TRUNC | O_CREAT | O_WRONLY, 0666) which automatically assigns fd 1 to point to /foo/tmp. --now, when first3 runs, it still has in its code: write(1,...), but "1" now means something else. What about $ sh < script > tmp1 where script contains echo abc echo def [draw picture] - The power of the fork/exec separation [an innovation from the original Unix. possibly lucky design choice at the time. but turns out to work really well. allows the child to manipulate environment and file descriptors *before* exec, so that the *new* program may in fact encounter a different environment] --To generalize redirections and pipelines, there are lots of things the parent shell might want to manipulate in the child process: file descriptors, environment, resource limits. --yet fork() requires no arguments! --Contrast with CreateProcess on Windows: BOOL CreateProcess( name, commandline, security_attr, thr_security_attr, inheritance?, other flags, new_env, curr_dir_name, .....) [http://msdn.microsoft.com/en-us/library/ms682425(v=VS.85).aspx] there's also CreateProcessAsUser, CreateProcessWithLogonW, CreateProcessWithTokenW, ... * The issue is that any conceivable manipulation of the environment of the new process has to be passed through arguments, instead of via arbitrary code. in other words: because whoever calls CreateProcess() (or its variant) needs to perfectly configure the process before it starts running. with fork(), whoever calls fork() **is still running** so can arrange to do whatever it wants, without having to work through a rigid interface like the above. allows arbitrary "setup" of the process before exec(). - Discussion: what makes a good abstraction? --simple but powerful --examples we've seen: --stdin (0), stdout (1), stderr (2) [nice by themselves, but when combined with the mechanisms below, things get even better] --file descriptors --fork/exec() separation --very few mechanisms lead to a lot of possible functionality Aside: - Fork bomb at the bash command prompt: $ :(){ : | : & }; : --------------------------------------------------------------------------- --here are some other system calls (these are included in the notes so that you know what the basic interface to a Unix-like OS looks like): --int open(char*, int flags, [, int mode]); --int read(int fd, void*, int nbytes): --int write(int fd, void* buf, int nbytes); --off_t lseek(int fd, off_t pos, int whence) --int close(int fd); --int kill(int pid, int signal) --void exit (int status) --int fork(void) --int waitpid(int pid, int* stat, int opt) --int execve(char* prog, char** argv, char** envp) --int dup2 (int oldfd, int newfd) --int pipe(int fds[2])