Class 26: Putting it all (or a lot of it) together
CS 202
04 May 2022
On the board
------------
A. Last time
B. What is today's class about?
C. Binaries and loading, assuming static linking
D. Power up to terminal
- Step 1: Power up
- Step 2: Firmware
- Step 3: OS Bootloader
- Step 4: Kernel
- Step 5: init(8)
- Step 6: login(1)
E. Remarks and observations
---------------------------------------------------------------------------
- Announcement: final exam logistics
- Review is Monday; we'll restate all of the class topics since the
midterm, and the rest of the time will be given over to Q&A. Please
bring questions!
A. Last time
- Reflections on Trusting Trust
B. This class:
- A walk through what happens between powering on a computer and
arriving at a terminal.
- Some remarks on the architecture of operating systems.
- This is intended to be "how stuff works" class, while also
drawing connections to many of the subsystems and mechanisms that
we've studied. fsck(), for example, makes an appearance.
C. Binaries and loading, assuming static linking
- All of the steps that follow require loading and executing programs.
Going to start by presenting a high level overview of this process.
- The question we want to answer: What happens when a program executes
the following code?
```
char *argv[];
char *envp[];
// Initialize argv and envp
// ...
if (fork() == 0) {
// Executed in the child process.
execve("hello", argv, envp);
}
```
[Draw picture of process memory space, getting obliterated by
munmap().]
How does the process's memory space get filled out, how
does code start executing? To answer that, need to talk about
the format of an executable file.
- Programs (e.g., 'hello' in the example above) are stored as executable
files in the file system. The format of executable files is dictated by
the operating system on which they are meant to be executed.
+ Linux executables are stored in the Executable and Linkable Format (ELF).
+ Windows executables are stored in the Portable Executable (PE) format.
- Regardless of format, executables contain the following information:
[Draw picture of an executable to the right of the picture of
process memory space. Then show how the loader uses the header
of the executable file to call mmap() to create the new address
space.]
+ Magic number: a known byte sequence at the beginning of the file that
identifies the file as an executable.
* ELF's magic number is 0x7f454c46 (0x7f followed by the string 'ELF').
* PE's magic number is 0x4d5a (the characters 'MZ').
+ The platform (operating system and machine architecture) on which the
executable can be run.
* For example, the ELF format is used by Linux, Solaris, IRIX,
all of the BSDs, MINIX, QNX, etc.; and these operating systems
run on processors including AMD64, x86, ARM of various flavors,
MIPS, PowerPC, SPARC, etc.
* Similarly the PE format is used by all Microsoft OSes, and EFI;
and run on processors including AMD64, x86, Alpha, ARM of various
flavors, etc.
While the format is reused across many platforms, most **executables**
can only be executed on one platform.
+ The **memory layout** a process running the executable should have. Memory
layout information is specified as an array of sections. Each section is a
struct specifying:
* The virtual address at which the section should be placed. The compiler
**assumes** that it knows the virtual address for instructions and variables
when compiling. The compiler does not however assume anything about the
address of the stack, which is accessed using %RSP as we have seen in previous
classes.
* The length of each section.
* Whether the contents of the memory should be read from the file
(e.g., when the entry corresponds to the `.text` or `.data` sections)
or left uninitialized.
* The offset in the file where the contents of the section are stored.
This offset information is ignored for sections which are not read
from the file.
* The memory protection bits that should be set for the section.
+ For the `.text` section then it is marked readable and
executable.
+ For the `.data` section then it is marked readable and
writable.
The memory layout information **does not** include information about the stack.
+ The program's entry point: the virtual memory address at which the first function
that should be executed when the program is run, that is the virtual address of
the `_start` function.
+ Other information that we omit, e.g., symbols, strings, etc.
- Loading and executing an executable, or what happens when
`execve("hello", argv, envp)` is executed:
+ The loader checks that the file is readable and can be executed:
* Check file permissions.
* Check the magic number to ensure file is correctly formatted.
* Check file header to ensure that executable can run on the current platform.
+ The loader calls munmap to unmap the processes memory.
+ The loader reads through the executable's memory layout array and uses mmap to
allocate and set up the processes memory layout:
* Use the executable's file descriptor when allocating sections that need to
be read from the executable file.
* Use MAP_ANONYMOUS for any sections which do not need to be read from
the executable.
* Set protection bits based on information in the section.
+ The loader uses mmap to allocate a stack, and sets RSP and RBP so they point to
the top of the allocated stack. The loader then copies argv to the stack.
+ The loader then sets %RDI to argc and %RSI to point to argv, and jumps to the
entrypoint specified in the executable.
[detail: loader is running in user-space. In general, one
cannot movq to %rip, so one has to jump to the entrypoint,
at least in instructional OSes. In reality, there is a
stub that executes a call to entrypoint; when the call returns
it calls the exit() syscall]
- The loading process above only works for *statically linked* executables, i.e.,
executables compiled with the `-static` flag:
> gcc -static -o hello hello.c
The vast majority of executables are *not* statically linked, and the process
above needs to be expanded to handle dynamic linking and relocations.
The process similarly needs to be changed in order to handle ASLR.
We *omit* information about linking and the handling of both of these cases in
these notes.
D. Power up to terminal
* Step 1: Power up
- Processor and all peripherals turn on, and initialize their internal state.
+ Zero out any registers.
+ Set control registers to default values.
[For Intel defaults are recorded in the Software Development
Manual.]
+ At this point the processor is in **real mode**. In this mode
* Paging is not enabled. All addresses are physical addresses, and
no memory protection is used.
* Programs can access up to 1 megabyte of physical memory.
- The processor copies an executable from a ROM (read-only memory) chip
connected to the processor into RAM, and jumps (using the jmp instruction)
to a known offset of this copied binary (for early processors this was 0xFFFF0,
but newer processors have somewhat more complex logic for this).
+ This initial binary is refered to as the *firmware*.
+ In all modern computers firmware is stored on EEPROMs or Flash, in order
to allow upgrades.
+ The firmware itself has access to stable storage, usually provided by a
battery backed CMOS chip. Firmware settings (e.g., where to boot from, etc.)
are stored in this CMOS chip.
* Step 2: Firmware
- Firmware is reponsible for hardware initialization and providing
a runtime for the kernel during early boot.
- On recent Intel machines, firmware can be broadly classified as either BIOS or
UEFI firmwares. Both are specifications, and many different implementations exist
for both. Here we will focus on UEFI for convenience.
BIOS = basic input output system
UEFI = unified extensible firmware interface
* Many (but not all) ARM64 machines also make use of firmware conforming to the
UEFI spec.
[Historical and pop culture note: Historically, each computer manufacturer would
produce their own firmware which was used to ensure that computers from no other
manufacturers (even computers using the same processor) could run software designed
for the manufacturer.
Patents, copyrights and other legal tools were used to prevent
anyone from copying firmware to get around this limitation.
In the 1980s IBM released the IBM PC without patenting its
firmware (we don't know exactly why they didn't).
This led to several competitors, starting with Columbia Data Products, reverse engineering IBM's
firmware, leading to the creation of IBM PC compatible machines. Today's BIOS descends
from this original effort, and this reverse engineering is what led to the creation and
growth of companies such as Dell.
Season 1 of AMC's Halt and Catch Fire (also available on Netflix) presents a fictional
account of this effort.]
- UEFI initialization steps: UEFI firmware runs the following initialization steps:
* Switch processor from **real mode** to **long mode**. Long mode enables
paging, 64-bit addressing, etc. and has been the mode we have considered
throughout this class. As a part of this switch the firmware:
+ Creates and switches to an identity mapped page table (similar to WeensyOS's
initial page table in Lab 4). Many UEFI implementations map all available
physical memory.
+ Creates and installs an initial Interrupt Descriptor Table (IDT). The IDT
specifies what code should run when interrupts are raised, sycalls are
invoked, etc. UEFI firmwares implement many functions including functions
that can fetch data from the network, wait on a timer, etc., and the
interrupt handlers installed by the firmware are functional (and not stubs).
+ Initializes other processor structures [TSS, GDT, LDT, etc.]
+ Set Control Registers
* Initialize devices including
+ All disk drives
+ The USB hub
+ Display
+ Keyboard and mouse.
+ Network cards
[Aside: In some cases initializing the devices listed above might require
loading a driver (stored with UEFI firmware) or loading and executing a
program stored on the device. This is especially common when initializing
high-performance network cards and some GPUs.]
* Constructs a device tree referred to as the `CONFIGURATION_TABLE` [described below].
* Mount a VFAT partition from the disk (or RAM in case of network boot)
containing the operating system to be booted.
Users configure the device from which the operating system will be loaded,
and this configuration is stored in the CMOS. It is common to provide
the firmware with a list of devices (e.g., USB, disk,
network) in which case the UEFI firmware looks for the first device whose
partition table contains a mountable UEFI VFAT partition.
The VFAT partition is mounted at /EFI.
* Loads and runs (using the same steps listed above) the OS boot loader.
The OS boot loader is an executable stored in the VFAT partition.
+ When booting from a hard disk, the path to the OS bootloader
is a firmware option stored in CMOS.
+ When booting from removable disks (e.g., USB sticks)
the executable must be at `/EFI/boot/bootx64.efi`.
The OS bootloader entry point has signature:
`void EntryPoint(EFI_HANDLE handle, EFI_SYSTEM_TABLE *table)`
+ `handle` is an integer that the bootloader needs to provide as input
when calling into the EFI firmware.
+ `table` is a pointer to a set of function pointers through which
the bootloader can call UEFI services (described below), and a
pointer to the device tree (also described below).
[Note: EFI uses a slightly different calling convention from what we have seen so far:
arguments are passed in %RCX, %RDX, %R8 and %R9.]
- UEFI services
* Like any operating system, UEFI offers the OS bootloader several APIs
including ones for:
+ Sending and receiving network messages.
+ Reading and writing files from the VFAT partition.
+ Drawing and printing on the screen.
+ Getting user input.
+ Get environment variables and arguments passed to the bootloader.
(arguments and environment variables are specified in CMOS.)
+ Etc.
* The OS bootloader must explicitly (using a function call) tell
the UEFI firmware when these services are no longer necessary. The
firmware exits when this is done.
- Device trees (CONFIGURATION_TABLE)
* In class 16 we talked about how operating systems use
device drivers to add support for new devices.
* But how does the operating system know what devices are present on
a machine, and what device drivers to load?
* Answer to the previous question is a device tree, which is a structure
containing:
+ A list of all devices attached to the machine.
+ Are they accessible through explicit I/O, memory-mapped I/O, or DMA?
+ For devices using explicit I/O: what is the device port number.
+ For devices using memory-mapped I/O: what is the physical memory address
mapped to the device.
+ For devices using DMA: what is the physical memory address mapped to the
device's control register.
+ Information about how interrupts from the device are routed.
* Why a tree? Structure also encodes the topology of how devices are connected.
Going to ignore this detail for now.
* Step 3: OS bootloader to Kernel
- The UEFI firmware loads and executes the OS bootloader.
+ On recent Linux kernels the bootloader is the vmlinuz file with a stub
for UEFI, we only consider this case here.
- The vmlinuz file is a compressed executable containing the Linux kernel.
+ High level layout:
(i) PE Header + EFI stub
(ii) ELF header + stub code to decompress bzip2 or gzip data.
(iii) Compressed bzip2 or gzip data (itself containing
an ELF header and the actual kernel).
- vmlinuz assumes that it is given the disk ID for the Linux
root directory ('/') as an argument (as 'root=<device>').
- When executed:
(a) The EFI stub loads and executes the decompression
stub (item (ii) in the layout above.)
(b) The decompression stub uncompresses the compressed
data (item (iii)) into memory. As stated above, the
decompressed data is a statically linked ELF executable
containing the Linux kernel.
(c) The decompression stub loads and executes the Linux kernel,
passing in a pointer to the CONFIGURATION_TABLE and other arguments as input.
(d) The kernel at this point terminates the UEFI
firmware; it doesn't need to terminate the decompression
stub because that stub jumped into the kernel, so there
is not some separate execution context.
* Step 4: Kernel
- The kernel begins by initializing the system. This includes
* Switching away from an identity-mapped virtual address space.
* Rewriting the interrupt descriptor table.
* Loading and initializing device drivers using information contained
in the CONFIGURATION_TABLE.
+ All device drivers run as a part of the kernel.
+ Device drivers communicate with user mode programs through files and
directories mounted in `/dev`.
* Mounting the root device.
+ If required, fsck is run at this point in the boot process.
- After initialization the kernel forks and launches the `init` process.
* Step 5: init
- `init` or PID 1 is the first process executed by the system.
* Many alternative init implementations exist on Linux. Currently
most distributions use `systemd`.
* We are assuming a simpler `init.rc`-like init script here.
- Executed as root (superuser)
- `init` serves three main purposes
* Finish initializing the system.
* Launch the user login manager.
* Relaunch user login manager whenever users logout.
- Finish initializing the system
* The kernel initialized device drivers, but many devices
require additional initialization:
+ For network cards, need to use DHCP or other means to
set IP address.
+ For GPU, need to set initial display resolution.
+ Set power management settings.
* init launches a set of individual programs that handle this
initialization.
+ Programs communicate with device drivers through files and
directories in `/dev`.
+ Early initialization programs are also run as root.
* Many systems run daemons that provide services. Examples include
sshd (allows sshing into a machine), httpd, etc. init is responsible
for launching these programs.
+ Many of these programs bind to network ports between 1-1024. From
class 24, this is only allowed for programs run as root.
+ Most `init` implementations have mechanisms through which the
programs do not have to directly bind to these ports.
See tcpserve in handout 12
(https://cs.nyu.edu/~mwalfish/classes/22sp/lectures/handout12.pdf)]
for an example of how this is done.
When such functionality is available, init (or the daemon) will
drop privileges and run the daemon as an unprivileged user (not root).
+ init is also responsible for detecting failures in daemon processes,
that is detecting cases where they die, and then restarting them
if appropriate. Relies on `wait(2)` for detecting failures.
- Launch the user session manager
* Once the system has finished initializing, users are presented with a login
prompt using which they can **create a session** on the machine.
* Login prompt is created by a session manager.
+ Linux distributions provide many graphical session managers
including slim, gnome-session, etc.
+ We are going to only focus on text based sessions. The cannonical
text based session manager (and the one we focus on) is `login(1)`.
* For the walkthrough in this class, we assume that after initialization,
init launches login(1).
* Step 6: login(1), also the target of Ken Thompson's attack
- login(1) must run as root.
* This is because of limitiations on the behavior of seteuid(2), setegid(2), etc.
that were presented in class 24.
```
> login
login: Cannot possibly work without effective root
```
- login(1) presents a prompt for a username and password.
* Values typed in by the user, checked against information in `/etc/passwd` and `/etc/shadow`.
- If user credentials match, login(1) runs the following steps:
* Forks a new process. The parent uses `waitpid(2)` to wait for child to die,
and then exits. All steps that follow are executed in the child process.
* Calls setuid(2) to set the effective user ID to that of the logged in user.
* Calls setgid(2) to set the effective group ID to the primary group for the user.
[Detail: * Calls setsid(2) to create a new session.]
* Changes directory to the user's home directory (using chdir(2)). `/etc/passwd` stores
the user's home directory, and login(1) relies on this information to change directories
in this case.
* Uses exec() to launch the user's login shell (e.g., bash). Login shell information for
each user is also stored in `/etc/passwd`
* The user now has a shell they can use as normal.
- Users log out by killing their shell process (which was created by the login
process that init created). This results in the parent login process exiting (as
noted in the previous point).
- `init` observes this exit using the same wait based mechanism used for daemons,
and starts a new login process.
E. Remarks and observations
- Observe that booting a machine involves at least two operating systems
* The firmware, for which we considered UEFI, acts as a simple operating system
providing a few services. It does not support multiple processes, and has
only limited functionality.
* The kernel, which provides a richer set of functionality, including schedulers,
etc.
This principle can extend at least two more layers: booting a virtual machine involves
running a virtual machine's firmware implementation on the host operating system, and
then having the virtual machine's firmware load and execute the guest operating system.
Booting a computer is, from one perspective, many repeated iterations of fork and exec.
- Recall that drivers are part of the kernel in Linux.
Kernels where device drivers are a part of the kernel are
referred to as **MONOLITHIC KERNELS**.
An alternate architecture, referred to as **MICROKERNELS**, is one where device drivers
and many other portions of the kernel (e.g., scheduling, etc.) are run as independent
processes. The main argument for microkernels is that they are more modular, which makes
it easier to change kernel behavior. However, this modularity often carries
a performance penalty, and has limited their wider adoption.
Several current operating system kernels, including the ones for Windows and Mac OS X, started
out as microkernels that were then converted to monolithic kernels due to performance concerns.
See the Tannenbaum-Torvalds debates:
https://cs.nyu.edu/~mwalfish/classes/ut/s10-cs372h/ref/ast-torvalds.html
for a classic monolithic-versus-microkernels debate.
----
[credit: Aurojit Panda for this content]