Handed out Tuesday, March 5, 2013
Due Monday, April 8, 2013, 11:59 PM
In this lab you will implement preemptive multitasking among multiple simultaneously active user-mode environments. To accomplish this, you will first add multiprocessor support to JOS and implement round-robin scheduling.
Use Git to commit your Lab 5 source, fetch the latest version of the course repository, and then create a local branch called lab6 based on our lab6 branch, origin/lab6:
tig% cd ~/cs439/labs tig% git commit -am 'my solution to lab5' Created commit 734fab7: my solution to lab5 4 files changed, 42 insertions(+), 9 deletions(-) tig% git pull Already up-to-date. tig% git checkout -b lab6 origin/lab6 Branch lab6 set up to track remote branch refs/remotes/origin/lab6. Switched to a new branch "lab6" tig% git merge lab5 Merge made by recursive. ... tig%Lab 6 contains a number of new source files, some of which you should browse before you start:
| kern/cpu.h | Kernel-private definitions for multiprocessor support |
| kern/mpconfig.c | Code to read the multiprocessor configuration |
| kern/lapic.c | Kernel code driving the local APIC unit in each processor |
| kern/mpentry.S | Assembly-language entry code for non-boot CPUs |
| kern/spinlock.h | Kernel-private definitions for spin locks, including the big kernel lock |
| kern/spinlock.c | Kernel code implementing spin locks |
| kern/sched.c | Code skeleton of the scheduler that you are about to implement |
As before, you will need to do all of the regular exercises described in the lab. You also need to answer all numbered questions and place your write-up in a file called answers.txt at the top-level of your lab 6 branch. Make sure to git add this file so it is turned in with the rest of your code. You do not need to do any of the challenge problems, though we suggest you try to do one if you have the time.
When you are ready to hand in your lab code and write-up, create a file called slack.txt noting how many slack hours you have used both for this assignment and in total. (This is to help us agree on the number that you have used.) Then run make turnin in the lab directory. This will first do a make clean to clean out any .o files and executables, and then create a tar file called lab6-handin.tar.gz with the entire contents of your lab directory and submit it via the CS turnin utility. If you submit multiple times, we will take the latest submission and count slack hours accordingly.
In this lab you will first extend JOS to run on a multiprocessor system. You will also implement cooperative round-robin scheduling, allowing the kernel to switch from one environment to another when the current environment voluntarily relinquishes the CPU (or exits). Later you will implement preemptive scheduling, which allows the kernel to re-take control of the CPU from an environment after a certain time has passed even if the environment does not cooperate.
We are going to make JOS support "symmetric multiprocessing" (SMP), a multiprocessor model in which all CPUs have equivalent access to system resources such as memory and I/O buses. While all CPUs are functionally identical in SMP, during the boot process they can be classified into two types: the bootstrap processor (BSP) is responsible for initializing the system and for booting the operating system; and the application processors (APs) are activated by the BSP only after the operating system is up and running. Which processor is the BSP is determined by the hardware and the BIOS. Up to this point, all your existing JOS code has been running on the BSP.
In an SMP system, each CPU has an accompanying local APIC (LAPIC) unit. The LAPIC units are responsible for delivering interrupts throughout the system. The LAPIC also provides its connected CPU with a unique identifier. In this lab, we make use of the following basic functionality of the LAPIC unit (in kern/lapic.c):
cpunum()). STARTUP interprocessor interrupt (IPI) from
the BSP to the APs to bring up other CPUs (see
lapic_startap()).apic_init()).A processor accesses its LAPIC using memory-mapped I/O (MMIO). In MMIO, a portion of physical memory is hardwired to the registers of some I/O devices, so the same load/store instructions typically used to access memory can be used to access device registers. You've already seen one IO hole at physical address 0xA0000 (we use this to write to the CGA display buffer). The LAPIC lives in a hole starting at physical address 0xFE000000 (32MB short of 4GB), so it's too high for us to access using our usual direct map at KERNBASE. The JOS virtual memory map leaves a 4MB gap at MMIOBASE so we have a place to map devices like this. Since later labs introduce more MMIO regions, you'll write a simple function to allocate space from this region and map device memory to it.
Exercise 1.
Implement mmio_map_region in kern/pmap.c. To
see how this is used, look at the beginning of
lapic_init in kern/lapic.c. You'll have to do
the next exercise, too, before the tests for
mmio_map_region will run.
Before booting up APs, the BSP should first collect information
about the multiprocessor system, such as the total number of
CPUs, their APIC IDs and the MMIO address of the LAPIC unit.
The mp_init() function in kern/mpconfig.c
retrieves this information by reading the MP configuration
table that resides in the BIOS's region of memory.
The boot_aps() function (in kern/init.c) drives
the AP bootstrap process. APs start in real mode, much like how the
bootloader started in boot/boot.S, so boot_aps()
copies the AP entry code (kern/mpentry.S) to a memory
location that is addressable in the real mode. Unlike with the
bootloader, we have some control over where the AP will start
executing code; we copy the entry code to 0x7000
(MPENTRY_PADDR), but any unused, page-aligned
physical address below 640KB would work.
After that, boot_aps() activates APs one after another, by
sending STARTUP IPIs to the LAPIC unit of the corresponding
AP, along with an initial CS:IP address at which the AP
should start running its entry code (MPENTRY_PADDR in our
case). The entry code in kern/mpentry.S is quite similar to
that of boot/boot.S. After some brief setup, it puts the AP
into protected mode with paging enabled, and then calls the C setup
routine mp_main() (also in kern/init.c).
boot_aps() waits for the AP to signal a
CPU_STARTED flag in cpu_status field of
its struct Cpu before going on to wake up the next one.
Exercise 2.
Read boot_aps() and mp_main() in
kern/init.c, and the assembly code in
kern/mpentry.S. Make sure you understand the control flow
transfer during the bootstrap of APs. Then modify your implementation
of page_init() in kern/pmap.c to avoid adding
the page at MPENTRY_PADDR to the free list, so that we
can safely copy and run AP bootstrap code at that physical address.
Your code should pass the updated check_page_free_list()
test (but might fail the updated check_kern_pgdir()
test, which we will fix soon).
Question
KERNBASE just like
everything else in the kernel, what is the purpose of macro
MPBOOTPHYS? Why is it
necessary in kern/mpentry.S but not in
boot/boot.S? In other words, what could go wrong if it
were omitted in kern/mpentry.S?
When writing a multiprocessor OS, it is important to distinguish
between per-CPU state that is private to each processor, and global
state that the whole system shares. kern/cpu.h defines most
of the per-CPU state, including struct Cpu, which stores
per-CPU variables. cpunum() always returns the ID of the
CPU that calls it, which can be used as an index into arrays like
cpus. Alternatively, the macro thiscpu is
shorthand for the current CPU's struct Cpu.
Here is the per-CPU state you should be aware of:
Per-CPU kernel stack.
Because multiple CPUs can trap into the kernel simultaneously,
we need a separate kernel stack for each processor to prevent them from
interfering with each other's execution. The array
percpu_kstacks[NCPU][KSTKSIZE] reserves space for NCPU's
worth of kernel stacks.
In Lab 4, you mapped the physical memory that bootstack
refers to as the BSP's kernel stack just below
KSTACKTOP.
Similarly, in this lab, you will map each CPU's kernel stack into this
region with guard pages acting as a buffer between them. CPU 0's
stack will still grow down from KSTACKTOP; CPU 1's stack
will start KSTKGAP bytes below the bottom of CPU 0's
stack, and so on.
We have revised inc/memlayout.h to show the new mapping.
Per-CPU TSS and TSS descriptor.
A per-CPU task state segment (TSS) is also needed in order to specify
where each CPU's kernel stack lives. The TSS for CPU i is stored
in cpus[i].cpu_ts, and the corresponding TSS descriptor is
defined in the GDT entry gdt[(GD_TSS0 >> 3) + i]. The
global ts variable defined in kern/trap.c will
no longer be useful.
Per-CPU current environment pointer.
Since each CPU can run different user process simultaneously, we
redefined the symbol curenv to refer to
cpus[cpunum()].cpu_env (or thiscpu->cpu_env), which
points to the environment currently executing on the
current CPU (the CPU on which the code is running).
Per-CPU system registers.
All registers, including system registers, are private to a
CPU. Therefore, instructions that
initialize these registers, such as lcr3(),
ltr(), lgdt(), lidt(), etc., must
be executed once on each CPU. Functions env_init_percpu()
and trap_init_percpu() are defined for this purpose.
Per-CPU idle environment.
JOS uses an idle environment as a fallback to run if there
aren't enough regular environments to run. However, an environment
can only run on one CPU at a time. Since multiple CPUs can be idle at
the same time, we create one idle environment per CPU. By convention,
envs[cpunum()] is the idle environment of the current
CPU.
Exercise 3.
Modify mem_init_mp() (in kern/pmap.c) to map
per-CPU stacks starting
at KSTACKTOP, as shown in
inc/memlayout.h. The size of each stack is
KSTKSIZE bytes plus KSTKGAP bytes of
unmapped guard pages. Your code should pass the new check in
check_kern_pgdir() and the SMP page
management test in make grade.
Exercise 4.
The code in trap_init_percpu() (kern/trap.c)
initializes the TSS and
TSS descriptor for the BSP. It worked in Lab 5, but is incorrect
when running on other CPUs. Change the code so that it can work
on all CPUs. (Note: your new code should not use the global
ts variable any more.)
When you finish the above exercises, run JOS in QEMU with 4 CPUs using make qemu CPUS=4 (or make qemu-nox CPUS=4), you should see output like this:
... Physical memory: 66556K available, base = 640K, extended = 65532K check_page_alloc() succeeded! check_page() succeeded! check_kern_pgdir() succeeded! check_page_installed_pgdir() succeeded! SMP: CPU 0 found 4 CPU(s) enabled interrupts: 1 2 SMP: CPU 1 starting SMP: CPU 2 starting SMP: CPU 3 starting [00000000] new env 00001000 [00000000] new env 00001001 [00000000] new env 00001002 [00000000] new env 00001003 [00000000] new env 00001004 [00000000] new env 00001005 [00000000] new env 00001006 [00000000] new env 00001007 [00000000] new env 00001008 kernel panic on CPU 0 at kern/sched.c:43: scheduler is not yet implemented Welcome to the JOS kernel monitor! Type 'help' for a list of commands. K>
If your output does not look like this, go back and check your work for the previous exercises in this lab. You will need to have these exercises working properly to be able to do the rest of the exercises in this lab.
Our current code spins after initializing the AP in
mp_main(). Before letting the AP get any further, we need
to first address race conditions when multiple CPUs run kernel code
simultaneously. The simplest way to achieve this is to use a big
kernel lock.
The big kernel lock is a single global lock that is held whenever an
environment enters kernel mode, and is released when the environment
returns to user mode. In this model, environments in user mode can run
concurrently on any available CPUs, but no more than one environment can
run in kernel mode; any other environments that try to enter kernel mode
are forced to wait.
kern/spinlock.h declares the big kernel lock, namely
kernel_lock. It also provides lock_kernel()
and unlock_kernel(), shortcuts to acquire and
release the lock. You should apply the big kernel lock at four locations:
i386_init(), acquire the lock before the BSP wakes up the
other CPUs.
mp_main(), acquire the lock after initializing the AP,
and then call sched_yield() to start running environments
on this AP.
trap(), acquire the lock when trapped from user mode.
To determine whether a trap happened in user mode or in kernel mode,
check the low bits of the tf_cs.
env_run(), release the lock right before
switching to user mode. Do not do that too early or too late, otherwise
you will experience races or deadlocks.
Exercise 5.
Apply the big kernel lock as described above, by calling
lock_kernel() and unlock_kernel() at
the proper locations. Your code should now pass the second
core SMP functionality test in make grade. If it
doesn't, go back and make sure that you locked and unlocked the kernel
in the right places.
Question
Challenge! The big kernel lock is simple and easy to use. Nevertheless, it eliminates all concurrency in kernel mode. Most modern operating systems use different locks to protect different parts of their shared state, an approach called fine-grained locking. Fine-grained locking can increase performance significantly, but is more difficult to implement and error-prone. If you are brave enough, drop the big kernel lock and embrace concurrency in JOS!
It is up to you to decide the locking granularity (the amount of data that a lock protects). As a hint, you may consider using spin locks to ensure exclusive access to these shared components in the JOS kernel:
Your next task in this lab is to change the JOS kernel so that it does not always just run the idle environments, but instead can alternate between multiple environments in "round-robin" fashion.
Before you will be able to run any tests for your scheduler, you
will need to allow environments to call the new
sys_bifurcate() syscall, which we have provided for
you.
This syscall acts very much like a primitive fork() in that it
creates a new child environment and copies over the entire address
space of the parent environment to the child environment.
However, unlike a fork() in UNIX, a child is not
entirely ready to start running after sys_bifurcate()
returns; some state in the user-level library needs to be set up
upon return. We have provided the code for the fork()
call for you in the user-level library to invoke the
sys_bifurcate() syscall and set up needed state in the
child environment.
Exercise 6.
Add a case to dispatch sys_bifurcate() in the
syscall() function in kern/syscall.c. This
will allow the fork() calls made in the user-level
scheduler test programs to succeed.
Now that environments in JOS are able to fork off new child environments, you can proceed to implement scheduling in the kernel. Round-robin scheduling in JOS works as follows:
NCPU
environments will from now on always be special idle
environments,
which always run the program user/idle.c.
The purpose of this program is simply to "waste time"
whenever the processor has nothing better to do -
it just perpetually attempts to give up the CPU
to another environment.
Read the code and comments in user/idle.c
for other useful details.
We have modified kern/init.c for you
to create these special idle environments in
envs[0] though envs[NCPU-1]
before creating the first "real" environment in envs[NCPU].sched_yield() in the new kern/sched.c
is responsible for selecting a new environment to run.
It searches sequentially through the envs[] array
in circular fashion,
starting just after the previously running environment
(or at the beginning of the array
if there was no previously running environment),
picks the first environment it finds
with a status of ENV_RUNNABLE
(see inc/env.h),
and calls env_run() to jump into that environment.
However, sched_yield() is aware
of the special idle environments,
and never picks schedules one unless
there are no other runnable environments.sched_yield() must never run the same environment
on two CPUs at the same time. It can tell that an environment
is currently running on some CPU (possibly the current CPU)
because that environment's status will ENV_RUNNING.sys_yield(),
which user environments can call
to invoke the kernel's sched_yield() function
and thereby voluntarily give up the CPU to a different environment.
As you can see in user/idle.c,
the idle environment does this routinely.Exercise 7.
Implement round-robin scheduling in sched_yield()
as described above. Don't forget to modify
syscall() to dispatch sys_yield().
Run make run-yield-nox to run the yield cooperative scheduler test program. You should see the environments switch back and forth between each other five times before terminating, like this:
... Hello, I am environment 00001008. Hello, I am environment 00001009. Hello, I am environment 0000100a. Back in environment 00001008, iteration 0. Back in environment 00001009, iteration 0. Back in environment 0000100a, iteration 0. Back in environment 00001008, iteration 1. Back in environment 00001009, iteration 1. Back in environment 0000100a, iteration 1. ...
After the yield programs exit, when only idle environments are runnable, the scheduler should invoke the JOS kernel monitor. If any of this does not happen, then fix your code before proceeding. Your code should now pass the make grade test for yield.
Question
env_run() you should have
called lcr3(). Before and after the call to
lcr3(), your code makes references (at least it should)
to the variable e, the argument to env_run.
Upon loading the %cr3 register, the addressing context
used by the MMU is instantly changed. But a virtual
address (namely e) has meaning relative to a given
address context -- the address context specifies the physical address to
which the virtual address maps. Why can the pointer e be
dereferenced both before and after the addressing switch?
Challenge! Add a less trivial scheduling policy to the kernel, such as a fixed-priority scheduler that allows each environment to be assigned a priority and ensures that higher-priority environments are always chosen in preference to lower-priority environments. If you're feeling really adventurous, try implementing a Unix-style adjustable-priority scheduler or even a lottery or stride scheduler. (Look up "lottery scheduling" and "stride scheduling" in Google.)
Write a test program or two that verifies that your scheduling algorithm is working correctly (i.e., the right environments get run in the right order).
Challenge!
The JOS kernel currently does not allow applications
to use the x86 processor's x87 floating-point unit (FPU),
MMX instructions, or Streaming SIMD Extensions (SSE).
Extend the Env structure
to provide a save area for the processor's floating point state,
and extend the context switching code
to save and restore this state properly
when switching from one environment to another.
The FXSAVE and FXRSTOR instructions may be useful,
but note that these are not in the old i386 user's manual
because they were introduced in more recent processors.
Write a user-level test program
that does something cool with floating-point.
Now, you will modify the kernel to preempt uncooperative environments.
Run the user/spin test program. This test program forks off a child environment, which simply spins forever in a tight loop once it receives control of the CPU. Neither the parent environment nor the kernel ever regains the CPU. This is obviously not an ideal situation in terms of protecting the system from bugs or malicious code in user-mode environments, because any user-mode environment can bring the whole system to a halt simply by getting into an infinite loop and never giving back the CPU. In order to allow the kernel to preempt a running environment, forcibly retaking control of the CPU from it, we must extend the JOS kernel to support external hardware interrupts from the clock hardware.
External interrupts (i.e., device interrupts) are referred to as IRQs.
There are 16 possible IRQs, numbered 0 through 15.
The mapping from IRQ number to IDT entry is not fixed.
pic_init in picirq.c maps IRQs 0-15
to IDT entries IRQ_OFFSET through IRQ_OFFSET+15.
In inc/trap.h,
IRQ_OFFSET is defined to be decimal 32.
Thus the IDT entries 32-47 correspond to the IRQs 0-15.
For example, the clock interrupt is IRQ 0.
Thus, IDT[IRQ_OFFSET+0] (i.e., IDT[32]) contains the address of
the clock's interrupt handler routine in the kernel.
This IRQ_OFFSET is chosen so that the device interrupts
do not overlap with the processor exceptions,
which could obviously cause confusion.
(In fact, in the early days of PCs running MS-DOS,
the IRQ_OFFSET effectively was zero,
which indeed caused massive confusion between handling hardware interrupts
and handling processor exceptions!)
In JOS, external device interrupts are always disabled
when in the kernel (and enabled when in user space).
External interrupts are controlled by the FL_IF flag bit
of the %eflags register
(see inc/mmu.h).
When this bit is set, external interrupts are enabled.
While the bit can be modified in several ways,
because of our simplification, we will handle it solely
through the process of saving and restoring the %eflags register
as we enter and leave user mode.
You will have to ensure that the FL_IF flag is set in
user environments when they run so that when an interrupt arrives, it
gets passed through to the processor and handled by your interrupt code.
Otherwise, interrupts are masked,
or ignored until interrupts are re-enabled.
We masked interrupts with the very first instruction of the bootloader,
and so far we have never gotten around to re-enabling them.
Exercise 8.
Modify kern/trapentry.S and kern/trap.c to
initialize the appropriate entries in the IDT and provide
handlers for IRQs 0 through 15. Then modify the code
in env_alloc() in kern/env.c to ensure
that user environments are always run with interrupts enabled.
The processor never pushes an error code or checks the Descriptor Privilege Level (DPL) of the IDT entry when invoking a hardware interrupt handler. You might want to re-read section 9.2 of the 80386 Reference Manual, or section 5.8 of the IA-32 Intel Architecture Software Developer's Manual, Volume 3, at this time.
After doing this exercise, if you run your kernel with any test program that runs for a non-trivial length of time (e.g., spin), you should see the kernel print trap frames for hardware interrupts. While interrupts are now enabled in the processor, JOS isn't yet handling them, so you should see it misattribute each interrupt to the currently running user environment and destroy it. Eventually it should run out of environments to destroy and drop into the monitor.
In the user/spin program, after the child environment was first run, it just spun in a loop, and the kernel never got control back. We need to program the hardware to generate clock interrupts periodically, which will force control back to the kernel where we can switch control to a different user environment.
The calls to lapic_init and pic_init
(from i386_init in init.c),
which we have written for you,
set up the clock and the interrupt controller to generate interrupts.
You now need to write the code to handle these interrupts.
Exercise 9.
Modify the kernel's trap_dispatch() function
so that it calls sched_yield()
to find and run a different environment
whenever a clock interrupt (IRQ_OFFSET + IRQ_TIMER)
takes place.
You should now be able to get the user/spin test to work:
the parent environment should fork off the child,
sys_yield() to it a couple times
but in each case regain control of the CPU after one time slice,
and finally kill the child environment and terminate gracefully.
This is a great time to do some regression testing. Make sure that you haven't broken any earlier part of that lab that used to work (e.g. yield, or the user test programs from lab 5) by enabling interrupts. Don't run grade-lab5 to do regression testing, as this script is not designed to recognize the new "incoming trap frame" output format used with the SMP modifications, so it will erroneously say that tests have failed when they haven't. To test programs from lab 5, run them manually e.g. make run-divzero-nox and inspect the output.
You should also try running with multiple CPUs using make CPUS=k target for values of k from 2 to 8. Make sure that all the tests you run function normally regardless of how many CPUs you run JOS with. You should also be able to pass stresssched now. Run make grade to see for sure. You should now get full points for the lab.
This ends the lab. Make sure you pass all of the make grade tests and don't forget to write up your answers to the questions in answers.txt.Before handing in, use git status and git diff to examine your changes and don't forget to git add answers.txt. When you're ready, commit your changes with git commit, run make turnin.
Last updated: Tue Mar 05 16:29:14 -0600 2013 [validate xhtml]