• Home
• Schedule
• Policies and grading
• Labs
• Reference materials
• Announcements

Released Tuesday, February 23, 2016
Part A (Sections 0 and 1) due Wednesday, March 2, 2016, 10:00 PM
Part B (Sections 2 and 3) due Friday, March 11, 2016, 8:00 PM
Part C (Section 4) due Friday, March 25, 2016, 9:00 PM

Lab 3: Privilege separation and server-side sandboxing

Introduction

This lab will introduce you to privilege separation and server-side sandboxing. The lab continues with the zoobar sequence from MIT's 6.858, which extend assignments developed in Stanford's CS155.

The context for this lab is a simple python web application called zoobar, where users transfer "zoobars" (credits) between each other. The main goal of privilege separation is to ensure that if an adversary compromises one part of an application, the adversary doesn't compromise the other parts too. To help you privilege-separate this application, the zookws web server used in the previous lab follows the design of the OKWS web server, discussed in class. In this lab, you will set up a privilege-separated web server and break up the application code into less-privileged components to contain the effects of vulnerabilities.

You will also (possibly for extra credit; we haven't decided whether this part is required) extend the Zoobar web application to support executable profiles: Python code, uploaded by users, that displays a page to other users. Letting users decide what code runs in the context of the Web server sounds dangerous; the work of the lab is making this arrangement safe(r). We describe the details in part 4, below.

[UPDATE] Parts B and C of this assignment may be completed pair programming-style. Pair programming means that both of you should be at the screen, together. Take turns with the keyboard, passing it back and forth. Only one member of each pair should submit. Do not begin collaborating until you have submitted part A.

To fetch the new source code, connect to the virtual machine, either using ssh or by directly logging on to the shell after starting the virtual machine, navigate to the ~/labs directory, and use git to checkout the lab3 branch from our repo.

$ cd ~/labs
$ git pull
....
$ git checkout -b lab3 origin/lab3
$ _

You'll then need to patch flask in order to get it to work with the lab:

$ sudo make fix-flask
password for httpd: cs480
./fix-flask.sh
patching file /usr/lib/python2.7/dist-packages/werkzeug/routing.py
Done
$ _

Once your source code is in place, make sure that you can compile and install the web server and the zoobar application:

$ cd ~/labs/lab3
$ make
cc -m32 -g -std=c99 -fno-stack-protector -Wall -Werror -D_GNU_SOURCE   -c -o zookld.o zookld.c
cc -m32 -g -std=c99 -fno-stack-protector -Wall -Werror -D_GNU_SOURCE   -c -o http.o http.c
cc -m32  zookld.o http.o  -lcrypto -o zookld
cc -m32 -g -std=c99 -fno-stack-protector -Wall -Werror -D_GNU_SOURCE   -c -o zookfs.o zookfs.c
cc -m32  zookfs.o http.o  -lcrypto -o zookfs
cc -m32 -g -std=c99 -fno-stack-protector -Wall -Werror -D_GNU_SOURCE   -c -o zookd.o zookd.c
cc -m32  zookd.o http.o  -lcrypto -o zookd
cc -m32 -g -std=c99 -fno-stack-protector -Wall -Werror -D_GNU_SOURCE   -c -o zooksvc.o zooksvc.c
cc -m32  zooksvc.o  -lcrypto -o zooksvc
$ sudo make setup
./chroot-setup.sh
+ grep -qv uid=0
+ id
...
+ python /jail/zoobar/zoodb.py init-person
+ python /jail/zoobar/zoodb.py init-transfer
$ _

Section 0: Understand the application

Let's begin by looking at some of the Web app code (as distinct from the code that implements the Web server).

One of the killer features—we are talking major disruption here—of the zoobar application is the ability to transfer credits between users. This feature is implemented by the transfer.py script which lives in lab3/zoobar.

To get a sense of what transfer does, start the zoobar Web site:

$ cd ~/labs/lab3
$ sudo make setup
[sudo] password for httpd: 
./chroot-setup.sh
+ grep -qv uid=0
+ id
...
+ python /jail/zoobar/zoodb.py init-person
+ python /jail/zoobar/zoodb.py init-transfer
$ sudo ./zookld zook.conf
zookld: Listening on port 8080
zookld: Launching zookd
...

Now, make sure you can run the web server and access the web site from your browser as usual. Check the IP address of your virtual machine:

$ /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 08:00:27:19:99:1a  
          inet addr:192.168.56.101  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe19:991a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:757 errors:0 dropped:0 overruns:0 frame:0
          TX packets:592 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:65439 (65.4 KB)  TX bytes:77035 (77.0 KB)

In this particular example, you would want to open your browser and go to http://192.168.56.101:8080/zoobar/index.cgi. You should see the zoobar web site.

Privilege separation

The web server for this lab uses the /jail directory to setup chroot jails for different parts of the web server, much as in the OKWS paper. The make command compiles the web server, and make setup installs it with all of the necessary permissions in the /jail directory.

As part of this lab, you will need to change how the files and directories are installed, such as changing their owner or permissions. To do this, you should not change the permissions directly. Instead, you should edit the chroot-setup.sh script in the ~/labs/lab3 directory, and re-run sudo make setup. If you change the permissions in a different script, your server might not work.

Two aspects make privilege separation challenging in the real world and in this lab. First, privilege separation requires that you take apart the application and split it up in separate pieces. Although we have tried to structure the application so that it is straightforward to split, there are places where you must redesign certain parts. Second, you must ensure that each piece runs with minimal privileges, which requires setting permissions precisely and configuring the pieces correctly. Ideally, by the end of this lab, you'll have a solid understanding of why many applications have security vulnerabilities related to failure to properly separate privileges: proper privilege separation is hard!

One problem that you might run into is that it's tricky to debug a complex application that's composed of many pieces. To help you, we have provided a simple debug library in debug.py, which is imported by every Python script we give you. The debug library provides a single function, log(msg), which prints the message msg to stderr (which should go to the terminal where you ran zookld), along with a stack trace of where the log function was called from.

If something doesn't seem to be working, try to figure out what went wrong, before proceeding further. If you cannot find the issue, contact the course staff.

Section 1: Privilege-separate the web server setup using Unix principals and permissions

As noted earlier, the zookws web server is modeled after OKWS. Similar to OKWS, zookws consists of a launcher daemon, zookld, that launches services configured in the file zook.conf; a dispatcher, zookd, that routes requests to corresponding services; and several services. For simplicity, zookws does not implement analogs of OKWS's pubd or logd.

The file zook.conf is the configuration file that specifies how each service should run. For example, the zookd entry:

[zookd]
    cmd = zookd
    uid = 0
    gid = 0
    dir = /jail

specifies that the command to run zookd is zookd and that it runs with user and group ID 0 (which is the superuser root) in the jail directory /jail.

The zook.conf file configures only one HTTP service, zookfs_svc, that serves static files and executes dynamic scripts. The zookfs_svc does so by invoking the executable zookfs, which should be jailed in the directory /jail by chroot. You can look into /jail: it contains executables (except for zookld), supporting libraries, and the zoobar web site. See zook.conf and zookfs.c for details.

The launcher daemon, zookld, which reads zook.conf and sets up all services, is running as root and can bind to a privileged port like 80. Note that in the default configuration, zookd and the services are inappropriately running under root and are not "jailed"; an attacker who exploits buffer overflows can cause damage to the entire server (for example, the link/unlink attack that you mounted in lab 2).

To fix the problem, you should run these services under unprivileged users rather than root. You will modify zookld.c and zook.conf to set up user IDs, group IDs, and chroot for each service. This will proceed in a few steps: first you will modify zookld.c to support chroot, second you will modify zookld.c to support user and group IDs other than root, and finally you will modify zook.conf to use this support.

As you work through the exercises below, "sudo make check" will check them; however, two disclaimers are in order. First, these tests are not exhaustive! Second, it is often more convenient to test by yourself rather than relying on make check. Nevertheless, as you plan your implementation for each of the exercises below, you may want to read over the test cases invoked by make check. Most of the log files produced by the test scripts will be saved in the /tmp directory.

set_perms 1234:5678 755 /path/to/file/or/directory

Now that none of the services are running as root, we will try to further privilege-separate the zookfs_svc service that handles both static files and dynamic scripts. The first reason to do this is that some Python scripts could easily have security holes; if we do not separate the static and dynamic files, then a vulnerable Python script could be tricked into deleting static files that the server is serving. Conversely, the static file-serving code might be tricked into serving up the databases used by the Python scripts, such as person.db and transfer.db. A better organization is to split zookfs_svc into two services, one for static files and the other for Python scripts, running as different users.

[safe_svc]
    cmd = zookfs
    uid = 0
    gid = 0
    dir = /jail
    args = 123 456

Handing in the lab (Part A)

Here is a checklist for before you submit:

• Please note how long this part of the lab took you at the bottom of your answers.txt file.
• Add any extra files you've created by doing:

  $ git add <new file 1> <new file 2> ... 
  

• You are now ready to submit. From within the ~/labs repo, do:

  $ git commit -am "My submission for lab 3 part A"
  ...
  $ git push -u origin lab3
  
  # Prompts for username and password
  
  Counting objects: ...
  ....
  $ _
  

• If you submit multiple times, we will only look at your latest submission and count slack hours accordingly.

Interlude: RPC library

The previous part privilege-separated the components of zookws. In this part, you will privilege-separate the zoobar application itself by placing different components in different processes. This way, if one piece of the zoobar application has an exploitable bug, an attacker cannot use the bug to break into other parts of the zoobar application.

A challenge in splitting the zoobar application into several processes is that the processes need to be able to communicate. You will first study a Remote Procedure Call (RPC) library that allows processes to communicate over a Unix socket. Then, you will use that library to separate zoobar into several processes that communicate using RPC.

To illustrate how our RPC library might be used, we have implemented a simple "echo" service for you in zoobar/echo-server.py. This service is invoked by zookld; look for the echo_svc section of zook.conf to see how it is started.

echo-server.py is implemented by defining an RPC class EchoRpcServer that inherits from RpcServer, which in turn comes from zoobar/rpclib.py. The EchoRpcServer RPC class defines the methods that the server supports, and rpclib invokes those methods when a client sends a request. The server defines a simple method that echoes the request from a client.

echo-server.py starts the server by calling run_sockpath_fork(sockpath). This function listens on a UNIX-domain socket. The socket name comes from the argument, which in this case is /echosvc/sock (specified in zook.conf). When a client connects to this socket, the function forks the current process. One copy of the process receives messages and responds on the just-opened connection, while the other process listens for other clients that might connect to the socket.

We have also included a simple client of this echo service as part of the Zoobar web application. In particular, when you direct your Web browser to the URL /zoobar/index.cgi/echo?s=hello, the request is routed to zoobar/echo.py. That code uses the RPC client (implemented by rpclib) to connect to the echo service at /echosvc/sock and invoke the echo operation. Once it receives the response from the echo service, it returns a web page containing the echoed response.

The RPC client-side code in rpclib is implemented by the call method of the RpcClient class. This method formats the arguments into a string, writes the string on the connection to the server, and waits for a response (a string). On receiving the response, call parses the string, and returns the results to the caller.

Section 2: Privilege-separating the login service in Zoobar

We will now use the RPC library to improve the security of the user passwords stored in the Zoobar web application. Right now, an adversary that exploits a vulnerability in any part of the Zoobar application can obtain all user passwords from the Person database.

The first step towards protecting passwords will be to separate the code that deals with user authentication data (meaning passwords and tokens) from the rest of the application. The current zoobar application stores everything about the user (profile, zoobar balance, and authentication info) in the Person table (see zoodb.py). We want to move the authentication info out of the Person table into a separate Cred table (Cred stands for Credentials), and move the code that accesses this authentication information (i.e., auth.py) into a separate service.

The reason for splitting the tables is that the tables are stored in the file system in zoobar/db/, and are accessible to all Python code in Zoobar. This means that an attacker might be able to access and modify any of these tables, and we might never find out about the attack. Once the authentication data is split out into its own database, however, we can set Unix file and directory permissions such that only the authentication service—and not the rest of Zoobar—can access that information.

Specifically, your job will be as follows:

• Decide what interface your authentication service should provide (i.e., what functions it will run for clients). Look at the code in login.py and auth.py, and decide what needs to run in the authentication service, and what can run in the client (i.e., be part of the rest of the zoobar code). Keep in mind that your goal is to protect both passwords and tokens. We have provided initial RPC stubs for the client in the file zoobar/auth_client.py.
• Create a new auth_svc service for user authentication, along the lines of echo-server.py. We have provided an initial file for you, zoobar/auth-server.py, which you should modify for this purpose. The implementation of this service should use the existing functions in auth.py.
• Modify zook.conf to start the auth-server appropriately (under a different UID).
• Split the user credentials (i.e., passwords and tokens) from the Person database into a separate Cred database, stored in /zoobar/db/cred. Don't keep any passwords or tokens in the old Person database.
• Modify chroot-setup.sh to set permissions on the Cred database appropriately, and to create the socket for the auth service.
• Modify the login code in login.py to invoke your auth service instead of calling auth.py directly.

def f(x,y,z):
  return z - x*x + y*z - x

b = {"x":4, "y":5, "z":5}

# The following two lines do the same
# thing. The first uses the "standard"
# approach whereas the second uses
# unpacking of the dictionary 'b'.

f(4,5,5)  # Standard way to call f
f(**b)    # Unpacking b as f's arguments

We can further improve the security of passwords by using hashing and salting. The current authentication code stores an exact copy of the user's password in the database. Thus, if an adversary somehow gains access to the cred.db file, all of the user passwords will be immediately compromised. Worse yet, if users have the same password on multiple sites, the adversary will be able to compromise users' accounts there too!

Hashing protects against this attack by storing a hash of the user's password (i.e., the result of applying a hash function to the password) instead of the password itself. If the hash function is difficult to invert (i.e., is a cryptographically secure hash), an adversary will not be able to directly obtain the user's password. The server can still check if a user supplied the correct password during login, though: it will just hash the user's password, and check if the resulting hash value is the same as was previously stored.

One weakness with hashing is that an adversary can build up a giant table (called a "rainbow table"), containing the hashes of all likely passwords. Then, if an adversary obtains someone's hashed password, the adversary can just look it up in its giant table to obtain the original password.

To defeat the rainbow table attack, most systems use salting. With salting, instead of storing a hash of the password, the server stores a hash of the password concatenated with a randomly-generated string (called a salt). To check if the password is correct, the server concatenates the user-supplied password with the salt and checks if the result matches the stored hash. Note that to make this work, the server must store the salt value used to originally compute the salted hash! However, because of the salt, the adversary would now have to generate a separate rainbow table for every possible salt value. This greatly increases the amount of work the adversary has to perform in order to guess user passwords based on the hashes.

A final consideration is the choice of hash function. Most hash functions, such as MD5 and SHA1, are designed to be fast. This means that an adversary can try lots of passwords in a short period of time, which is not what we want! Instead, you should use a special hash-like function that is explicitly designed to be slow. A good example of such a hash function is PBKDF2, which stands for Password-Based Key Derivation Function (version 2).

A surprising side-effect of using a very computationally expensive hash function like PBKDF2 is that an adversary can now use this to launch denial-of-service (DoS) attacks on the server's CPU. For example, the popular Django web framework recently posted a security advisory about this, pointing out that if an adversary tries to log in to some account by supplying a very large password (1MB in size), the server would spend an entire minute trying to compute PBKDF2 on that password. Django's solution is to limit supplied passwords to at most 4KB in size. For this lab, we do not require you to handle such DoS attacks.

Section 3: Privilege-separating the bank in Zoobar

Finally, we want to protect the zoobar balance of each user from adversaries that might exploit some bug in the Zoobar application. Currently, if an adversary exploits a bug in the main Zoobar application, they can steal anyone else's zoobars, and this would not even show up in the Transfer database if we wanted to audit things later.

To improve the security of zoobar balances, our plan is similar to what you did above in the authentication service: split the zoobar balance information into a separate Bank database, and set up a bank_svc service, whose job it is to perform operations on the new Bank database and the existing Transfer database. As long as only the bank_svc service can modify the Bank and Transfer databases, bugs in the rest of the Zoobar application should not give an adversary the ability to modify zoobar balances, and will ensure that all transfers are correctly logged for future audits.

Finally, we need to fix one more problem with the bank service. In particular, an adversary that can access the bank's service (i.e., can send it RPC requests) can perform transfers from anyone's account to their own. For example, it can steal 1 zoobar from any victim simply by issuing a transfer(victim, adversary, 1) RPC request. The problem is that the bank service has no idea who is invoking the transfer operation. Some RPC libraries provide authentication, but our RPC library is quite simple, so we have to add it explicitly.

To authenticate the caller of the transfer operation, we will require the caller to supply an extra token argument, which should be a valid token for the sender. The bank service should reject transfers if the token is invalid.

Handing in the lab (Part B)

Here is a checklist for before you submit:

• Note how long this part of the lab took you, also in the answers.txt file.
• Add any extra files you've created by doing:

  $ git add <new file 1> <new file 2> ... 
  

• You are now ready to submit. From ~/labs/lab3, do:

  $ git commit -am "My submission for lab 3 part B"
  ...
  $ git push -u origin lab3
  
  # Prompts for username and password
  
  Counting objects: ...
  ....
  $ _
  

• If you submit multiple times, we will only look at your latest submission and count slack hours accordingly.

Section 4: Server-side sandboxing for executable profiles

As stated at the outset of the lab, a user can upload Python code to the server; the server runs that code when any user (including the user who uploaded the profile) selects that profile to view.

To make a profile, a user saves a Python program in their profile on their Zoobar home page. (To indicate that the profile contains Python code, the first line must be #!python.) An example profile is one that keeps track of the last several visitors to that profile.

Supporting this safely requires sandboxing the profile code on the server. This means preventing the code from performing arbitrary operations. What makes this challenging is that the profile code may wish to interact with the environment, for example to read and write files (as in the visitor-tracking example). So the system somehow needs to allow profiles to get their work done, while preventing them from interfering with other profiles or the rest of the system. Sound familiar? Indeed, for this purpose, you will use similar ideas and mechanisms to those you used in the earlier parts: the RPC library, file permissions, etc.

At this point, familiarize yourself with the following components:

• First, the profiles/ directory contains several example profiles. These are executable Python scripts, which you will use as examples throughout this part of the lab:

  • profiles/hello-user.py: prints back the name of the visitor when the profile code is executed, along with the current time.

  • profiles/visit-tracker.py: keeps track of the last time that each visitor looked at the profile, and prints out the last visit time (if any).

  • profiles/last-visits.py: records the last three visitors to the profile, and prints them out.

  • profiles/xfer-tracker.py: prints out the last zoobar transfer between the profile owner and the visitor.

  • profiles/granter.py: gives the visitor one zoobar. To make sure visitors can't quickly steal all zoobars from a user, this profile grants a zoobar only if the profile owner has some zoobars left, the visitor has less than 20 zoobars, and it has been at least a minute since the last time that visitor got a zoobar from this profile.

• Second, zoobar/profile-server.py: an RPC server that accepts requests to run some user's profile code, and returns the output from executing that code.

  This server uses sandboxlib.py (described below) to create a Sandbox and execute the profile code in it (via the run_profile function). profile-server.py also sets up another RPC server (besides itself) that allows the profile code to get access to things outside of the sandbox, such as the zoobar balances of different users. The ProfileAPIServer implements this interface; profile-server.py forks off a separate process to run the ProfileAPIServer, and also passes an RPC client connected to this server to the sandboxed profile code.

  Because profile-server.py uses sandboxlib.py, which in turn needs to call setresuid to sandbox some process, the main profile-server.py process needs to run as root. As an aside, this is a somewhat ironic limitation of Unix mechanisms: if you want to improve your security by running untrusted code with a different user ID, you are forced to run some part of your code as root, which is a dangerous thing to do from a security perspective.

• Finally, zoobar/sandboxlib.py is a Python module that implements sandboxing for untrusted Python profile code; see the Sandbox class, and the run() method which executes a specified function in the sandbox. The run method works by forking off a separate process and calling setresuid in the child process before executing the untrusted code, so that the untrusted code does not have any privileges; this is a key component of the sandboxing approach. The parent process reads the output from the unprivileged child process (which is now running the untrusted profile code) and returns this output to the caller of run() (the direct caller is profile-server.py; the indirect caller is the RPC client of the profile server). If the (sandboxed) child doesn't exit after a short timeout (5 seconds by default), the parent process kills the child.

  Another key aspect of the sandboxing is our friend chroot: Sandbox.run() uses this system call to jail the untrusted code, passed as an argument to the Sandbox constructor. This allows the untrusted profile code to perform some limited file system access, but the creator of Sandbox gets to decide what directory is accessible to the profile code.

  Sandbox uses just one user ID for running untrusted profiles. This means that it's important that at most one profile be executing in the sandbox at a time. Otherwise, one sandboxed process could tamper with another sandboxed process, since they both have the same user ID! To enforce this guarantee, Sandbox uses a lockfile; whenever it tries to run a sandboxed process, it first locks the lockfile, and releases it only after the sandboxed process has exited. If two processes try to run some sandboxed code at the same time, only one will get the lockfile at a time. It's important that all users of Sandbox specify the same lockfile name if they use the same UID.

  How does Sandbox know that some sandboxed code has fully exited and it's safe to reuse the user ID to run a different user's profile? After all, the untrusted code could have forked off another process. To prevent this, Sandbox uses Unix's resource limits to prevent sandboxed code from forking. Specifically, the sandbox uses setrlimit, which limits the number of processes with a given user ID; the sandbox sets this limit to 0. As a consequence, after the parent process kills the child process (or notices that the child has exited), the parent knows that there are no remaining processes with that user ID.

Python profiles with privilege separation

To get started, you will need to add profile-server.py to your zook.conf and modify chroot-setup.sh to create a directory for its socket, /jail/profilesvc. Remember that profile-server.py needs to run as root, so put 0 for the uid in its zook.conf entry.

The next problem we need to solve is that some of the user profiles store data in files. (See last-visits.py and visit-tracker.py.) All of the user profiles currently run with access to the same files, because ProfileServer.rpc_run() sets userdir to /tmp and passes that as the directory to Sandbox (which it turn chroots the profile code to that directory). As a result, one user's profile can corrupt the files stored by another user's profile.

Finally, recall that all of profile-server.py currently runs as root because it needs to create a sandbox. This is dangerous, and we would like to reduce the amount of code in profile-server.py that runs as root. In particular, the ProfileAPIServer that runs as part of profile-server.py does not strictly need to run as root (it does not invoke the sandbox), and in fact, it might be the most vulnerable part of the code to attacks, because it accepts RPC commands from the untrusted profile code!

You are now done with the basic sandbox.

Handing in the lab (Part C)

Here is a checklist for before you submit:

• Please note how long this part of the lab took you at the bottom of your answers.txt file.
• Add any extra files you've created by doing:

  $ git add <new file 1> <new file 2> ... 
  

• You are now ready to submit. From within the ~/labs repo, do:

  $ git commit -am "My submission for lab 3 part C"
  ...
  $ git push -u origin lab3
  
  # Prompts for username and password
  
  Counting objects: ...
  ....
  $ _
  

• If you submit multiple times, we will only look at your latest submission and count slack hours accordingly.