CS439 Spring 2013 Lab 8: Web Proxy

Handed out Saturday, April 20, 2013
Part A due ~~Friday, April 26~~ Monday, April 29, 2013, 11:59 PM
Part B due ~~Friday, May 3~~ Monday, May 6, 2013, 11:59 PM
UPDATE: NOTE THAT LATE PART Bs CANNOT BE ACCEPTED.

Introduction

In this lab, you will write a concurrent Web proxy that logs requests. In the first part of the lab, you will write a simple sequential proxy that repeatedly waits for a request, forwards the request to the end server, and returns the result back to the browser, keeping a log of such requests in a disk file. This part will help you understand basics about network programming and the HTTP protocol.

In the second part of the lab, you will upgrade your proxy so that it uses threads to deal with multiple clients concurrently. This part will exercise your experience with concurrency and synchronization.

Late policy. Late policy: Part A of the lab will be treated as every other lab has been: you are welcome to use late hours, if you have them, and if you turn in part A late, your grade on that part of the lab decreases each hour (as usual). However, you cannot use late hours for part B, and furthermore, late part Bs will not be accepted (this is an exception to the usual lateness policy). .

Getting Started

Use Git to commit your Lab 7 source, fetch the latest version of the course repository, and then create a local branch called lab8 based on our lab8 branch, origin/lab8:

Lab Requirements

As before, you will need to do all of the regular exercises described in the lab. You do not need to do any of the challenge problems, though we suggest you try to do one if you have the time. The challenge problems here, however, can be done for extra credit. Follow the instructions given in the challenge problems if you implement them so that we, the teaching staff, know to look at and grade them. Note that you can never lose points from attempting to do a challenge problem. We will only check the challenges at the part B turn-in deadline, so you have the full two weeks to implement a challenge if you so choose.

Hand-In Procedure

When you are ready to hand in your lab code, run make turnin-parti where i is the part you want to turnin in the labs directory. This will first do a make clean to clean out any .o files and executables, and then create a tar file called lab8i-handin.tar.gz with the entire contents of your lab directory and submit it via the CS turnin utility. If you submit multiple times, we will take the latest submission and count lateness accordingly.

Background

A Web proxy is a program that acts as a middleman between a Web browser and an end server. Instead of contacting the end server directly to get a Web page, the browser contacts the proxy, which forwards the request on to the end server. When the end server replies to the proxy, the proxy sends the reply on to the browser.

Proxies are used for many purposes. Sometimes proxies are used in firewalls, such that the proxy is the only way for a browser inside the firewall to contact an end server outside. The proxy may do translation on the page, for instance, to make it viewable on a Web-enabled cell phone. Proxies are also used as anonymizers. By stripping a request of all identifying information, a proxy can make the browser anonymous to the end server. Proxies can even be used to cache Web objects, by storing a copy of, say, an image when a request for it is first made, and then serving that image in response to future requests rather than going to the end server. Proxies can also be used to turn the Web upside-down.

Overview of HTTP/1.0

The HyperText Transfer Protocol (HTTP) is a simple protocol for requesting and receiving documents. In version 1.0 of HTTP, every request/response pair gets its own network connection. A request message has four parts:

By default, HTTP servers use port number 80. So, if you receive a request for http://www.utexas.edu/index.html, this means connect to machine www.utexas.edu using port 80 to request document www.utexas.edu/index.html. Sometimes a server will use a different port number. In this case, a port number is added to the end of the URL. For example http://www.utexas.edu/index:3000 would say to use port 3000 on machine www.utexas.edu.

After receiving a request, a server sends a response. Your proxy will just send the response bytes back to the client unmodified, so you don't need to parse the response. A response message looks like this:

Changes in HTTP/1.1

HTTP/1.1 added a significant performance optimization to HTTP. Instead of opening an closing a connection for each request, multiple requests could be pipelined on a single connection: a client sends a series of requests on a connection without waiting for any responses, then the server sends a series of replies (in the same order as the request) to the client.

HTTP/1.1 is supported by all major browsers, so your proxy will see many HTTP/1.1 requests from clients. Fortunately, for backwards compatibility, clients running HTTP/1.1 must be prepared to deal with servers that only know about HTTP/1.0, so your proxy can simply read one request, send it to the server, receive the response, and close the connection. The client will then know to send its next request on a new connection.

Part A: Implementing a Sequential Web Proxy

In this part you will implement a sequential logging proxy. Your proxy should open a socket and listen for a connection request. When it receives a connection request, it should accept the connection, read the HTTP request, and parse it to determine the name of the end server. It should then open a connection to the end server, send it the request, receive the reply, forward the reply to the browser, and close the connection.

Since your proxy is a middleman between client and end server, it will have elements of both. It will act as a server to the web browser, and as a client to the end server. Thus you will get experience with both client and server programming.

Unlike in previous labs, we have left the design of your proxy largely up to you. While you must conform to the multi-threaded coding standards we have given you and while you must write clean and readable C code, we will not generally dictate how you should implement your proxy, only what it must be capable of doing and what format its outputs must be in.

Before going any further, read every word of chapter 12 of Bryant and O'Hallaron. This chapter contains not only explanations of concepts which are extremely relevant to this project, but also provides code snippets which you may find useful to use or model your code after.

Logging Requirements

Your proxy should keep track of all requests in a log file named proxy.log. Each log file entry should be a file of the form:

Note that size is essentially the number of bytes received from the end server, from the time the connection is opened to the time it is closed. Only requests that are met by a response from an end server should be logged. We have provided the function format_log_entry in csapp.c to create a log entry in the required format.

Port Numbers

You proxy should listen for its connection requests on the port number passed in on the command line, like so:

You will develop your sequential proxy in three stages. In the first two stages, you will implement some of the core networking functionality for the proxy to handle HTTP/1.0 communication with remote servers and to accept incoming connections from clients. In the last stage, you will combine the functionality from the first two stages to complete your sequential proxy.

Exercise 1. Write the client-side portion of your proxy. You should be able to connect your proxy to a remote HTTP server, send a HTTP/1.0 GET request for the root web page of a remote HTTP server, and then dump the returned HTTP response to a file (separate from the log file) or to the console. The particular remote server doesn't matter in this case (in the full proxy, this is dependent on what the client to the proxy requests); you can hard-code one for now.

For example, your client should be able to send a GET request to Google's web servers that looks like the following:

GET http://www.google.com/ HTTP/1.0

and then print out the response you get to the console. You can check to see if your code is working by using telnet to emulate what your code is doing (or should be doing) and comparing the output of the two programs. Here's an example of how to test with telnet:

tig% telnet www.google.com 80
Trying 74.125.227.81...
Connected to www.google.com.
Escape character is '^]'.
GET http://www.google.com/ HTTP/1.0

HTTP/1.0 200 OK
Date: Wed, 17 Apr 2013 20:07:32 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
... [more of the header, then a blank line, then a bunch of HTML] ...

Note that you need to press enter twice after typing the GET line.

You may find the following hints helpful as you embark on this part of the project:

A good way to get going is to start with the basic echo client (CS:APP 12.4.9) and tailor or modify it to work in the case of sending a single HTTP GET request and receiving a complete HTTP response.
Use the RIO (Robust I/O) package (CS:APP 10.4, 11.4) for all I/O on sockets. Do not use standard I/O on sockets. You will quickly run into problems if you do. However, standard I/O calls such as fopen and fwrite are fine for I/O on the log file.
The Rio_readn, Rio_readlineb, and Rio_writen error checking wrappers in csapp.c are not appropriate for a realistic proxy because they terminate the process when they encounter an error. Instead, you should write new wrappers called Rio_readn_w, Rio_readlineb_w, and Rio_writen_w that simply return after printing a warning message when I/O fails. When either of the read wrappers detects an error, it should return 0, as though it encountered EOF on the socket.
Reads and writes can fail for a variety of reasons. The most common read failure is an errno = ECONNRESET error caused by reading from a connection that has already been closed by the peer on the other end, typically an overloaded end server. The most common write failure is an errno = EPIPE error caused by writing to a connection that has been closed by its peer on the other end. This can occur for example, when a user hits their browser's Stop button during a long transfer.
Writing to connection that has been closed by the peer first time elicits an error with errno set to EPIPE. Writing to such a connection a second time elicits a SIGPIPE signal whose default action is to terminate the process. To keep your proxy from crashing you can use the SIG_IGN argument to the signal function (CS:APP 8.5.3) to explicitly ignore these SIGPIPE signals.

Exercise 2. Write the server-side code of your proxy. Your proxy should be able to listen for connections on a socket, wait for a client to send a GET request, and then send the client an error message thereafter. Your error message should have the format of an HTTP response, with a bit of HTML for a browser to display a nicely-formatted error message to the user. For the time being, you may disable the client-side code of the proxy to develop the server-side code.

A nicely-formatted error response might look something like this example modified from the IANA website (www.iana.org):

HTTP/1.0 400 Bad Request
Content-Length: 162
Content-Type: text/html

<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not
understand.<br />
</p>
</body></html>

You can test the server-side code of your proxy by using telnet. In one terminal, run ./proxy <port number>, where the port number is any number p in the range 1024 ≤ p ≤ 65536. In another terminal, on the same machine, run telnet localhost <port number>. Once telnet is running, you can submit a GET request (much like you did for testing in exercise 1) to your proxy. Once you submit the GET request, you should see your error message print on the console, and then the connection should close.

You should continue to pay heed to the hints given in exercise 1 with respect to doing I/O on sockets.

Exercise 3. Implement a HTTP/1.0 proxy which can handle a single client at a time and log all requests it recieves. In the previous exercises, you've built up the client and server components of your proxy. Now, you need to fuse those two parts to accept incoming connections from clients, parse GET requests from them, forward those GET requests to the correct remote server, and then siphon the response from that remote server back to the client. If there's an error in processing a request, you can send the client back an error message. Remember to log all requests you get!

Since we want you to focus on network programming issues for this lab, we have provided you with two additional helper routines: parse_uri, which extracts the hostname, path, and port components from a URI, and format_log_entry, which constructs an entry for the log file in the proper format.

Once you've written some more functionality into your proxy, you should test your proxy by pointing a real browser to it. You should make sure that your proxy properly redirects HTTP traffic between you, the end-user, and the remote server you connect to through the proxy. Here are pointers to instructions for how to use a proxy in a variety of modern browsers: Chrome, Firefox, Internet Explorer, and Opera.

Challenge! (15 points) Update your proxy so that if launched with the option "-persistent" it will support HTTP/1.1 persistent connections, pipelining multiple requests on a single TCP connection. Your proxy should continue to log requests as in the cases above.

Note that for grading purposes, if the "-persistent" option is not passed in on the command line, the proxy should accept HTTP/1.0 or HTTP/1.1 requests from clients, but it should only emit HTTP/1.0 requests to servers as described in the earlier part of the lab.

To get credit for this extra credit portion of the lab, you must add a line to the initial comment in the proxy.c file:

* EXTRA CREDIT: HTTP/1.1 PERSISTENT CONNECTIONS IMPLEMENTED AND TESTED

Challenge! (10 points) Update your proxy so that if launched with the option "-upsidedown" it will flip all .jpg, .gif, .png, and other image files downloaded so that the user sees all images as upside-down on their display. (If this option is not passed, then transformation should not happen.)

Hint: check out the man page for the mogrify program, which is available on the CS Linux machines in /usr/bin/mogrify.

Feel free to add additional options so that your proxy can distort the Internet in other amusing ways. How about Pig-Latin-Net, Elmer-Fudd-Net, or 5up4-1337-h@X0r-Net? Perhaps you can modify hyperlinks to point to totally different web pages? Be creative!

To get credit for this extra credit portion of the lab, you must add a line to the initial comment in the proxy.c file:

* EXTRA CREDIT: UPSIDE DOWN INTERNET IMPLEMENTED AND TESTED
*   OTHER SUPPORTED TRANSFORMATIONS: [list command line options here]

This completes part A of the lab. Make sure that any files you have added to the proxy have also been git added and that your proxy builds just by running make. Then make sure you have committed your changes with git commit, then run make turnin-partA.

Part B: Handling Concurrent Requests

Real proxies do not process requests sequentially. They deal with multiple requests concurrently. Once you have a working sequential logging proxy, you should alter it to handle multiple requests concurrently. Your solution should use a thread pool of NTHREADS threads (where NTHREADS has been defined in proxy.c). In particular, you must not be creating threads on-demand to handle requests. This is one design point which you must adhere to, though we otherwise have left the design of the proxy up to you.

With this approach, it is possible for multiple peer threads to access the log file concurrently. Thus, you will need synchronize access to the file such that only one peer thread can modify it at a time. If you do not synchronize the threads, the log file might be corrupted. For instance, one line in the file might begin in the middle of another.

You must follow all multi-threaded coding conventions that are standard in this class. These conventions are codified in Coding Standards for Programming with Threads, by Mike Dahlin. Note that you will have to adopt an object-oriented style of programming with the non-object-oriented C language you are using for this project.

For example, your implementation should create a struct Log representing the log file and its synchronization variables, and you should define a set of functions that operate on that struct. Each of these functions should be named log_<something> and each should take a pointer to a struct Log as its first argument.

Exercise 4. Extend your proxy to handle multiple concurrent requests from many clients, using a pool of NTHREADS threads to handle requests as they arrive. Do not spawn a new thread for every client request. You may find the following hints helpful as you work to extend your proxy:

Be careful about memory leaks. When the processing for an HTTP request fails for any reason, the thread must close all open socket descriptors and free all memory resources before terminating.
You will find it very useful to assign each thread a small unique integer ID (such as the current request number) and then pass this ID as one of the arguments to the thread routine. If you display this ID in each of your debugging output statements, then you can accurately track the activity of each thread.
To avoid a potentially fatal memory leak, your threads should run as detached, not joinable (CS:APP 13.3.6).
Since the log file is being written to by multiple threads, you must ensure mutual exclusion semaphores whenever you write to it. You may use the supplied sthread package or use POSIX pthreads locks and/or condition variables directly.
Be very careful about calling thread-unsafe functions such as inet_ntoa, gethostbyname, and gethostbyaddr inside a thread. In particular, the open_clientfd function in csapp.c is thread-unsafe because it calls gethostbyaddr, a Class-3 thread unsafe function (CS:APP 13.7.1). You will need to write a thread-safe version of open_clientfd, called open_clientfd_ts, that uses the lock-and-copy technique (CS:APP 13.7.1) when it calls gethostbyaddr.

This completes part B of the lab. Make sure that any files you have added to the proxy have also been git added and that your proxy builds just by running make. Then make sure you have committed your changes with git commit, then run make turnin-partB.