Proxy Lab: Writing a Caching Web Proxy

Introduction

A proxy server is a computer program that acts as an intermediary between clients making requests to access resources and the servers that satisfy those requests by serving content. A web proxy is a special type of proxy server whose clients are typically web browsers and whose servers are web servers providing web content. When a web browser uses a proxy, it contacts the proxy instead of communicating directly with the web server; the proxy forwards the client’s request to the web server, reads the server’s response, then forwards the response to the client.

Proxies are useful for many purposes. Sometimes, proxies are used in firewalls, so that browsers behind a firewall can only contact a server beyond the firewall via the proxy. A proxy may also perform translations on pages, for example, to make them viewable on web-enabled phones. Importantly, proxies are used as anonymizers: by stripping requests of all identifying information, a proxy can make the browser anonymous to web servers. Proxies can even be used to cache web objects by storing local copies of objects from servers and then responding to future requests by reading them out of its cache rather than by communicating again with a remote server. This lab has three parts. An implementation of the first part will be submitted as your checkpoint. Your final submission will then incorporate the extensions forming the second and third parts. For the first part, you will create a proxy that accepts incoming connections, reads and parses requests, forwards requests to web servers, reads the servers’ responses, and forwards the responses to the corresponding clients. The first part will involve learning about basic HTTP operation and how to use sockets to write programs that communicate over network connections. In the second part, you will upgrade your proxy to deal with multiple concurrent connections. This will introduce you to dealing with concurrency, a crucial systems concept. In the third and last part, you will add caching to your proxy using a simple main memory cache of recently accessed web content.

Handout Instructions

To get your lab materials, click ”Download Handout” on Autolab. Clone your repository on a Shark machine by running:

linux> git clone https://github.com/18-x13/proxy-lab-yourgithubid

3.1 Robust I/O package

The handout directory contains the files csapp.c and csapp.h, which comprise the CS:APP package discussed in the CS:APP3e textbook. The CS:APP package includes the robust I/O (RIO) package. When reading and writing socket data, you should use the RIO package instead of low-level I/O functions, such as read, write, or standard I/O functions, such as fread, and fwrite.

The CS:APP package also contains a collection of wrapper functions for system calls that check the return code and exit when there’s an error. You will find that the set of wrapper functions provided is a subset of those from the textbook and the lecture notes. We have disabled ones for which exiting upon error is not the correct behavior for a server program. For these, you must check the return code and devise ways to handle these errors that minimize their impact.

3.2 HTTP parsing library

The handout directory contains the file http parser.h, which defines the API for a small HTTP string parsing library. The library includes functions for extracting important data fields from HTTP response headers and storing them in a parser t struct. A brief overview of the library is given below. Please refer to the source files in your handout for the full documentation of the types, structs, and functions available for use in the library. To create a new instance of a parser struct, call parser new(). The returned pointer can then be used as the first argument to the other functions. parser parse line() will parse a line of an HTTP request and store the result in the provided parser t struct. Parsed fields of specified types may be retrieved from the struct by calling parser retrieve() and by providing a string pointer for the function to write to. Particular headers may also be retrieved by name via parser lookup header(). Headers may instead be accessed in an iterative fashion by successive calls to parser retrieve next header().

3.3 Modularity

The skeleton file proxy.c, provided in the handout, contains a main function that does practically nothing. You should fill in that file with your proxy implementation. Modularity, though, should be an important consideration, and it is important for you to separate the individual modules of your implementation into different files. For example, your cache should be largely (or completely) decoupled from the rest of your proxy, so one good idea is to move the implementation of the cache into separate code and header files cache.c and cache.h.

POSIX Threads

Production web proxies usually do not process requests sequentially; they process multiple requests in parallel. This is particularly important when handling a single request can involve a lengthy delay (as it might when contacting a remote web server). While your proxy waits for a response from the remote web server, it can work on a pending request from another client. Indeed, most web browsers reduce latency by issuing concurrent requests for the multiple URLs embedded in a single web page requested by a single client. Once you have a working sequential proxy, you should alter it to simultaneously handle multiple requests.

5.1 POSIX Threads

You will use the POSIX Threads (Pthreads) library to spawn threads that will execute in parallel to serve multiple simultaneous requests. A simple way to serve concurrent requests is to spawn a new thread to process each request. In this architecture, the main server thread simply accepts connections and spawns off independent worker threads that deal with each request to completion and terminate when they are done. Other designs are also viable: you might alternatively decide to have your proxy create a pool of worker threads from the start. You may use any architecture you wish as long as your proxy exhibits true concurrency, but spawning a new worker thread for each request is the most straightforward approach. A new thread is spawned in Pthreads by calling pthread create with a start routine as its argument. New

threads are by default joinable, which means that another thread must clean up spare resources after the thread exits, similar to how an exited process must be reaped by a call to wait. For a server, a better approach is to detach each new thread, so that spare resources are automatically reaped upon thread exit. To properly detach threads, the first line of the start routine should be as follows: pthread_detach(pthread_self());

5.2 Race conditions

While multithreading will almost certainly improve the performance of your web proxy, concurrency comes at a price: the threat of race conditions. Race conditions most often arise when there is a shared resource between multiple threads. You must find ways to avoid race conditions in your concurrent proxy. That will likely involve both minimizing shared resources and synchronizing access to shared resources. Synchronization involves the use of locks, which come in many varieties. The Pthreads library implements the locking primitives you will need, as is discussed in Section 6.2.

5.3 Thread safety

The open clientfd and open listenfd functions described in the CS:APP3e textbook (and included in csapp.c) are based on the modern and protocol-independent getaddrinfo function, and thus are thread safe.