1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
|
## Proxy
### Programming Assignment 3 - HTTP Caching Proxy
In this assignment, you will build a proxy server that is capable of relaying HTTP
requests from clients to HTTP servers. With the HTTP client-server protocol, the client
usually connects directly to an HTTP server. Oftentimes it is useful to introduce an
intermediate entity called a proxy. With a proxy, the browser instead sends an HTTP
request to the proxy first. The proxy then forwards this request to the intended HTTP
server, and then receives the reply from the HTTP server. The proxy finally forwards
this reply to the requesting HTTP client. There are several advantages to utilising a
proxy:
* **Performance**: If we cache a page that is frequently accessed by clients connected
to the proxy, we can reduce the need to create new connections to the origin HTTP server
for every request.
* **Content Filtering**: The proxy can inspect URLs from the HTTP client and decide
whether to block connections to some specific URLs. Or to reform webpages (eg. image
transcoding for clients with limited processing capability).
* **Privacy & Security**: When the origin HTTP server tries to log IP address
information from the HTTP client, it can only get the information of the proxy but not
the actual client. Forcing all web traffic through a proxy "checkpoint" can help network
administrators centralise the detection of viruses and other threats.
### The Proxy Server
The program must take one command line argument; a port number for the server to use:
```shell
./proxy 8080 # running proxy with port number 8080
```
Your proxy must be able to handle multiple requests at the same time, such that multiple
clients can access different servers with your proxy simultaneously. As with
[PA2](../pa2_webserver), you may achieve this requirement through either a forking or
multi-threaded implementation.
Your proxy should run in an infinite loop once it has started. You are expected to exit
out of the proxy gracefully when pressing Ctrl + C. When an HTTP request comes in, your
application should determine whether this request is properly formatted with the
following criteria:
* The HTTP method is valid. Our web proxy only needs to support `GET`
* The HTTP server specified in the requested URL exists (usually by calling
`gethostbyname()`)
If the above criteria are not met, a corresponding error message will be sent to the
HTTP client (eg. `400 Bad Request`). Once a valid request is received, the proxy will
parse the requested URL into the following 3 parts:
1. Requested host and port number (you should use the default port 80 if no port number
is specified)
2. Requested path (used to access resources at the HTTP server)
3. Optional message body (this may not appear in every HTTP request)
If the requested host is not resolved to any IP address, then the error message
`404 Not Found` must be returned to the client. You can do this right after a
`gethostbyname()` call as this will return an IP address only for valid hostnames.
The request received by the proxy will be identical to that received by an HTTP server.
fif we were set to our browser to use a proxy running on port 8000 and capture the `GET`
request to [www.yahoo.com](www.yahoo.com). it would look something like the following:
```shell
$ nc -l 8000
GET http://www.yahoo.com/ HTTP/1.1
Host: www.yahoo.com
Proxy-Connection: keep-alive
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36 DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: B=fgvem7ha9r7hv&b=3&s=0o; ucs=sfcTs=1425977267&sfc=1
```
### Forwarding requests to the HTTP server and relaying data
After the proxy has parsed the information from the client, it constructs a new HTTP
request and sends it over a second socket connection to the indicated HTTP server. The
proxy will then forward the reply from the HTTP server back to the client. So you will
need to manage two concurrent socket socket connections - one from the client to the
proxy, and one from the proxy to the server. In summary:
* Your proxy listens on the specified port number and accepts a connection
* You will then parse the request received from this connection and check if the request
is properly formatted
* If properly formatted, your proxy will create a second socket and send a request to
the specified HTTP server
* Your proxy will then relay the results from the server to the client
### Blocklist
Create a file called `./blocklist` which has a list of websites and/or IP addresses that
must be blocked. Thus, for every incoming request, the proxy must look up the requested
host in `./blocklist` and deny requests for these websites or IPs. When a client
requests one of them, your proxy should return `403 Forbidden`. This file should be
formatted with one hostname or IP address per line:
```
www.google.com
mail.yahoo.com
154.240.28.35
```
### Caching
Caching is one of the most common performance enhancements that web proxies implement.
Caching takes advantage of the fact that most pages on the web don't change that often,
and that any page that you visit once, you (or someone else using the same proxy) are
likely to visit again. A caching proxy server saves a copy of the files that it
retrieves from remote servers. When another request comes in for the same resource, it
returns the saved copy instead of creating a new connection to a remote server. This
can create more significant savings in the case of a more distant server or a remote
server that is overloaded.
Caching introduces a few new complexities as well. First of all, most web content is
dynamically generated, and as such shouldn't really be cached. Second, we need to
decide how long to keep pages in our cache. If the timeout is set too short, we negate
most of the advantages of having a caching proxy. If the timeout is set too long, the
client may end up looking at pages that are outdated or irrelevant.
There are a few considerations that will affect the caching behaviour of web proxy:
1. **Timeout setting**: Alter your proxy so that you can specify a timeout value on the
command line as a runtime option. For example, `./proxy 8080 60` will run the proxy
with a timeout value of 60 seconds.
2. **Page Cache**: You'll need to alter how your proxy retrieves pages pages. You
should now check to see if a page exists in the cache before retrieving a page from a
remote server. If there is a valid cached copy of the page, that should be presented to
the client instead of creating a new server connection. You will need to create local
files to store the cached pages.
3. **Expiration**: You will need to implement cache expiration. The timing does not
need to be exact (ie. it's okay if a page is still in your cache after the timeout has
expired, but it's not okay to serve an expired page from the cache). To keep things
simple, we are not handling the cache-control header, which is sent by the server and
specifies how long the client or the proxy should cache the content. Instead, we will
use the timeout specified by the command line argument.
4. **Dynamic content**: Many webpages have parameters embedded in the URL to specify
that a page be dynamically generated. You must consider these parameters when
determining whether or not to server a previously cached page. You'll most likely
want to not cache dynamic pages.
Tip: when you check to see if a specific URL exists in the cache, you might want to
compare hashes instead of the entire URL. For example, suppose that you store
[http://www.yahoo.com/logo.jpg](http://www.yahoo.com/logo.jpg) with the hash key
`0xAC10DE97073ACD81` using a hash function like `md5sum` of the URL. When you receive
the same URL from the client, you can simply calculate the hash value of the requested
URL and compare it with the hash keys stored in the cache. Keep cached files in a
`./cache` directory.
### Link Prefetch
Building on top of your proxy and caching code, the last piece of functionality
desired is called link prefetching. The idea behind prefetching is simple: if a user
asks for a particular page, the odds are that they will next request a page linked from
that page. Prefetching uses this information to attempt to speed up browsing by parsing
requested pages for links, and then fetching the linked pages in the background. The
pages fetched from the links are stored in the cache, ready to be served to the client
when they are requested without the client having to wait around for the remote server
to be contacted.
Parsing and fetching links can take a significant amount of time, especially for a page
with a lot of links. In order to fetch multiple links simultaneously in background, you
have to use multi-threading or forking. One thread/process should remain dedicated to
the tasks that you have already implemented: reading requests from the client and
serving pages from either the cache or a remote server. In a separate thread/process,
the proxy will parse a page and extra the HTTP links, request those links from the
remote server, and add them to the cache.
|