1. Unexpected Protocol Implications Elliot Jaffe with Dror Feitelson, Scott Kirkpatrick Networking Seminar Artzi 2008

2. Networking Seminar Artzi 2008 May 29, 2008 Data Set Internet Archive Media files http://www.archive.org Create Commons, Unrestricted files.75 PB of files 5 GB/sec network connection 20-60M hits per day

3. Networking Seminar Artzi 2008 May 29, 2008 Observation

4. Networking Seminar Artzi 2008 May 29, 2008 What Happened? IA Policy is to retain all material even if it is copyrighted IA Policy is to deny access to restricted material The items were marked 0000 in the file system: No access allowed The web server returned a 403

5. Networking Seminar Artzi 2008 May 29, 2008 Users per day Number of users has not changed When people get a 403, they retry multiple times

6. Networking Seminar Artzi 2008 May 29, 2008 Top URLs

7. Networking Seminar Artzi 2008 May 29, 2008 Sample Activity Initial days of normal download (200, 206) Sometime on day 22, it was turned off Not to scale: 10.4M Forbidden hits A few forbidden hits on later days

8. Networking Seminar Artzi 2008 May 29, 2008 403: Forbidden The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. : RFC 2616 section 10 Someone forgot to tell the users!

9. Networking Seminar Artzi 2008 May 29, 2008 Implications Avg HTTP request uses.06 ms of CPU time 30M requests = 2.7 CPU hours per day That is one computer 80% idle IA uses more than 1000 computers

10. Networking Seminar Artzi 2008 May 29, 2008 Implications Average size of 403 return: 425 bytes 30M 403 returns = 1.2 GB of data IA delivers 300TB of data per day

11. Networking Seminar Artzi 2008 May 29, 2008 Conclusions Browsers should obey the RFC and not retry 403 failures Huge numbers of bad events are not necessarily indicative of a problem The cost of failure may be below the critical threshold