1. CIS5930 Internet Computing
HTTP
Prof. Robert van Engelen

2. HyperText Transfer Protocol
HTTP is the World Wide Web protocol
HTTP1.1 is defined by RFC2616
HTTP not just for hypertext markup (HTML) anymore
Carries data in standard formats identified with MIME (Multipurpose Internet Mail Extensions) media types, e.g.:
text/html
image/jpeg
application/x-shockwave-flash
MIME media types are approved and assigned by IANA (Internet Assigned Number Authority)
HTTP message header with ASCII control messages
Optional HTTP message body contains data
2/22/2019
CIS 5930 Fall TCP/IP

3. HyperText over TCP/IP HTTP usually over TCP/IP
Wide area and possibly large messages: reliable communications needed
HTTP1.0 is transaction-based
Establish TCP connection
Client (user agent) sends HTTP request message with request URL and client info
Origin server sends HTTP reply message with status line and data
Connection is closed
HTTP1.1 keeps connection open by default (strict turn-taking)
Client sends HTTP request message
Server sends HTTP reply message
Repeat 2 & 3 until session ends
2/22/2019
CIS 5930 Fall TCP/IP

4. A Single Connection
Most HTTP communication is initiated by a user agent and consists of a request to be applied to a resource on some origin server
Accomplished via a single connection (v) between the user agent (UA) and the origin server (O):
Request chain UA v O
Response chain
2/22/2019
CIS 5930 Fall TCP/IP

5. Intermediaries
One or more intermediaries are present in the request/response chain
Three common forms of intermediaries
Proxy, gateway, and tunnel
Three intermediaries (A, B, C) between the user agent (UA) and the origin server (O):
Request chain UA v A v B v C v O
Response chain
2/22/2019
CIS 5930 Fall TCP/IP

6. Proxy A proxy is a forwarding agent that acts as a server and client
Receives requests for a URI in its absolute form
Multiple message exchanges may take place between requestor and proxy
A non-transparent proxy rewrites all or part of the message
May consume and remove authentication information
Apply protocol conversion
Forwards the reformatted request toward the server identified by the URI
Caching proxies may not forward request at all and send cached response to requestor
Receives response from server
Rewrites all or part of the message
Sends message to requestor
2/22/2019
CIS 5930 Fall TCP/IP

7. Gateway A gateway takes the role of an origin server
Requestor may not be aware it is talking to a gateway
Acts as a layer above some other server(s)
May translate the requests to the underlying server's protocol
2/22/2019
CIS 5930 Fall TCP/IP

8. Tunnel A tunnel is a “dumb” forwarding agent
Once active, the tunnel is not considered to be part of the HTTP communication processing
Tunnel ceases to exist when both ends close
Tunnels can be useful to bypass firewalls that block all but a few standard ports
For example database access (ODBC/JDBC) via stunnel tool over TLS/SSL port (requires ssh deamon on server side)
2/22/2019
CIS 5930 Fall TCP/IP

9. HTTP Request Message Format
method requestURI HTTP/1.1<CR><LF>
[header]* Host: domain[:port]<CR><LF>
[header]*
<CR><LF>
[body]
method: GET retrieve entity data from requestURI (body should be absent)
POST send entity (data in body) and retrieve data from requestURI
HEAD same as GET but no entity data should be returned
PUT save entity (data in body) under requestURI
DELETE delete resource associated with requestURI
domain[:port]: a domain name or IP followed by optional port number
requestURI: an absolute URI (/…) or relative URI referring to a resource
body: data content with encoding and type specified by headers
2/22/2019
CIS 5930 Fall TCP/IP

10. HTTP Request Message Header
header: User-Agent: token/token<CR><LF>
Authorization: credentials<CR><LF>
Connection: connection<CR><LF>
header: (applies to body content or resource identified by requestURI)
Transfer-Encoding:transfer-encoding<CR><LF>
Content-Length: nnnn<CR><LF>
Content-Type: type/subtype[;parameter]*<CR><LF>
Content-Encoding: encoding<CR><LF>
connection: Keep-Alive (keep connection open, note: by default)
Close (end of session)
transfer-encoding:
Chunked (chunked, content-length must be absent)
content-encoding:
Gzip (content is compressed)
2/22/2019
CIS 5930 Fall TCP/IP

11. Example HTTP GET Request
GET /%7Eengelen/courses/CIS5930/index.html HTTP/1.1
Host:
Connection: close
Close is implicit with HTTP/1.0 HTTP/1.1 requires explicit close
2/22/2019
CIS 5930 Fall TCP/IP

12. Example HTTP POST Request
POST /demo/calculator.php HTTP/1.1
Host:
Content-Length: 34
Content-Type: application/x-www-form-urlencoded
Connection: close
val1=1&val2=2&op=add&submit=Submit
Form data generated by browser’s processing of forms:
<form action=“calculator.php” method=“post”>
2/22/2019
CIS 5930 Fall TCP/IP

13. HTTP Response Message Format
HTTP/1.1 status reason<CR><LF>
[header]* <CR><LF>
[body]
status: a numeric status code 100<status<600
1xx - informational, request received processing continues
2xx - success: request received, accepted, and processed
3xx - redirection: further action needed to complete request
4xx - client error: request message malformed
5xx - server error: server failed to process a valid request
reason: a textual description of the status
body: data content with encoding and media type specified by headers
2/22/2019
CIS 5930 Fall TCP/IP

14. Common HTTP Status Codes
100
Continue
Continue processing (can ignore this status line)
200 OK
Success
201
Created
A response to a PUT request
202
Accepted
Processing started, but no complete (e.g. submitted for batch processing)
400
Bad request
Request message is malformed
401
Unauthorized
Authorization required
403
Forbidden
Not allowed to access resource
404
Not found
Resource not found
405
Method not allowed
The method used is not allowed
500
Internal server error
Server failure
2/22/2019
CIS 5930 Fall TCP/IP

15. Example HTTP Response (1)
HTTP/ OK
Content-Length: 93
Content-Type: text/html
Connection: close
<html>
<head><title>My HomePage</title></head> <body>Welcome to my home page!</body>
</html>
2/22/2019
CIS 5930 Fall TCP/IP

16. Example HTTP Response (2)
HTTP/ Not Found
Content-Length: 48
Content-Type: text/html
Connection: close
<html>The information could not be found</html>
2/22/2019
CIS 5930 Fall TCP/IP

17. Example HTTP Response (3)
HTTP/ Not Found
Transfer-Encoding: chunked
Content-Type: text/html
Connection: close 30
<html>The information could not be found</html>
Chunk length in hex followed by <CR><LF>, data, and <CR><LF>
Indicates end of data (followed by <CR><LF>)
2/22/2019
CIS 5930 Fall TCP/IP

18. Persistent Connections
Supported by some HTTP/1.0 implementations explicitly (RFC2068) and HTTP/1.1 by default, using the Connection header: Connection: keep-alive Connection: close
Opens and closes fewer TCP connections
CPU time is saved and memory usage is lower in routers and hosts
Network congestion is reduced by reducing the number of packets caused by TCP opens
Allows sufficient time to determine the congestion state of the network
Latency on subsequent requests is reduced since there is no time spent in TCP's connection opening handshake
HTTP requests and responses can be pipelined on a connection
Pipelining allows a client to make multiple requests without waiting for each response
HTTP can evolve more gracefully, since errors can be reported without the penalty of closing the TCP connection
2/22/2019
CIS 5930 Fall TCP/IP

19. Server Access Authentication
Origin server may require client authentication (RFC2617)
For example, Apache .htaccess file
Client request without authentication info is rejected
HTTP/ Unauthorized
Server includes authentication challenge header in response WWW-Authenticate: challenge
The challenge is Basic or Digest followed by a comma-separated list of name=value parameters, usually one of which is realm=“name”, which is shown in the browser’s authentication dialog box for manual login
Client resends request with authentication info
Authorization: credentials
The credentials contain basic authentication info with user name and password (not secure!)
Digest authentication info contains a cryptographically hashed password
2/22/2019
CIS 5930 Fall TCP/IP

20. Proxy Access Authentication
Similar to server access authentication, but:
Proxy includes authentication challenge header to response Proxy-Authenticate: challenge
Client (re)sends request with authentication info
Proxy-Authorization: credentials
A browser has to be configured to include proxy authentication with each HTTP request message
Connection settings for HTTP proxy host (IP) and port needed
2/22/2019
CIS 5930 Fall TCP/IP

21. Basic Authentication
Authorization credentials contain UID and password encoded in base64, which is effectively plain text Authorization: Basic code code = base64(UID “:” password)
Base64 is a common encoding format for raw data
Uses ASCII (7-bit) range, messages are only 33% larger
Splits byte stream into triples of 3 bytes (24 bits) and maps these to 4 bytes containing only 6 bits of information (4*6=24) using A-Za-z0-9+/
Never use basic authentication over non-encrypted channels
2/22/2019
CIS 5930 Fall TCP/IP

22. Digest Authentication
Avoids sending password in the clear
Both parties know a shared secret (password)
Uses a common cryptographic hashing scheme to verify secret:
Client computes the MD5 hash of the textual concatenation of the HTTP method (GET/POST), requestURI, UID, realm, password, nonce
Server also computes the MD5 hash and compares it to the client’s hash value in the request message Authorization header
If they match, access is granted
Quality of protection: can also hash with body entity, to ensure integrity of entire message (qop=auth-int)
A nonce is used to prevent replay attacks
A nonce is a unique (random) value to identify a transaction
Nonce and nonce counts are used to prevent replay attacks
A man-in-the-middle only sees the hash value and cannot deduce the password
2/22/2019
CIS 5930 Fall TCP/IP

23. Digest Authentication Example
Server
HTTP/ Unauthorized
WWW-Authenticate: Digest
qop="auth,auth-int",
nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",
opaque="5ccc069c403ebaf9f0171e9517f40e41"
Client
Authorization: Digest username="Mufasa",
nonce="dcd98b7102dd2f0e8b11d0f600bfb0c093",
uri="/dir/index.html",
qop=auth,
nc= ,
cnonce="0a4f113b",
response="6629fae49393a c4ef1",
opaque="5ccc069c403ebaf9f0171e9517f40e41"
2/22/2019
CIS 5930 Fall TCP/IP

24. Content Negotiation
How to prevent a server from sending data in a format a user agent cannot handle?
User agent need indicate the content encodings and media types it accepts
Actually, there are two types of negotiation
Server-driven: server decides the best representation for a response
Agent-driven: client decides the representation after receiving an initial response from the server
These are orthogonal and may be used in combination
2/22/2019
CIS 5930 Fall TCP/IP

25. Server-Driven Content Negotiation
Pro: server’s best guess often suffices
Cons: server may not be able to guess the best representation and user agent must describe all of its capabilities which adds message overhead
User-agent indicates accepted representations with HTTP headers:
Accept: mediaRange where mediaRange is a list of type/subtype[; q=x.y] example: Accept: */* example: Accept: audio/*;q=0.2, audio/basic “prefer audio/basic but send any audio type after 80% mark down in quality” example: Accept: text/plain;q=0.3, text/html
Accept-Charset: charsets
Accept-Encoding: codings where codings is a list of compression encodings example: Accept-Encoding: compress, gzip
Accept-Language: languages where languages is a list of language abbreviations, e.g. da, en-us, nl
2/22/2019
CIS 5930 Fall TCP/IP

26. State Management HTTP is inherently stateless
Persistent connections are meant to optimize for speed and can be terminated at any time
How to encode client state so server can deliver content based on client state when the client reconnects?
URL query strings:
But URL may be limited in size (256 bytes)
URLs are stored in logs, possible privacy concerns
Browser-based interaction: HTTP POST with HTML forms that have hidden state values
Cookies (RFC2965)
But cookies may have been disabled at the client side
2/22/2019
CIS 5930 Fall TCP/IP

27. Cookies
Server sends one or more Set-Cookie (or Set-Cookie2) HTTP headers to the user agent
Set-Cookie: name=“value”;path=“path”;domain=“domain” where domain and path are the target domain and request URL the user agent has to send cookies to
Domain is optional, when absent the user agent must use the server domain name
Path is optional, when absent the user agent must use the current request URI
Cookie lifetime determined by max-age=seconds
Upon receipt, the user agent stores the cookies in a local repository, that may be discarded then the user agent exits
Proxies must not set or return cookies
2/22/2019
CIS 5930 Fall TCP/IP

28. Cookies (cont’d)
Before the user agent sends a request, it searches the repository for matching domain/path and includes all the matching cookies in the request header: Cookie: name=“value”; $path=“path”
Cookie domain matching by client for returning cookies:
Match if origin server domain name matches a domain name in repository that has leading . (dot) and tail-part of strings match
a.b.com matches .b.com and a.b.c.com matches .c.com
Cookie path matching:
The path of the cookie must match a prefix of the request URI
User agent may need to limit the number of cookies it stores
Should be at least 300, at least 20 per host name, and should accommodate at least 4096 bytes per cookie value
2/22/2019
CIS 5930 Fall TCP/IP

29. Example
1. User Agent -> Server (user identifies self via a form): POST /acme/login HTTP/1.1 [form data]
2. Server -> User Agent (cookie reflects user's identity): HTTP/ OK Set-Cookie2: Customer="WILE_E_COYOTE"; Version="1"; Path="/acme” [data]
3. User Agent -> Server (user selects an item for "shopping basket"): POST /acme/pickitem HTTP/1.1 Cookie: $Version="1"; Customer="WILE_E_COYOTE"; $Path="/acme” [form data]
4. Server -> User Agent (shopping basket contains an item): HTTP/ OK Set-Cookie2: Part_Number="Rocket_Launcher_0001"; Version="1"; Path="/acme” [data]
2/22/2019
CIS 5930 Fall TCP/IP

30. Example (cont’d)
5. User Agent -> Server (user selects shipping method from form): POST /acme/shipping HTTP/1.1 Cookie: $Version="1"; Customer="WILE_E_COYOTE"; $Path="/acme"; Part_Number="Rocket_Launcher_0001"; $Path="/acme” [form data]
6. Server -> User Agent (new cookie reflects shipping method): HTTP/ OK Set-Cookie2: Shipping="FedEx"; Version="1"; Path="/acme” [data]
7. User Agent -> Server (user chooses to process order): POST /acme/process HTTP/1.1 Cookie: $Version="1"; Customer="WILE_E_COYOTE"; $Path="/acme"; Part_Number="Rocket_Launcher_0001"; $Path="/acme"; Shipping="FedEx"; $Path="/acme” [form data]
8. Server -> User Agent (transaction is complete): HTTP/ OK [data]
2/22/2019
CIS 5930 Fall TCP/IP

31. HTTP Upgrade Header
When a client or server uses HTTP1.0, the other side must downgrade to HTTP1.0 as well
A client or server may require the other end to upgrade to another protocol, usually HTTPS (HTTP+SSL/TLS)
For example, client attempt to connect to a secure site but sends a clear-text request message: GET /encrypted-area HTTP/1.1 Host:
Server refuses, insisting that client uses HTTPS: HTTP/ Upgrade Required Upgrade: TLS/1.0, HTTP/1.1 Connection: Upgrade
Client reconnects, this time using HTTPS
2/22/2019
CIS 5930 Fall TCP/IP

32. HTTP Applications
Web servers and Browsers (of course) and related technologies such as RSS and Ajax
wget
HTTP virtual filesystems
WebDAV
XML Web services
Application servers
Caches (optimization)
Tunneling (e.g. SOCKS and Java RMI over HTTP)
2/22/2019
CIS 5930 Fall TCP/IP

33. A Simple HTTP 1.1 Web Server for Browser-based Access
Must support at least:
GET method
Either support or disable:
To support POST method must also support chunked transfers
If keep-alive connection not supported: always use close response
For GET method header parser should extract at least
GET, RequestURI, and HTTP versioning info
Host header
For POST method header parser should extract at least
POST, RequestURI, and HTTP versioning info
Transfer-Encoding
Content-Length
Content-Type
2/22/2019
CIS 5930 Fall TCP/IP