server tools

Site server tools can be utilised to build, host, track and monitor transactions on a business site. There are a wide range of possibilities but today we are looking at Web server logs. Server based statistics analysis tools utilise server logs for their information

Built in web server logging tools Web servers such as Apache and MS IIS provide built-in site activity logging functionality to provide information such as: Who has accessed the website? When was it accessed? What was accessed? This information can make it possible to identify bottlenecks, security breaches, content popularity and facilitate troubleshooting.

File format of the web log files The W3C specification for extended log file format, describes a number of log fields that can be used for web server activity logging. The specification calls for an ASCII file format and time is logged as UTC (Greenwich Mean Time). Although different servers have different log formats they all look very similar

Information in Server Logs Web server logs are plain text (ASCII) files, independent of server platform. There are some differences between server software, but traditionally there are four types of server logs: 1.Transfer (access) log 2.Error log 3.Referrer log 4.Agent log The first two types of log files are standard. The referrer and agent logs may or may not be “turned on” at the server or may be added to the transfer log file to create an “extended” log file format. Each HTTP protocol transaction, whether completed or not, is recorded in the logs, and some transactions are recorded in more that one log. For example, most (but not all) HTTP errors are recorded in the transfer log and the error log.

Transfer (access) log The line below is an example of a single line in a common transfer log. This typically displays as one long line of ASCII text, separated by tabs and spaces (useful for importing it into a spreadsheet program or other analytical program). 1Cust216.tnt1.tpg.com.au - -[08/May/2007:12:13: ] GET /gen/meeting/ssi/next/HTTP/ Mozilla/7.01-C-MACOS8 (Macintosh; I; PPC) GET /gen/meeting/ssi/next/ - HTTP/1.0 Lets look at each section of this entry.

Remotehost Address or IP 1Cust216.tnt1.tpg.com.au This is the address of the computer making the HTTP request. The server records the IP and then, if configured, will lookup the Domain Name Server (DNS). However, with all the dynamically assigned IP addresses these days, you don’t learn as much as you’d expect from the domain name. In this case the visitor seems to be a customer of an ISP.

RFC931 (or identification) - Rarely used, the field was designed to identify the requestor. If this information is not recorded, a hyphen (-) holds the column in the log.(The remote logname of the user) Authuser - List the authenticated user, if required for access. This authentication is sent via clear text, so it is not really intended for security. This field is usually filled by a hyphen (-).

Time Stamp [08/May/2007:12:13: ] The date, time, and offset from Greenwich Mean Time (GMT x 100) are recorded for each hit. The date and time format is: DD/Mon/YYYY HH:MM:SS. The example above shows that the transaction was recorded at 12:13 pm on May 8, 2007 at a location 7 hours behind GMT.

request GET /gen/meeting/ssi/next/index.html HTTP/1.0 The request line exactly as it came from the client GET is the standard request for a document or program. POST tells the server that data is following. HEAD is used by link checking programs, not browsers, and downloads just the information in the HEAD tag information. The specific level of HTTP protocol is also recorded.

status 200 The resulting status code. "200" is success. If the requested URL didn't exist, this is where the dreaded "404" would have shown up in the log. Response codes fall into 4 groups Success (200 series) Redirect (300 series) Failure (400 series) Server Error (500 series) A status code of 200 means the transaction was successful. Common 300- series codes are 302, for a redirect from to and 304 for a conditional GET. This occurs when the server checks if the version of the file or graphic already in cache is still the current version and directs the browser to use the cached version. The most common failure codes are 401 (failed authentication), 403 (forbidden request to a restricted subdirectory), and the dreaded 404 (file not found) messages. Sever errors are red flags for the server administrator.

example server response codes 200- Request successful Partial request successful (not complete) URL has been redirected to another document Bad request was made by the client Authorization is required for this document Access to this document is forbidden Document not found Server internal error Application method (either GET or POST) is not implemented Server is out of resources

Transfer Volume 9887 For GET HTTP transactions, the last field is the number of bytes transferred. For other commands this field will be a hyphen (-) or a zero (0). The transfer volume statistic marks the end of the common log file. The remaining fields make up the referrer and agent logs, added to the common log format to create the “extended” log file format. Lets look at these fields.

Referer URL The referrer URL indicates the page where the visitor was located when making the next request. The actual request is shown in the last field of the entry GET /gen/meeting/ssi/next/ - HTTP/1.0 and is duplicated from the HTTP Request, the fifth field in this log. If you were looking at just the referrer log, not integrated into the transfer log, it would be made up of just two fields. The left field is the starting URL and the right field is where the reader went from the URL. Transfers within your site would also show in the transfer log. For example, movement from one page to another within a web site might show in the referrer log as: north.tafe.edu/ -> /gen/meeting/ssi/next/ The visitor went from the top-level page to another deeper level page through a link on the page.

User Agent Mozilla/7.01-C-MACOS8 (Macintosh; I; PPC) The user agent is information about the browser, version, and operating system of the reader. The general format is: Browser name/version (operating system) The confusion comes from the word “Mozilla,” which is the original code name for Netscape. Now almost all browsers compatible with Netscape use the Mozilla code.

Exercises 1. Interpreting Transaction Log File Data Take a look at the following excerpt from a real extended-format web server log file and answer the questions on the next slide Transaction #1 dejh.ipm.ac.ir - - [08/May/2007:00:47: ] "GET /spires/form/hepfnal.html HTTP/1.0" " "Mozilla/4.05 [en] (Win2003; I)" GET /spires/form/hepfnal.html - "HTTP/1.0" Transaction # [08/May/2007:02:11: ] "POST /cgi-bin/form-mail.pl HTTP/1.1" " BI.ERNET.IN" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)" POST /cgi- bin/form-mail.pl - "HTTP/1.1" Transaction #3 oeias1-p2.telepac.pt - - [08/May/2007:03:16: ] "GET /BFROOT/Images/BABAR2.gif HTTP/1.1" " www/Physics/Workshops/wkshp_home.html" "Mozilla/4.0 (compatible; MSIE 4.01; WinXP Pro)" GET /BFROOT/Images/BABAR2.gif - "HTTP/1.1"

Exercise Questions 1.How many visitors are reflected in these web transactions? Explain. 2.On what date did these transactions take place? 3.How many minutes passed between the first and last entry? Were all three transactions successful? If not, explain. 4.Which transaction requested the largest file? 5.What size was the file? 6.What browsers are being used to access these pages? 7.What platforms are being used to access these pages? 8.Can you determine the path the visitor from took through the website?

A few things to note : Each line in the web log files represents a single "hit" on a file on the web server A web server "hit" is not the same as a web page "hit". For example in a log file if there a user requests a page which consists of 3 gifs, 1 jpg then in the log file there will be 4 “hits” or records If this page also included a graphic which is being retrieved from a different site then it doesn't actually show up in this site's log.