2. Evaluation Workshop: Quantitative Evaluation Methods Peter Dowdell NOF-digitise Technical Advisory Service email: p.dowdell@ukoln.ac.uk web: http://www.ukoln.ac.uk UKOLN is supported by:

3. Aims of this presentation: To explain the need to establish a measurement policy using a range of performance indicators To explain some of the possible measurements we can record To discuss some of the pitfalls and problem areas in attempting to measure performance

4. Why Have Performance Indicators? Performance indicators for Web sites can be used for several purposes: Use in management reports showing service growth For Service Level Agreements with funding agencies To identify gaps in service provision To predict and plan for future load patterns To monitor performance levels To advise on deployment of new technologies To inform and motivate contributors

5. Why have a measurement policy? To create stable view points over your site. To make sense of the available data. To clarify and measure broader objectives. To answer questions.

6. What are we logging? We log each request the web server receives. We record information like: remote IP address date and time response code request string method GET, POST etc execution time data transfer

7. How can we log this? Each request is appended to a log file. There are different accepted formats: W3C … Alternatively requests can be logged to a database. Log files can become very LARGE! Should we start a new log according to: daily / weekly / monthly / by file size?

8. Server logs: #Software: Microsoft Internet Information Server 4.0 #Version: 1.0 #Date: 1999-12-25 00:00:21 #Fields: date time c-ip cs-username cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs(User-Agent) cs(Cookie) cs(Referer) 1999-12-25 00:00:21 194.237.174.119 - GET /issue1/jobs/Default.asp - 200 20407 AltaVista- Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 00:03:39 194.237.174.119 - GET /statistics/ExpIntHits1.asp - 200 10519 AltaVista- Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 00:26:54 209.67.247.158 - GET /robots.txt - 200 303 FAST-WebCrawler/2.0.9+(crawler@fast.no;+http://www.fast.no/…) - - 1999-12-25 00:32:47 194.237.174.119 - GET /issue2/default.asp - 200 5332 AltaVista-Intranet/V2.3A+(www.altavista.co.uk+jan.gelin@av.com) - - 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/bg.gif - 200 300 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit- lib.org/issue1/webtechs/ 1999-12-25 01:49:54 206.186.25.7 - GET /issue1/webtechs/Default.asp - 200 24659 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) - http://www.statslab.cam.ac.uk/%7Esret1/analog/webtechs.html 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/global_home_h.gif - 200 487 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit- lib.org/issue1/webtechs/ 1999-12-25 01:49:54 206.186.25.7 - GET /resources/images/main/global_search.gif - 200 534 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit- lib.org/issue1/webtechs/ 1999-12-25 01:49:56 206.186.25.7 - GET /resources/images/main/local_home01.gif - 200 663 Mozilla/2.0+(compatible;+MSIE+3.02;+AK;+Windows+NT) ASPSESSIONIDGQQGQGAD=IIHCBIFDIECKPAPGICDEOJII;+SITESERVER=ID=22e0a17296b8c2ed1f77460cde75c27f http://www.exploit- lib.org/issue1/webtechs/

9. How to analyse the log files Analog ( http://www.analog.cx ) Webaliser ( http://www.mrunix.net/webalizer ) WebTrends ( http://www.webtrends.com ) Bespoke? One of the scripting solutions?

10. What do we measure? Hits: the overall number of requests that the server is handling. Includes all files making up a web page. Pages: the number of files designated as base pages: determined by file extension ….htm /.html /.asp /.php /.cfm? Visits: many assumptions must be made User Agents.. Total Data transfer Average processing time Search terms in referrer string Failed requests

11. What are the problems? Robots and other agents Developers and in-house access Caches and Proxy servers can conceal site usage IP addresses lookup can mislead IP address can mask multiple users: firewalls, NAT

12. Fluctuations in Hits & Requests In 2000 images are introduced across a web site (two images per text page) Result: Nos. of hits trebles, while number of page requests remains constant In 2001 external JavaScript files are used to animate menus when they are selected Result: Nos. of hits increases while number of page requests remain constant In 2002 internal style sheets are used to replace images of Result: Nos. of hits decrease while number of page requests remain constant

13. How to access the results Who needs to see the processed reports? Do you need a private area on your website? Will you allow 3rd party access to reports, possibly to a reduced set of information? How much configuration and re-processing will you allow?

14. External Services Server usage can also be determined by third-party services: www.nedstat.com www.sitemeter.com Non-commercial only - or you pay! No guarantee of service Includes client-side sniffing

15. Client-side sniffing Some interesting statistics are not derivable from server logs - Browser plug-in status - Javascript support - CSS support Client sniffing code can be used to log this information. Browser cookies may allow us to detect individual user sessions.

16. Service Monitoring We also would like to know that our service is available: Remote monitoring services Alerting or reporting? Does this overlap with your hosting SLA?

17. Link monitoring How is the site ranked in search engines? Use url= syntax in common search engines How many sites have linked to you? www.linkpopularity.com

18. Coverage By Search Engines Have you promoted your Web site? Can your Web site be accessed by search engines? Are you near the top of the search results? Search engines can report on their coverage of your Web site Coverage is an indication of potential use of your Web site For information on how to ensure that your web site has been indexed see

19. Links To Your Site Search engines can be used to report on the numbers of links to a Web site LinkPopularity.com provides an interface to 3 search engines Monthly reports can be obtained Links are an indication of potential use of your Web site A survey of the number of links to University web sites is available at. www.linkpopularity.com

20. Links From Your Web Site Links from your Web site: –Usually implemented using: Foo –Not normally possible to monitor nos. of users following link –Is possible if using link of the form: Foo

21. Considerations What will we measure? How often will we produce reports? How will we handle our raw server logs? Will we be able to view the results over the web? Will we need different levels of reporting detail for different users? Technical / Executive / 3rd party?