Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 1 Evaluation of Adaptive Web Sites Elizabeth LaRue by

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 2 Presentation Resources  Gena, C. (2005). Evaluation of Adaptive Web Sites. Retrieved September 6, 2005, from Courseweb.  Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA.  Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3),  Cheverst, K. Davies, N. Mitchell, K. et al. (1-6 April 2000). Developing a context-aware electronic tourist guide: some issues and experiences. CHI. [electronic source] retrieved from ACM Digital Library.

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 3 Objectives Review associated terminology Describe project phases Examine various evaluation methods

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 4 Terminology Adaptive Web Sites – present and format information material automatically on a web page for the user Evaluation – to examine carefully 1 Empirical – relying on or gained from observation or experiment rather than theory 1 1 Houghton Mifflin Company. (1996). Webster's ii new riverside dictionary (Revised Edition ed.). Boston: Houghton Mifflin Company.

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 5 Benefits of Evaluation Helps design Insight into user behavior Tests usability and functionality Assists with user models

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 6 Adaptive Systems Evaluation Consider a layered approach – separate evaluation into the adaptive components of the adaptive systems A. Content layer B. Interface layer

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 7 Software Lifecycle Development Waterfall Methodology Planning/Requirements Design/Analysis Code/Program Testing Implementation Evaluation (development)

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 8 Planning/Requirements Why? Continual EVALUATION helps to complete the requirements specification document Assists in Focusing on users – typical behavior, actions, needs, etc… Focusing on tasks – actions, needs, etc…

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 9 Why? Design/Analysis Assists in Prohibiting expensive design mistakes Prohibiting costly redesign

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 10 Code/Program (development) Why? Assists in Evaluating overall quality of a system

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 11 Before Evaluating Data collection = query technique –Usually used at each step in the life cycle –Usually cheap –Usually simple

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 12 Questionnaires Interviews Before Evaluating - Tools

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 13 Verbal and think aloud (pre and post) User’s observation Logging Use Before Evaluating – Tools Observational Methods

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 14 Formal Analytical Empirical Three Types of Evaluation

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 15 Rarely used Look at prediction of performance, complexity and learnability Formal Type of Evaluation

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 16 Formal Methods Task Analysis Cognitive –GOMS –KLM (Keystroke Level Model) Socio-technical models

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 17 Used without users, must have clear evaluation criteria of the system Benefits –Quick –Cheap –Reasonably effective Analytical Type of Evaluation

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 18 Analytical Models Heuristic Evaluation Cognitive walkthrough Accessibility Automatic Usability Testing and Web usage mining

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 19 Empirical Type of Evaluation Best for objective assessment of design Best for broadest range of usability problems Requirements to do empirical evaluation 1. Careful planning 2. Careful execution 3. Users must represent actual user population 4. Must have expense account – pay users

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 20 Empirical Methods Contextual evaluation Focus group Systematic Observation Expert review Wizard of Oz prototyping Prototyping Card sorting Cooperative evaluation Participative evaluation Usability testing Accessibility Automatic Usability Testing and Web usage mining Controlled experiments Ethnography Grounded theory Metric to evaluate the effectiveness of adaptive contents

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 21 Specific Examples Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA. Mixed Methods – Empirical Method survey parallel design task analysis questionnaire - post

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 22 Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA. Evaluation of adaptive commercial web site

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 23 Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA. Evaluation of adaptive commercial web site

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 24 Evaluation of adaptive commercial web site Test: 1. Measure the task completion time 2. Amount of within page navigation 3. The satisfaction of the users Evaluation Methods Interaction simulation with pre-test Accomplish task in non-adaptive version and then repeat in adaptive version Study Sample 14 subjects Aged High Internet knowledge and web browsers and use the Internet during work Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA.

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 25 Specific Example 2 Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3), Mixed Methods – Observational Method Questionnaire Think aloud

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 26 Evaluation of adaptive museum web site Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3),

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 27 Evaluation of adaptive museum web site Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3),

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 28 Evaluation of adaptive museum web site Test: 1. Does the virtual guide make a visit more pleasant and instructive 2. Learn if the type of information provided by the guide was interesting to the user 3. Is the presentation of the material effective or could be improved Evaluation Methods Think aloud without tasks – minimal 30 minutes with and without adaptive system Post-questionnaire – 29 questions Study Sample 40 subjects Aged (16 males and 24 females) Previous web experience and half rarely accessed a museum web site Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3),

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 29 Evaluation of adaptive museum web site Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3), Think aloud methodology Evaluators recorded particularly negative and positive reactions Evaluators noted specific paths followed by users Evaluators were careful to not make suggestions to users Questionnaire 15 questions on general usability of interface, navigation and ease of access 5 questions related to each of the five types of information provided by the virtual guide 9 questions related to user personal information, i.e. kn of museum application domains

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 30 Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3), Specific Example 2A Evaluation of tourist web site: (GUIDE) Mixed Methods Expert walkthrough – one hour Talk aloud Post-interview Study Sample 4 expert subjects Knowledge area of user-centered design and computer supported learning

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 31 Evaluation of tourist web site: (GUIDE) Cheverst, K. Davies, N. Mitchell, K. et al. (1-6 April 2000). Developing a context-aware electronic tourist guide: some issues and experiences. CHI. [electronic source] retrieved from ACM Digital Library. l

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 32 Evaluation of tourist web site: (GUIDE) Prototype Evaluation Methods Talk aloud – without tasks Time stamped log files Post semi-structured interview Study Sample 60 volunteer subjects Aged Knowledge area of user-centered design and computer supported learning Cheverst, K. Davies, N. Mitchell, K. et al. (1-6 April 2000). Developing a context-aware electronic tourist guide: some issues and experiences. CHI. [electronic source] retrieved from ACM Digital Library.

Evaluation of Adaptive Web Sites 3954 Doctoral Seminar 33 Thank You