Evaluation into the usability of a Virtual Learning Environment By Ian Cole (University of York) Dina Lewis (University of Hull) United Kingdom

Background A partnership approach - NHS working with University of Hull and University of York. NHS northern region e-Learning strategy. Need to up skill workforce with knowledge and understanding of the potential of e-Learning to support work based practice. We wanted to look at function and usability of a live course using a VLE. FOLC course designed by University of Hull.

Facilitating online learning communities programme 3 face-to-face workshops. 7 online activities embedding e-theory in collaborative group work, case studies, presentations, use of web resources and e- learning pedagogy. Course used the University of Hull’s Blackboard VLE. 16 e-learning champions recruited. 15 participants completed the course.

Participation rates were phenomenal 10 week programmeHits Weeks ,593 Weeks 1-518,441 Weeks ,464 Tutor2,827 Most active participant3,501 Least active participant 871 Usability test group1,045 Average participant1,773

User access hours of the day

User access days of the week

Evaluation findings The programme was very successful in terms of: The learning community, participation levels, motivation and commitment levels were extremely high. Raising knowledge and understanding of key issues in supporting e-Learning. Enabling participants to identify the potential of e-Learning to support work-based learning within their organisation.

Evaluation issues Time was a key issue for all participants. Need for coherent Senior Management strategy across organisations in terms of: protected work time for e-Learning, resources, venues, access to computers, IT support staff. E-learning is not an add on it should be an integral mode of NHS core business.

Usability Study Usability study conducted by the University of York. Used Jakob Nielsen’s ‘framework of system acceptability’ where usability is a part of the usefulness of a system.  Learnability.  Efficiency of use,  Memorability,  Few and Noncatastrophic Errors  Subjective Satisfaction.

Usability Methodologies Questionnaire Details:  100 questions based on The Questionnaire for User Interaction Satisfaction (QUIS) tool.  Two part questionnaire. a person specification. a usability section.  Questionnaire based on Ben Shneiderman’s comprehensive criteria list for usability testing.  We modified Shneiderman’s scale slightly to include a 0 to 9 scale instead of 1 to 9.

Ben Shneiderman’s criteria: Images. Readability. Use of colour. Navigation. Interactivity. Learning to use the VLE including learning styles. System capabilities. General and overall reactions. Accessibility.

Usability Methodologies Cognitive Walkthrough Experiment. Grounded in Lewis & Polson’s CE+ Theory.  4 step information processing model: 1. The user sets a goal to be accomplished with the system. 2. The user searches the interface for available actions. 3. The user selects the action that seems likely to make progress toward a goal. 4. The user performs an action and evaluates the system's feedback for evidence that progress is being made. Videoed the CWE tester interacting with Bb. Results mapped against usability team & students responses.

Q85 – Using the VLE is: Frustrating Satisfying NR’sMeans Students Control Experiment 1 Totals Comment:  S: Had quite a few problems with access speed, which really frustrated me, as I didn’t have a lot of time to spend waiting for something to happen.  5 person usability team.  The 15 FOLC students.  1 Cognitive Walkthrough Experiment tester. Study Participates:

Study Results Logging on to Bb was easy (8.1 mean). The speed of logging in to the VLE is ok (5.0 mean). Screen layout, organisation of info, navigation & text size were all between 5.0 & 7.1 mean. Use of colour & images were 6.0 & 6.1 mean. The instructions for using the Virtual Classroom are Inadequate (3.9 mean). The Virtual Classroom interface is dull (4.8 mean). Using the Discussion board is acceptable (above 5.7 mean) except search facility (4.5 mean).

Getting lost when using the VLE is easy (4.3 mean). Personalising the VLE screen is difficult (2.3 mean). The flexibility of the VLE for dyslexic users (i.e. changing screen colours) is Inadequate (2.0 mean). The flexibility of the VLE for visually impaired users (i.e. changing text size) is inadequate (3.5 mean). The speed of the VLE is Too Slow (2.5.mean). The VLE is unreliable (3.4 mean). All mean scores are totals. Study Results

Overall Reactions to the VLE:  Terrible to Wonderful = 5.1 mean  Dull to Stimulating = 5.0 mean  Difficult to Easy = 6.3 mean  Frustrating to Satisfying = 4.1 mean  Rigid to Flexible = 4.8 mean  Inconsistent to Consistent = 5.2 mean Study Results

Conclusions & Recommendations: Highly motivated students. Problems with NHS networks being able to support VLE adequately. Problems of speed accessing Bb server. Bb should be more flexible regarding accessibility. Better signposting & navigation in Bb. More flexibility of the system for users and course developers.