Presentation on theme: "Brussels,18. March 2010 1RURAL WINGS IP RURAL WINGS Final Review Meeting Usability Evaluation Thomas Köhler, Nina Kahnwald Media Center, Dresden Univ."— Presentation transcript:
Brussels,18. March 2010 1RURAL WINGS IP RURAL WINGS Final Review Meeting Usability Evaluation Thomas Köhler, Nina Kahnwald Media Center, Dresden Univ. of Technology
Brussels, 18. March 2010 2RURAL WINGS IP structure 1.Introduction 2.Usability Questionnaire Procedure Response rates Results -Personal data, Infrastructure, Usage Profiles -Usability of Rural Wings infrastructure, training and support -Usability of Rural Wings applications 3.Usability monitoring during user training Procedure and Results 4.Heuristic Evaluation Procedure and Results 5.Summary
Brussels, 18. March 2010 3RURAL WINGS IP procedure Overall usability evaluation activities: Usability Assessment through Questionnaires – after a few months of usage (11/2007-2/2008, 12/2008-2/2009 and 10/2009-12/2009) Usability Testing through monitoring of questions and problems – during implementation and training sessions (6/2007-12/2009) Usability heuristic Evaluation of Rural Wings CAP – prototype stage (8/2007) and implementation stage (12/2009) Technical Evaluation – continued monitoring during test runs (11/07- ongoing)
Brussels, 18. March 2010 4RURAL WINGS IP usability evaluation questionnaire Usability questionnaire Structure of the questionnaire (based on „Questionnaire for User Interface Satisfaction“ [QUIS]): Personal data Infrastructure Usage profile Performance/Usability of technical infrastructure Usability of CAP Usability of applications Scales are applicable to all user groups identified in the user needs analysis (WP3).
Brussels, 18. March 2010 5RURAL WINGS IP results Response rates data entry that was completed until 20th of December 2009 analysis of all questionnaires where users specified their pilot site questionnaire was completed by 159 end users from 90 pilot sites
Brussels, 18. March 2010 6RURAL WINGS IP response rates I CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) All sites52133159 Armenia000 Cyprus258 CYP01 (Λεμύθου / Lemýthou)1-1webTV, TeacherNet CYP02 (Κρήτου Τέρρα / Krī́tou Térra)1-1Health Training CYP03 (Καμπιά / Kampiá)-11YouRA CYP04 (Παραμύθα / Paramýtha)-11 YouRA, D-Space agroweb, webTV, YouRA, D-Space, TeacherNet, NEMED, HealthTraining CYP05 (Ασγάτα / Asgáta)-11webTV, YouRA CYP06 (Μαρώνι / Marṓni)-11YouRA CYP07 (Άγιος Γεώργιος / Ágios Geṓrgios)--1 Xplora, YouRA, UNITE, Connect, TeacherNet, CYP08 (Κάμπος / Kámpos)-11 YouRA, D-Space agroweb, webTV, YouRA, D-Space, TeacherNet, NEMED, AgroTD, HealthTraining
Brussels, 18. March 2010 7RURAL WINGS IP response rates II CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) Estonia3127 EST01 (Ruhnu)211webTV, Health Training EST02 (Piirissaare)111- EST03 (Pamma)-21 webTV, Health Training HealthTraining EST04 (Nasva)-21 Health Training agroweb, webTV, Xplora, NEMED, HealthTraining EST05 (Oitme)-21 Health Training EST06 (Panga)-31 Health Training, Xplora Health Training EST07 (Võhma)-11Health Training
Brussels, 18. March 2010 8RURAL WINGS IP response rates III CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) France71913 FRA01 (INSEAD)51 Agroweb, RCCM, MEDSKY RCCM, MEDSKY FRA02 (Martinique)672 webTV, Xplora, UNITE, RCCM, MEDSKY webTV, RCCM, MEDSKY RCCM, MEDSKY FRA03 (Manso)113 D-Space, Teacher net, MEDSKY, RCCM, MEDSKY RCCM, MEDSKY FRA04 (CC2F)-12RCCM, MEDSKY FRA05 (Puscaghia)-21webTV, MEDSKY FRA06 (La Grande Paroisse)-32RCCM, MEDSKY FRA08 (Letia)--2-
Brussels, 18. March 2010 response rates IV CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) Greece114132 GRE01 (Σάλακος (Sálakos)114 YouRA, D-Space, CONNECT, YouRA webTV, YouRA, D-Space, TeacherNet, NEMED, AgroTD, HealthTraining, RCCM GRE02 (Αιγιάλη / Aigiálī)121YouRA, D-Space, CONNECT GRE03 (Πυλές / Pylés)111 NEMED, AgroTeleDiag. webTV, YouRA, NEMED webTV, YouRA, TeacherNet GRE04 (Κλειστός / Kleistós)-21 YouRA GRE05 (Βαλτεσίνικο / Valtesíniko)131 YouRa, Teacher net Xplora, YouRA webTV, Xplora, TeacherNet GRE06 (Άγιος Νικόλαος Βοιών / Ágios Nikólaos Voiṓn)212VEMUS, YouRA GRE07 (Μεστά / Mestá)111 webTV, VEMUS webTV GRE08 (Γεράκι / Geráki)311 agroweb, webTV, Health Training AgroTeleDiag webTV GRE09 (Νυμφαίο / Nymfaío)-21 WebTV webTV GRE10 (Άγιος Λαυρέντιος / Ágios Lavréntios)-11 MEDSKY RCCM
Brussels, 18. March 2010 10RURAL WINGS IP response rates V CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) GreeceGRE11 (Λέχοβο / Léchovo)--1WebTV, Xplora, YouRA, TeacherNet GRE12 (Ψαρά / Psará)-11YouRA, NEMED GRE13 (Γλαύκη / Glávkī)-11 ExperiNet Xplora, ExperiNet GRE14 (Οργάνη / Orgánī)-11 D-Space Xplora, D-Space, ExperiNet GRE15 (Ορεστιάδα (Orestiáda)-11 ExperiNet, TeacherNet ExperiNet GRE16 (Φλομοχώρι / Flomochṓri)-11WebTV, NEMED GRE17 (Κορυστά / Korystá)-12ExperiNet, EUDOXOS GRE18 (Ανώπολη / Anṓpolī)-11 YouRA GRE19 (Σίσες / Síses)-11 webTV, Health Training NEMED GRE20 (Γκούρα / Gkoúra)12webTV GRE21 (Κομοτηνή / Komotīnī́)-11 YouRa, ExperiNet, CONNECT Xplora
Brussels, 18. March 2010 response rates VI CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) GreeceGRE22 (Αργύρι / Argýri)-2-YouRa, Teacher Net, NEMED GRE23 (Βουνιχώρα / Vounichṓra)-71Health Training GRE24 (Καλλιφώνι / Kallifṓni)-11 YouRA YouRA, D-Space, ExperiNet GRE25 (Βερτίσκος / Vertískos)-11- GRE26 (Λυσσαρέα / Lyssaréa)-21 agroWeb, Xplora, ExperiNet, Teacher Net Xplora GRE27 (Άγιος Δημήτριος / Ágios Dīmī́trios)-31NEMED Hungary--2 HUN03 (N.N.)--2YouRA Israel-159 ISR01 ( حورة / חורה / Ḥ ūra ) --5agroweb, VEMUS, webTV ISR02 (Fourier " فورييه / פורייה ")--8Agroweb, webTV
Brussels, 18. March 2010 response rates VII CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) IsraelISR03 ( اﺕﺩ / Atid)--10VEMUS, D-Space, NEMED ISR04 ( ابو تلول / Abū Tlūl)--11VEMUS, Xplora, YouRA, TeacherNet ISR05 ( ביר הדאג ' / بئر هداج / Bir Hadāğ)--7agroweb, webTV, UNITE ISR06 ( بئر ﻡﺵاﺵ / Bir Mišāš)--18 agroweb, VEMUS, webTV, Xplora, YouRA, D-Space, ExperiNet, TeacherNet ISR07 ( ابو قويدر / Abū Qwaydar)-10- Poland21712 POL01 (Babiogórski PN, Zawoja)111webTV POL02 (Polana)121 D-Space, Teacher net, VEMUS Xplora POL03 (Wiśniowa)-21 Xplora, UNITE, D-Space, ExperiNet, CONNECT, AgroTeleDiag Xplora POL04 (OAUJ, Mt. Lubomir)21D-Space POL05 (Rokiciny)11 webTV, Teacher Net webTV, Xplora, YouRA POL06 (Skawa)-22RCCM
Brussels, 18. March 2010 response rates VIII CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) POL07 (Nowy Łupków)-11 D-Space, teacher Net, Health training webTV POL08 (Kęty)-62 VEMUS, webTV, Xplora, YouRA, CONNECT agroweb, Xplora, YouRA, TeacherNet POL09 (Myczkowce)--1WebTV, AgroTD POL10 (Harkabuz)--1D-Space Romania81915 ROM01 (Dezna)821 VEMUS, webTV, Xplora, YouRA, UNITE, D-Space, ExperiNet, CONNECT, Teacher net, NEMED, AgroTeleDiag., Health Training, RCCM agroWeb, VEMUS, Xplora, YouRA, UNITE, D-Space, ExperiNet, CONNECT, Teacher net, NEMED VEMUS, YouRA, NEMED ROM02 (Piatra Arsă)11Health Training ROM03 (Cozieni)-11 agroweb, VEMUS, webTV, Xplora, YouRA, UNITE, D- Space, ExperiNet, CONNECT, Teacher net, NEMED VEMUS, YouRA, NEMED ROM04 (Golu Grabicina)-11 VEMUS, webTV, Xplora, D-Space NEMED ROM05 (Arefu)-11 Teacher Net NEMED ROM06 (Brebu)-31 VEMUS, ExperiNet, CONNECT, Teacher net, NEMED, Health Training NEMED
Brussels, 18. March 2010 14RURAL WINGS IP response rates IX CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) Romania (cont.) ROM07 (Măguri- Răcătău)-11 Xplora, UNITE, D-Space, ExperiNet, NEMED, AgroTeleDiag NEMED ROM08 (Dăbâca)-11 Xplora, CONNECT, TeacherNet NEMED ROM09 (Orlat)-12 agroweb, UNITE, CONNECT VEMUS, NEMED ROM10 (Rod)11 VEMUS, TeacherNet, NEMED NEMED ROM11 (Deal)11VEMUS, NEMED ROM12 (Nadăş)-11 agroweb, VEMUS, webTV, Xplora, D-Space, NEMED, MEDSKY NEMED ROM13 (Tilişca)-11 agroweb, VEMUS, webTV, Xplora, YouRA, D- Space, NEMED, AgroTD NEMED ROM14 (Bucureşti)-31 VEMUS, YouRA, UNITE, Teacher Net, NEMED, Health Training, MEDSKY VEMUS; YouRA, NEMED, Health Training
Brussels, 18. March 2010 15RURAL WINGS IP response rates X CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) SpainSPA05 (Rellinars)-21 YouRA SPA06 (Laredo)-1-YouRA, webTV SPA07 (Santa Comba)-11YouRA, webTV SPA08 (Vilaverd)1- YouRA, webTV, D-Space webTV, YouRA SPA09 (Reboreda)2- YouRA, webTV YouRA, D-Space SPA10 (Blancafort)21YouRA, webTV Ramallosa7-- NEMED, AgroTeleDiag., agroweb, webTV, Xplora, YouRA Sant Sernip-1-webTV, YouRa, NEMED Sweden133 SWE01 (Tarfala)1-1MEDSKY SWE02 (Lomträsk)-21webTV, MEDSKY SWE03 (Avaviken)-11-
Brussels, 18. March 2010 response rates XI CountryPilot siten Test Run n Final Run Phase A n Final Run Phase B RW applications used and rated (red=test run, black=final run A, green=final run B) Spain9148 SPA01 (Teo-Campos)111 NEMED, AgroTeleDiag, webTV, YouRA, NEMED webTV, YouRA, NEMED, EUDOXOS SPA02 (Prats)1-1 NEMED, AgroTeleDiag. WebTV, YouRA SPA03 (Bamonde)-21 YouRA SPA04 (Arcos de Furcos)-11YouRA UK1020 Bewholme42-webTV, CONNECT Cilcennin5--webTV, CONNECT, D-Space Biggar1---
Brussels, 18. March 2010 17RURAL WINGS IP personal data I Demographic aspects: Gender: 66 female, 86 male participants Age: majority (69%) is between 25 and 49 years old. Professions: Other: i.e. civil servant, school consultant, journalist or librarian
Brussels, 18. March 2010 18RURAL WINGS IP personal data II
Brussels, 18. March 2010 19RURAL WINGS IP infrastructure Infrastructure characteristics: 75% of the RW users use MS Internet Explorer as webbrowser, Firefox is used by 22% of the users. Netscape and Opera are used by only 1,3%. Nearly all responding RW users (96%) have MS windows installed as operating system. Five (3%) use a Linux system and no one uses a Macintosh. Connection to the RW internet access: via Sat modem: 36,5% via local area network: 32,1% via indoor wireless network connection: 14,5% via outdoor wireless network: 6,9% web browserIENetscapeFirefoxOperaSafari % overall75.2%1.3%22.2%1.3%0.0% operating systemWinWin VistaWinXPWin2000Win98 WinN TLinuxMacN/A users15329118600502 in %96.218.02184.108.40.206 3.10.01.3
Brussels, 18. March 2010 20RURAL WINGS IP usage profiles I
Brussels, 18. March 2010 21RURAL WINGS IP usage profiles II Services used: 1.web browsing (88,7%) 2.e-mail (74,8%) 3.RW applications and tools (41,5%) 4.Instant messaging (31,4%) 5.software updates (28,9%) Websites visited (3 most visited): 1.search engines 2.news-sites 3.online-mailing 4.RW-CAP 5.weather forecast
Brussels, 18. March 2010 22RURAL WINGS IP usage profiles III CAP use: Duration per weekUser Location
Brussels, 18. March 2010 23RURAL WINGS IP usability of infrastructure I All categories are rated as excellent or good by at least 39% Highest satisfaction can be seen in the area of assistance ratings which every fifth user (19.5% and 22%) judged as excellent. The initial installation was rated good or excellent by 69% of the users. Deficits or problems seem to exist with the availability of the service (rated “poor” by 17.6% of the end users) and the reliability of the satellite terminal (rated “poor” by 7.5%). Also free text answers and comments refer to low bandwidth and unreliable connections with repeated failures.
Brussels, 18. March 2010 24RURAL WINGS IP usability of infrastructure II
Brussels, 18. March 2010 25RURAL WINGS IP usability of infrastructure III Summary usability ratings „poor“ (part I)
Brussels, 18. March 2010 usability of infrastructure IV Summary usability ratings „poor“ (part II)
Brussels, 18. March 2010 27RURAL WINGS IP usability infrastructure V
Brussels, 18. March 2010 28RURAL WINGS IP usability RW-CAP I RW applications are often used independently from the CAP: Data from Google Analytics has shown that slightly more than 50% of the users did access these RW applications directly. The following evaluation results thus reflect the experience of the remaining up to 50% users. Some users perhaps don‘t see the added value of the CAP: „Do not see a strong need to always enter through the portal when you already know where you want to reach out.” RW-CAP ratings overall impression was similar to the interim evaluation: A critical topic was the missing localization and translation of RW- CAP contents, as stated by several users in the free text comments: “The CAP is in English, because of that we cannot judge its relevancy.” (Spain) / „Not available in French.“ (France) / „The main problem in using the site is the language barrier.” (Poland) However this has been solved by the beginning of 2010 with a new Greek + French version. Also national partners may insert translations by themselves easily if needed.
Brussels, 18. March 2010 29RURAL WINGS IP usability RW-CAP II
Brussels, 18. March 2010 30RURAL WINGS IP usability RW applications I 15 different RW applications were used and rated during the final runs. 39% of the participants in the survey had not used any RW application when answering the questionnaire (32% in final runs Phase A evaluation). On the whole the applications were rated positively. less positive rating than during the last evaluation: Agroweb (15 users), VEMUS (14 users), WebTV (30 users), Xplora (29 users), D- Space (13 users) and Teacher eTraining (13 users) lost slightly: YouRA (27 users), UNITE (2 users) and ExperiNet (8 users) rated as positive as the last time: NEMED (25 users), AgroTeleDiag (3 users), the Health Training (12 users), RCCM (9 users) and MEDSKY (6 users) See Annex A of this presentation for detailed usability ratings for each application.
Brussels, 18. March 2010 31RURAL WINGS IP usability monitoring Principles of RW usability monitoring User problems and questions can be indicative for aspects of the technical infrastructure, the manuals, the central access point or the rural wings platforms and applications that are not designed intuitively. It can thus be used to identify possible usability problems and areas of improvement during the training of the end users. A structured template for the collection of user questions and problems arising during the training of the end users was provided to all NCs. Data collection realized after February 2009; in final test runs feedback only from Spain.
Brussels, 18. March 2010 32RURAL WINGS IP results usability monitoring Emerging topics: Language was not a problem as localized sites are available; PC-skills become an issues; RW applications and CAP (problems with D-Space and CAP-Structure); RW Infrastructure (Problems with network - suggested usage of a faster browser did improve the performance somewhat but reportedly not enough). Manuals:1 Tools:6 Infrastructure:1 Central Access Point:3 Other:0 Overall questions/ problems7
Brussels, 18. March 2010 33RURAL WINGS IP heuristic evaluation Design of heuristic evaluation Expert evaluation of the RW Platform by usability experts of TUD: checklist of recognized usability principles (heuristic) will be used to identify possible usability problems (see Schweibenz/Thissen 2003 and Nielsen/Mack 1994). identified usability problems and possible recommendations will be aggregated and communicated to FORTHnet and all NCs by TUD. Expert evaluation was conducted twice: at prototype stage (2007) and end of the project (2009) after improvements were made. Results overall better impression than during 1st usability evaluation: lucid, structured, not overloaded. but information offered and options could be structured and grouped better and adapted to the needs of the users often available in English only, even the help function (!) language specific (sub) sites should be developed for Further details of the heuristic evaluation cp. annex A
Brussels, 18. March 2010 34RURAL WINGS IP Conlcusions I Language and localisation Language barrier had been identified as one of the most crucial issues in previous evaluations. RW Infrastructure RW services are mostly used during the week and earlier in the day, while the usage during the weekend has increased as compared to the last evaluation. Apart from language barriers, problems with network performance and wifi coverage most dominant in user feedbacks. Availability of the RW internet service has been rated as “poor” by 17.6% of the end users (compared to 11% during the last evaluation phase). Negative experiences with bandwidth reliability did again retard local activities, hinder training sessions or even prevent proper use of RW services most of the time.
Brussels, 18. March 2010 35RURAL WINGS IP Conlcusions II RW-CAP Development of CAP-usage: in test run evaluation nearly all participants had used RW- CAP, in the final runs 18% stated “not applicable”; only 29% of the participants compared to 40% in the test runs did name it as one of the three most visited websites. Language barriers and navigation problems remain (monitoring and heuristic evaluation). Overall feedback less critical than in the test runs (higher satisfaction or lower relevance of CAP?). Suggested improvements: Single log-in for all applications via CAP which would provide an added value; Community features (have been implemented roughly but not promoted).
Brussels, 18. March 2010 36RURAL WINGS IP Conlcusions III RW Applications 16 different RW applications were used and rated – predominantly positive, albeit slightly lower than during final phase A. Although 39% of the participants in the survey had not used any RW applications when answering the questionnaire. Free text comments division between users who found the applications “very good”, “marvellous” or “motivating” and those who, for various reasons, didn’t manage to get some of the applications working at all. RW training and support The overall rating of the RW installation services and the RW support was very positive. About a fifth of the participating users did rate it as “excellent”. Suggestion for successive projects: to ensure sustainability meta-competencies should be imparted (i.e. problem-solving strategies in case of computer problems) to enable users to help themselves outside of training sessions. Supported by community features in RW CAP to allow end-users to share best practice with other pilot sites and support each other when problems occur.
Brussels, 18. March 2010 37RURAL WINGS IP Conlcusions IV Positive feedback despite all the criticism that an evaluation has to reflect we can conclude the usability evaluation of the Rural Wings Project with positive comments of users on the RW-project: “Thanks to this project we were able to contact other different realities and to share experiences and knowledge gained with them.” “It is an extraordinary help for rural localities such as ours.” “It's good to feel myself as a part of international project.” “It is a great project that allows the educational rural communities to have access to Technologies of Information and Communication.”
Brussels, 18. March 2010 38RURAL WINGS IP Thank you for your attention. Prof. Dr. Thomas Köhler Technische Universität Dresden Media Centre http://mz.tu-dresden.de Phone: +49-(0)351-463-32772 Fax: -463-34963
Brussels, 18. March 2010 ANNEX A Annex A usability ratings for RW applications
Brussels, 18. March 2010 40RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 41RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 42RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 43RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 44RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 45RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 46RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 47RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 48RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 49RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 50RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 51RURAL WINGS IP usability RW applications
Brussels, 18. March 2010 usability RW applications
Brussels, 18. March 2010 usability RW applications
Brussels, 18. March 2010 usability RW applications
Brussels, 18. March 2010 usability RW applications
Brussels, 18. March 2010 usability RW applications
Brussels, 18. March 2010 Further details heuristic evaluation I Issues of 2007 that still remain In a second phase of the evaluation process the issues that were identified during the first evaluation of the CAP website in 2007 were reassessed. General problems website suggests depth, but there merely are link lists still triggering wrong expectations scope has to be made clearer still a problem, e.g. labels, country/user group specific adaptations navigation seems oversized in relation to available content information cannot be accessed quickly enough overall structure ok, but still no breadcrumb-type of navigation 57RURAL WINGS IP
Brussels, 18. March 2010 Further details heuristic evaluation II Missing features „help“-page/function should be added and links to it on every page ok, but probably still more help needed by ordinary users, English only(!) FAQs should be added and updated on a regular basis not implemented, but less serious Structure structure is not clear/consistent issue remains, s.a. navigation and paths should match! paths seem to have been eliminated - not the preferred solution Design / Layout layout will have to be improved (i.e. home>All rural Wings>learning@work) issue remains, s.a. 58RURAL WINGS IP
Brussels, 18. March 2010 Further details heuristic evaluation III Broken Links / missing attributes corrected, but new ones occur – cf. Annex B of evaluation report Other Features that should be improved external and internal links have to be distinguishable serious issue still remains, only solved in sitemap final URL should be international (.org or.net) and easier to remember than http://ruralwings.rd.forthnet.gr again considered a good idea, but not implemented wording should be target-group-specific (i.e. tools for students) very good idea, not implemented 59RURAL WINGS IP
Brussels, 18. March 2010 ANNEX B Annex B Additional material
Brussels, 18. March 2010 61RURAL WINGS IP usability evaluation Usability Evaluation Usability of user interfaces is one narrow concern of the overall system acceptability. Figure 1 depicts Nielsen’s Taxonomy of System Acceptability, showing the relative placement of usability (Nielsen 1993, p.25).
Brussels, 18. March 2010 62RURAL WINGS IP evaluation focus What will be evaluated? utility vs. usability: utility is the question of whether the functionality of the system in principle can do what is needed, usability is the question of how well users can use that functionality. Thus, the usability evaluation of the rural wings project will not investigate, if the services offered are suitable to meet the user needs, but how well end users and content providers can use the provided interface.
Brussels, 18. March 2010 63RURAL WINGS IP research questions I Evaluation focus: usability of the RW “common access point” (CAP) and RW platforms and services. This includes Internet access services (hybrid: satellite-WiFi system), documentation and training materials and installation and maintenance procedures. Central research questions are: 1.How well can users access and use the functionalities of RW CAP and the applications? 2.How is system usability of RW CAP and applications rated by end users in terms of the most important criteria of system usability ( Learnability, Efficiency, Memorability, Errors,Satisfaction) ?
Brussels, 18. March 2010 64RURAL WINGS IP research questions II 3.How satisfied are end-users with the Rural Wings Services and Infrastructure, i.e. the installation services, the availability of the Rural Wings internet service, the reliability of the WiFi client equipment, the reliability of the Satellite terminal equipment, the technical support for the satellite and WIFI infrastructure, training and introduction to the rural wings infrastructure, platform and applications? 4.What questions, problems and needs arise during the training of the end users (i.e. are additional trainings required for specified pilot sites and/or contexts? Do specific target groups have special needs in terms of support, content or competencies?)?
Brussels, 18. March 2010 65RURAL WINGS IP coaching study B.2.2 Coaching Method The Coaching Method (Mack and Burdett 1992) includes explicit interaction between the test subject and the coach. During a coaching study, the test user is allowed to ask any system- related question of an expert coach. Coaching thus focuses on the novice user and is aimed at discovering the information needs of such users in order to provide better training and documentation, as well as possibly redesigning the interface to avoid the need for questions (Nielsen 1993). The coaching method can relatively easy and without large expenses be integrated into the workshops that will be held to train the users and/or multiplicators (Task 6B.2) and can in addition serve to evaluate the provided support material (Task 6A.4).