1. E VALUATION F RAMEWORK (WP5, T5.1, D5.1) C atania, September 9-11, 2009. Manuele Manente, Q uality manager.

2. S UMMARY Evaluation of the second test and comparison with the first one. 1. Users’ needs analysis 2. Usability of the prototype 3. Test1 and Test 2 Comparison Estimated time: half an hour. Evaluation framework 2

3. U SERS ’ NEEDS ANALYSIS Evaluation framework 3 Target group comparisonFirst testSecond test Number and gender21 learners / 3 women and 18 men15 learners / 6 women and 9 men Age average30.321.1 Learning Estonian for (number of years) 19 mother tongue/ 2 foreign language (studied for 5 years) 4 mother tongue/ 11 foreign language (studied for 11,5 years) Evaluator comments: 1.The number of participant is reduced, the group is limited for representativeness. Gender equity is improved. 2.The target group is clearly made of younger, so they should have a better approach to ICT, digital literacy - no digital divide. 3.The target group is composed mainly of non-Estonian mother tongue, rightly in line with project aims, even if the target group have an high level of Estonian grammar knowledge (average: 11,5 years of studies).

4. 4 Evaluation framework U SERS ’ NEEDS ANALYSIS PART A: FACILITIES Facilities are appropriated according to users’ learning needs and to the e-learning methodology planned. All user working station are connected to internet. Thus, the course can be also use on-line (not only by CD-ROM). No particular differences with the first test.

5. U SERS ’ NEEDS ANALYSIS Evaluation framework 5 PART B: EXPECTATION The majority of the target group think that e- learning is good but with a certain “distrust”. The most of them are available to carry out a whole e-learning vocational training experience. Target group expect to receive paper copies of guidelines and booklets.

6. U SERS ’ NEEDS ANALYSIS PART C: BENEFITS Most of them think that an e-learning course can improve their communication skills. The group thinks that the e-learning for the language learning is also useful for the non EBBG member. The use of e-learning in the workplace is very useful to improve professional skills. Evaluation framework 6

7. USABILITY OF THE PROTOTYPE 7 Evaluation framework PART A: NAVIGATION Users would like to have a tools enabling the printings of content. We suggest a virtual printing tools (PDF), reducing the footprint and paper printing. (n.8) Students complain that interrupting an e-lesson, at the new access, they have to re-start from the beginning. The prototype does not save users results and progress. (n.11). It happens because the course is accessible by the web and not through a LMS. In any case it can be exported in SCORM and up-loaded in an e-learning platform. The whole navigation is easy, and intuitive. It can be added an on-line guide about the course navigation (the help tool exists but it is empty).

8. 8 Evaluation framework USABILITY OF THE PROTOTYPE PART B: INSTRUCTIONAL DESIGN Users underline the lack of audio/video learning activities to improve listening and comprehension skills (n.17, 18, 19). It is important to establish if the project methodology needs this kind of activities (in relation to project aims and budget). The instructional design is very good and in line with the educational aims. The Images are fully representatives of the educational scenario. There is an adequate number of learning activities and Edu-Games with user-interaction. Very good the idea of cross-words. Please send us the solution ;-)

9. 9 Evaluation framework PART C: CONTENTS Some users think that sentence and linguistic style is not so easy and clear. (n.6) The educational material does not take account of cultural differences or gender. (n.13) The overall content are logically organised and exhaustive (learning on the job – n.4). Very appreciated the glossary. USABILITY OF THE PROTOTYPE

10. 10 Evaluation framework USABILITY OF THE PROTOTYPE PART D: EFFECTIVENESS OF LEARNING Learning activities are highly appreciated. (n.7) Some improvement can be done concerning the feedback to learners (e.g. score and improvements). The overall course is constructive and useful to address project educational aims.

11. 11 Evaluation framework USABILITY OF THE PROTOTYPE PART E: OVERALL Target users are happier compared to their initial expectations, and they will recommend the course to a colleague. The educational experience is mildly pleasant (n.3), but it not replace the classroom training. Duration: 30-60 min.

12. Evaluation framework 12 TEST COMPARISON

13. T HANK YOU FOR YOU ATTENTION manente@jogroup.eu 13 Evaluation framework