1. LEARNING TECHNOLOGY DEVELOPMENT UNIT Learning Technology & the Evaluation of Learning Outcomes Andrew Oliver

2. LEARNING TECHNOLOGY DEVELOPMENT UNIT Evaluation Techniques Educational effectiveness: ability to impart knowledge and understanding Techniques :  Quantitative (pre / post testing)  Qualitative (questionnaire, interview) The method used depends on the context Ideally one method supplements the other

3. LEARNING TECHNOLOGY DEVELOPMENT UNIT Pre & Post Testing When?  directly before and after (control group)  exam performance (compare with previous year ) But  time between exercise & assessment  knowledge gain through other methods, (books, tutorial, revision)  varying levels of validity

4. LEARNING TECHNOLOGY DEVELOPMENT UNIT Questionnaire - Usability Common usage: usability of the technology Example  Did you find it difficult to move from Section A to B?  Were the download times acceptable?  Did you enjoy the exercise? Relates to pedagogical design

5. LEARNING TECHNOLOGY DEVELOPMENT UNIT Questionnaire - Educational Rarely educational: student self rating (Likert) Example  By how much did you feel the application increased your knowledge  How much did you feel the Worked Examples helped your understanding of the topic Key word “feel” - accuracy & honesty

6. LEARNING TECHNOLOGY DEVELOPMENT UNIT Questionnaire Reality  mixture of Educational and Usability  impression of attitude towards technology  idea of the learning environment  hence context in which the application works best

7. LEARNING TECHNOLOGY DEVELOPMENT UNIT Phenomenographics Quantiqualitative!  analysis of student (mis)conceptions of the topic  builds up a conceptual student model Why?  future design (encoding feedback)  increased quality interaction

8. LEARNING TECHNOLOGY DEVELOPMENT UNIT Phenomenographics Methods  interviews  observation  recording group work  examination of test scripts More formative – takes place during development

9. LEARNING TECHNOLOGY DEVELOPMENT UNIT Problems, Problems Problems associated with each method Pre & Post Testing  Preparation time (new questions)  Timing  Co-operation (staff & student)  Not anecdotal  Control group (selection & reliability)  More time (interpretation)

10. LEARNING TECHNOLOGY DEVELOPMENT UNIT Problems, Problems Questionnaire  Data limitations (banding & differentiation)  Self rating (accuracy and honesty) Are students honest?….depends…  Anecdotal (poor trends)

11. LEARNING TECHNOLOGY DEVELOPMENT UNIT Yet more… Logistics  Timing (student workload, availability) – can you wait?  Resources (workstations, rooms, network)  Liaison (network admin, LRC) – cooperation

12. LEARNING TECHNOLOGY DEVELOPMENT UNIT Will it ever end? Does it do what the label says it does?  measure what its supposed to measure? (learning outcomes)  measured accurately?  any outcomes not measured? Time needed re design the test items (Bloom’s taxonomy)

13. LEARNING TECHNOLOGY DEVELOPMENT UNIT Context of Evaluation The intended context of use Stand alone resource:  Evaluate immediately after exercise  Pre & post test  Control group Integrative approach:  Questionnaire

14. LEARNING TECHNOLOGY DEVELOPMENT UNIT Context of Evaluation Relate the context of the evaluation to the intended educational setting of the technology

15. LEARNING TECHNOLOGY DEVELOPMENT UNIT So Which Method’s Best? (or what can you realistically expect at the UH?) Ideal  pick n mix approach – one method augments the other  but time required (design, analysis) Realistic  questionnaire (least time, user friendly, mix of quant/ qual)  determine attitude towards IT and logistical problems – curriculum planning

16. LEARNING TECHNOLOGY DEVELOPMENT UNIT The Obligatory Case Study  Heriot-Watt University: ESP  (naively) assess student attitude towards IT  pigeon holes - very low response  ambush during lecture – but low attendance  demand notes - ignored But the questionnaires were only a part of it…

17. LEARNING TECHNOLOGY DEVELOPMENT UNIT The Obligatory Case Study … the main exercise  evaluation of learning package (ESP)  pre & post testing and questionnaire But  low attendance possible  hence compulsory …but…assessment aspect  disguised exercise as a “lab” Unpopular but rich in data

18. LEARNING TECHNOLOGY DEVELOPMENT UNIT Ways around it (what you have to do to get what you want )  pre / post test: disguise the exercise pre & post testing and questionnaire  bribery: marks awarded for attendance  questionnaires: bribery or ambush or both Finally  no one method gives the complete answer  use a variety and your experience / intuition

19. LEARNING TECHNOLOGY DEVELOPMENT UNIT LTDU web resources / references  Questionnaire item database  LTDU website: http://www.herts.ac.uk/ltduhttp://www.herts.ac.uk/ltdu References:  Bloom, B. S. (ed.), 1956. Taxonomy of Educational Objectives, Handbook 1: Cognitive Domain, David McKay Co: New York  Gronlund, N. E., 1971. Measurement and Evaluation in Teaching, Macmillan Co: New York  Laurillard, D., 1993. Rethinking University Education, Routledge: London