1. Effective? use of VITAL (in Engineering) Tim Bullough, Engineering with thanks to: John Marsland, Electrical Engineering and Electronics Mark Johnson and Richard Dodds, Engineering University of Liverpool L&T Conference June 06

2. QUESTION What Faculty are you from? A. Arts B. SES C. Science D. Medicine/Vets E. Engineering F. Teaching Support (CLL…) G. Other Press your response, then press SEND 1/1

3. Blackboard (VITAL) is a VLE (virtual learning environment) Module information Learning resources weblinks interactive self-study lecture handouts/notes Communications discussion forums file exchange online chat Online assessment tests and surveys formative and summative feedback to students automated marking and analysis student evaluation Module management student registration student tracking resource control group-work

4. In theory a VLE should: allow individual and self-paced learning direct students to lots of data/information encourage student-student interaction (peer learning) allow testing of student understanding provide guidance when errors or misconceptions are found be cheaper and less time-consuming than conventional teaching Still not clear to e-learning designers or teaching staff: what students expect from e-learning how to motivate students to learn using the technologies how to bring in human interaction. how to blend with conventional teaching

5. QUESTION Do you use VITAL for teaching? A. Yes (a lot) B. Yes (moderate) C. Yes (very occasionally) D. Not at all E. don’t teach Press your response, then press SEND 1/4

6. QUESTION What does VITAL stand for? A. Very Irritating Test for fAiling Lecturers B. Virtual Interactive Teaching at Liverpool C. Version One To Aid Learners D. Vital Initiation Tool for Archiving Lectures E. Don’t Know Press your response, then press SEND 2/4

7. QUESTION What software does VITAL use? A. WebCT B. Whiteboard C. Blackboard D. LUSID E. Access Press your response, then press SEND 3/4

8. QUESTION What topic are you most interested in? A. Formative assessment B. PBL C. Group/teamwork D. Student feedback/evaluation E. All of the above Press your response, then press SEND 4/4

9. (Very) short history of VITAL/Blackboard at Liverpool WebCT trail in 02/03 Launched VITAL(Blackboard) in May 03 Started properly in 03/04 session Peak monthly usage: SessionActive users (unique)Modules 03/04 9572 (Mar 04) 3512 04/05 12453 (Nov 05)4181 05/06 14329 (Jan 06)5006 (24.5million hits) Staff using VITAL: BioSciChemEnggGeogHealth LawSociolMangt Vet Mar 04:25202323-25224121 Mar 05:482832214720244427 Mar 06:634554326026265848

10. VLEs: A National Picture UCISA/JISC national survey 2005 into VLEs in HE: VLEs in ~95% of institutions Blackboard (43%) and WebCT (37%) dominate Subject take-up (pre-92 HE): Business/Law (85%) English/History/Psychol/Sociol (72%) Languages (70%)…….. Medicine/Engineering/Phys.Sci (52%) 46% of respondents do more than just post lecture notes and weblinks

11. Barriers to development of e-learning: 1. lack of time 2. lack of money 3. lack of support staff 4. lack of academic staff development 5. lack of incentives/career development Very little concern about technical issues, student demand, institutional strategy etc. VLEs: A National Picture UCISA/JISC national survey 2005 into VLEs in HE:

12. Engineering in HE: Engineering students…. in Yr 1: Don’t know how to self-learn Don’t know how to access/evaluate information Not comfortable mixing maths and physics Have no idea at all about how well they are learning at University - but they really want to know! Engineering programmes: High direct contact hours (20 lectures/week + labs) Lecture-based modules compete with 100%-coursework modules for the attention of students A little bit about Engineering……. Industry skills/issues….& Management Teamwork & Problem solving Applied Maths/Physics topics Engineering Design +

13. Effective? use of VITAL (in Engineering): 1.Formative assessment 2.Developing a Student Case-Study: Sustainable Development 3.PBL and group-work 4.Collecting student Feedback & Evaluation 5.Other things: recorded lectures, e-lectures, summative assessment

14. QUESTION Which quotation do you think best describes “Formative assessment” …? A. A range of …. assessment procedures ….. to modify and enhance learning and understanding. B. A process to recognise and respond to student learning in order to enhance that learning, during the learning. C. The provision of information as feedback to modify the teaching and learning activities. D. Providing the capacity for students to monitor the quality of their own work during its production, so they can improve it. E. Any assessment designed to promote further improvement of student attainment Press one response, then press SEND 1/2

15. Press response, then press SEND QUESTION Have you used the VITAL assessment tool in your teaching? A. Yes (auto marked, formative) B. Yes (auto marked, summative) C. Yes (manually marked, text responses), D. No (but have been considering it), E. No (can’t see it is relevant to my teaching) F. Don’t teach 2/2

16. 1.Formative assessment (me and John Marsland)

17. Feedback-assisted learning and self-diagnostic: scores do not contribute (much) to grades smaller, more frequent assessments detailed (immediate) feedback to every student tutor and students can monitor progress students can take the tests as often as they wish to assist learning and revision What is formative assessment?

18. Benefits (in theory) anywhere anytime students work at their own pace enables frequent small tests provides rapid feedback can include multimedia etc analysis of results automated eliminates marking burden/errors scales well for large groups Limitations investment of time & effort cannot test “higher order” knowledge requires access to networked PC staff need to learn new skills for summative assessments: reliability, security and validation Computer-aided assessment

19. Assessment in Blackboard (VITAL) Types of questions: True/False (easy, but of limited use) Multiple Choice (good for simple recall of facts, and calculations) Multiple Answer (tests recall and relationships) Ordering (tests recall of sequences, but difficult to think of questions) Matching (same as mcq. Can be used for feedback) Fill in the Blank (facts and spellings, but need to be careful to include all variants) Calculated formula (question includes an equation with different random variables for each student) Hot Spot (student has to point to specific area on image) Essay/Short Answer (text box - needs manual marking) Include figures, attached documents, web-links, videos, etc….. Randomise questions (from pool?)?

20. Formative assessment using VITAL: what’s been tried in Engineering? Both taught traditional Yr 1 engineering lecture-modules, with one “class-test” or paper-based “home-work” problem sheet around wk8 Major coursework marking burden for 100-300 students in Yr1 classes More importantly, wanted to improve student engagement with taught material during the semester, not just in the week before the exam. + Give new Yr 1 students a better idea of their understanding? BUT How much time & effort to set-up, mark and maintain? How do you get students to do the assessments? Do you provide academic support to any students who have problems? (Me and John Marsland)

21. Year 1 Structural Materials (265 students) 2003/4: ~100 mcq’s put into VITAL as “Practice Questions”  But only the best students used practice questions! Single mid-semester class-test, and mcq exam  no better than before 2004/5 onwards: Replaced paper class-test with four VITAL assessments (5% each). Tests 1 and 2: knowledge/understanding of technical information. Tests 3 & 4 include some real engineering problems. Retained Practice Questions, but linked explicitly to the formative tests Pre-2003/4: No VITAL. Single mid-semester paper class-test, and mcq exam. (me)

22. Create content area for the assessments

23. encourage peer discussion ~2 weeks to complete test Attempt ALL questions (negative marking?)? Scaling? Save and resume? Ask for feedback Give clear instructions Delay score-return by 1 week

24. Types of questions: factual knowledge….. decide a point value for each question Effective questions? clear, succinct questions and responses don’t give away the answer (to other questions) avoid ‘negative’ questions? are the wrong responses plausible (esp for calculations)? avoid negative questions?!

25. Types of questions: simple numerical questions…. Effective questions? Are the wrong responses plausible

26. Include images and links to websites….

27. Language subtleties and over-complicated questions can be impossible for non-native English speakers:

28. Optional and instantly marked >120 questions don’t count towards module mark marked instantly ~50% of practice questions are "directed related" to test and/or exam questions Practice Questions…. to replace direct tutor support

29. Attempt rate for Practice Questions >60-70% Practice Questions (cont.)

30. Detailed worked solutions to numerical questions provided after test marked tutorial support (cont.)

31. MARKING automatic and IMMEDIATE when test submitted: returns “score only”, “score and correct answers”, “score and feedback”, UNLESS…… Include a (0-point) short-answer text feedback question: delays disclosure of score and provides feedback Release marks manually later on…

32. Submission rate for formative VITAL tests 90-95% Missing marks are often students who have left the Univ, NOT non-submissions Gradebook: Alphabetic “Grade List” shows all students’ scores for all tests: individual student's responses Summary of all responses to each question in Assessment Attempt Details  …..

33. Assessment Attempt Details: provides feedback about level of whole class understanding:

34. Exam Start of sem Test 1&2 Test 3 Test 4 Did it work? Good test submission (95% in 05/06) and Practice Question usage (~70%) Most usage just before test deadlines: Regular feedback comments VERY informative: discussed in class Response data indicates topics they are having problems with

35. …..Probably added about 5% to average exam performance 03/04: 1 class-test + optional VITAL practice questions Exam 41.5% 04/05: 4 VITAL tests + linked Practice Questions Exam 49.5% Exactly the same exam! Problems: Need to be very careful with language, esp for Yr1 sem1 OS students “work-around” to temporarily withhold marks on test submission (VITAL Equation Editor not very good for Engg) Did it work? EFFORT: 2-3 weeks to set up questions, and ~2 days each year to maintain

36. Now: 8 x weekly VITAL mcq-type 10-question assessments Each test worth 2-3% Best to allow each test to be taken at any time over 3 week period Students told to work alone, but working together ok Return “score only” instantly on submission of test. Correct responses available 3 weeks after deadline. No other feedback/help given/needed. Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students Year 1 Electronic Circuits and Devices 80 students (John Marsland) 2003/4: One in-term paper-based homework assignment - lots of marking. Relatively poor exam performance: little student engagement with lecture material.

37. Problems: Late/incorrect registrations in year 1 sem 1. (Some Equation Editor problems). Did it Work? Excellent completion rate: ~90% submission VERY positive student feedback Interaction with students probably greater than if no VITAL for large classes: students view it as active participation Excellent for tracking student retention/drop-out in Yr1 sem1 EFFORT: about 1 week/module to set up questions, ~1 day/session to maintain Year 1 Digital Electronics 250 students Year 1 Audio Electronics 50 students Year 1 Electronic Circuits and Devices 80 students (John Marsland)

38. “Engineering students are fundamentally lazy”. Do the formative assessments really improve student performance, or is it the threat of failure in any assessment that motivates and rewards them so they attend more lectures and labs, and do a bit more private study? …. a few final comments on using Blackboard (VITAL) for (formative) assessment: are there educational benefits? [yes] are there any other advantages (financial, time)? [yes] are there problems? [yes] does it integrate with the rest of the module? [yes] how much additional effort is required... to create the online assessments and surveys? [quite a lot] to manage their use? [a little]

39. 2. Developing a Student Case-Study: Sustainable Development (Richard Dodds and me)

40. QUESTION In what year did the first definition of sustainable development appear? “to ensure that it meets the needs of the present without compromising the ability of future generations to meet their own needs.” (The Brundtland Report) A. 1887 B. 1922 C. 1964 D. 1976 E. 1987 Press response, then press SEND 1/3

41. QUESTION Which one of the following is not one of the "Five Principles" of sustainable development (UK Govt 2005 Strategy, Securing the future) A. Living within environmental limits B. Achieving a sustainable economy C. Promoting good governance D. Using sound science responsibly E. Ensuring a strong, healthy and just society F. Minimising pollution and waste Press response, then press SEND 2/3

42. QUESTION Have you ever used the VITAL Assignment Tool and/or Word templates? A. Only the Assignment Tool B. Only Word templates C. Both Assignment Tools and Word templates D. Neither (but I know what they are) E. Neither (don’t know what they are) Press response, then press SEND 3/3

43. Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students) The Challenge for 2005/6 (I took over as coordinator) Retain individual assessment Wanted on-going individual formative feedback to improve quality of final document Marking load limit set at ~10 mins/student, with just one pg student and 1 academic staff. Individual paper-based “structured essay”, based on four lectures. Only about 75% submission (typically 4-10 pages). VERY disappointing report (both technical content and style). VERY heavy marking load (6 academic staff). Pre-2005/6

44. Approach taken VITAL Assignment Tool and WORD Template (drop-down menus and limited text areas  standard format) Word template issued via VITAL in wk1 and resubmitted part- filled using Assignment Tool in 3 weekly stages, before FINAL version submitted.template Nominal marks (5%, 3% or 0) at each stage Optional feedback comments - provided for all at penultimate version Electronic + hard-copy retained for marking FINAL versiong FINAL version Teaching of Sustainable Development in Design to all Year 1 Engineers (250 students)

45. Submit part-completed case-study each week…….. word template downloaded by student

46. part-completed word template doc uploaded as attached file

47. part-completed word doc for marking mark feedback Assignment grading tool opened from gradebook…..

48. final doc submitted after 4 weeks

49. Advantages of process 80-90% submission at each stage: easy and quick for pg to mark Feedback comments possible (but takes time!). Used for penultimate version or when student wildly off-track. Much higher quality FINAL case-study (technically and structurally). Download all submitted work as a record (and for safekeeping). Weekly mark in VITAL (+comments) useful feedback to students: but also discuss general issues in subsequent lecture. Suggested Improvements/problems Not all students use feedback comments before doing current version? Easy to accidentally delete all submitted documents! Chinese character filenames

50. 3. PBL and group-work in VITAL

51. QUESTION Which of these is NOT one of the 7 stages in a typical Problem Based Learning exercise? A. Make sure everyone understands the problem precisely B. Establish what members of the group already know about the problem C. Formulate the questions which need answering to enable the problem to be solved D. Go away and gather all the knowledge E. Check to see if the problem can be solved, and if not, reformulate new questions F. Reflect upon the experience G. Identify which students have not been pulling their weight, and shoot them. Press response, then press SEND 1/3

52. QUESTION What is the main problem with PBL from a student's viewpoint? A. They don't think they are learning anything unless they hear it in a lecture/seminar B. They don't like group-work because they have to rely on other students C. They find it difficult to manage their own time D. They don't like group-work because it may not be fairly assessed E. They don't know how to critically evaluate information F. Something else G. Don't know enough about PBL to comment Press response, then press SEND 2/3

53. QUESTION Have you used "Groups" and/or run a PBL activity in VITAL? A. Just "Groups" B. Just PBL C. Both Groups and PBL D. Neither, but do use group-work and/or PBL in teaching E. Neither Press response, then press SEND 3/3

54. Year 2 Materials Design PBL AIMS: Evaluate and solve a real engineering/science/design “problem”, with real current industrial relevance: 21 st Century Autobody Steel Each group acts as a team of (materials) engineers in the steel industry charged by their company’s Executive Board with producing a response to a very specific “threat” of aluminium replacing steel in the automotive marketplace (….. “The Problem”). Mimic group-working industrial practice and procedures, as far as possible. Make use of “industrial” specialists, input and visits Design The Problem to utilise excellent internet learning resources produced by MATTER

55. PBL Process: Groups of ~6. One 50-minute facilitated formal group "committee" meeting each week (chairperson, minutes secretary etc.) VITAL: Becomes the Virtual Steel Company’s intranet. Database of company resource documents Agenda and Minutes (incl Actions) for each meeting - keeps tutor aware of progress. Limited access (3 questions/week) to “virtual consultants” via the discussion board - technical support and steering groups. 2001/2: standard lectures and a group report 2002/3: PBL. Use WebCT for group activities (Agenda and Minutes of meetings etc) and tutor interactions 2003/4 onwards: PBL. Use VITAL for group activities, tutor support via "virtual consultancy", and test of technical understanding. Specific assessment of group's effective use of VLE.

56. Company resource database

57. Each group has its own site

58. technical support each week Agenda, Minutes, web-resources, docs…. Effective use of the VLE is 20% of assessment! Bainite group site

59. 3 questions per week allowed real expert used insist on professional approach Virtual Consultancy

60. Individual assessment in VITAL utilised to steer students to web-resources and technical issues….

61. Positive outcomes VITAL provides excellent framework for PBL tracking Virtual “consultants” provide tutorial support Always positive comments from facilitators and students Minor problems Some group functionality limited (no tests in group sites, and can’t rename some group activities). Intra-group peer-moderation too cumbersome in VITAL - used paper.

62. 4. Collecting student Feedback & Evaluation informally at module level formally at Year/Programme level

63. QUESTION Does your Department have a Dept-wide general VITAL site for all students? A. Yes B. No C. Don't know D. Not in a Dept Press response, then press SEND 1/2

64. QUESTION Does your Department use VITAL to obtain student feedback? A. Yes, at module level B. Yes, at programme/year level C. Yes, regularly for various issues D. Not aware E. Not associated with a Dept. Press response, then press SEND 2/2

65. 4. Collecting student Feedback & Evaluation informally at module level: use the formative assessments formally at Year/Programme level

66. Use a (0-point) short-answer feedback question at the end of every formative assessment:

67. Does it work? 140 out of 250 submitted assessments included some comments

68. Can’t always do much about the issues - spend 15 minutes in a lecture discussing the issues

69. 4. Collecting student Feedback & Evaluation informally at module level: use the formative assessments annually at Year/Programme level: trial this session

70. 1026 users Used our Department of Engineering VITAL site:

71. Set up an Annual Programme Review content area:

72. Divide into MSc(Eng)/MRes pgms & 6 BEng/BSc/MEng pgm groups:

73. Each ug pgm group divided into 3-5 years, and BEng/MEng: Use same Annual Pgm Review SURVEY in each content area….

74. Identical 10 questions for every pgm/year  43 identical surveys: all responses anonymous (no cross-correlation)

75. 1. Question Refering to the modules you have studied this session, please tell us which module(s), or specific parts of modules, you feel have been taught particularly well, and give your reasons. (Your programme structure, giving module codes and titles, can be viewed via the link below. Please give both the module code(s) and title(s) if possible) Programme Structures UG 05/06 Programme Structures UG 05/06 2. Question Refering to the modules you have studied this session, please tell us which module(s), or specific part of modules, you feel have been taught poorly, and give your reasons and suggestions for improvement. (Your programme structure, giving module codes and titles, can be viewed via the link below. Please give both the module code(s) and title(s) if possible) Programme Structures UG 05/06 Programme Structures UG 05/06 3. Question Please give an overall rating of the quality of teaching you have experienced during the current session (1=excellent, 5=very poor): 4. Question What are your views on the quality of the lecture theatres and teaching rooms? Please specifically name any lecture theatres that you wish to mention. 5. Question What are your views on the Department's Student Support Office in the Derby building 6. Question What are your views on the general quality of the information you are given during the year (such as timetables, emails about changes to lectures, student handbooks, VITAL announcements, careers/vacation work information etc)? 7. Question What things have helped you most with your learning this session (examples could include...library resources, PC access at University/home, use of VITAL, access to teaching staff, personal development planning etc)? 8. Question What things have made learning difficult during this session? (examples....lack of background knowledge, having to work, poor accommodation, illness, social/sport commitments etc, and the examples in the previous question)? 9. Question Please give an overall rating of how much you have enjoyed your studies during the current session (1=very enjoyable, 5=not at all enjoyable): 10. Question What actions would you suggest the Programme Director or Department should consider in order to improve the current year of your degree programme for future students? Annual Programme Review Questions….

76. Mainly text responses:

77. Rating-questions for year-on-year, or pgm-to-pgm comparison:

78. Access survey feedback comments via the gradebook: 1026 students 43 surveys

79. What sort of information do you get? Trial in 05/06: 3 emails over 4 weeks either side of Easter sem2 ~20% response rate from Yr 1, decreasing to ~5% in Yr 4 Need someone to process comments Need a strategy in place to deal with all the information Did it work? What do you do with the information? Year 1 Aerospace feedback… Year 1 Aerospace feedback…

80. 5. Other things: recorded lectures, e-lectures, summative assessment

81. Potential advantages: Learn at one's own pace, any time and place Repeat a lecture for revision Optimised delivery every time Save time spent delivering lecture Potential for highly flexible timetable and on-line courses Potential disadvantages: Lack of life/spontaneity in e-lectures. No opportunity to ask questions - vital to have tutorials/problem classes. Lecturers could be redundant! …or more staff time spent interacting with students in tutorials etc. e-lectures: ppt slides with voice-over commentary (Mark Johnson)

82. e-lectures: ppt slides with voice-over commentary Used NO live lectures at all - but additional tutorials/problem classes - for past 2-3 years for Year 4 CFD module (25-30 students) and Year 1 Thermo module (about 150 students). Use highly-animated powerpoint slides, with synchronised voice track (prepared using the powerpoint software) Takes about a day (8-10 hours) to convert a one hour lecture (i.e. payback only after 8-10 years!) about 50% preferred e-lectures and about 50% live-lectures: no indifference… either loved them or hated them! exam performance in 2005 and 2006 was as good as, or better than, before Sept resit exam performance even better Did it work? The Process:

83. Conclusions 1.Formative assessment 2.Developing a Student Case-Study: Sustainable Development 3.PBL and group-work 4.Collecting student Feedback & Evaluation 5.Other things: recorded lectures, e-lectures, summative assessment

84. QUESTION How many VITAL hits in Jan 06? A. 2 450 B. 24 500 C. 245 000 D. 2.45 million E. 24.5 million F. 245 million Press response, then press SEND 1/2

85. QUESTION How would you rate this presentation/session? A. A complete waste of my time B. Useful and interesting in small parts C. Mostly useful and interesting D. Exactly what I came for E. Would rather not say Press response, then press SEND 2/2