1. Aplication of CAS in automatic assessment of math skills Przemysław Kajetanowicz ( przemyslaw.kajetanowicz@pwr.wroc.pl) Jędrzej Wierzejewski (jedrzej.wierzejewski@pwr.wroc.pl) Wrocław University of Technology

2. Agenda Knowledge assesment in math instruction Automatic assessment in algebra E-course: history, functionality, implementation, results and students’ feedback How automatic tests work Recent developments: Java+Mathematica- based tests Future

3. Assessment of progress in math Measuring progress in math = measuring mastery in problem solving. Grading of math problems –Grading procedure usually reflects typical solution structure –Common though not always applicable: partial credit for partial solution Assessment tools in computer-aided learning systems –LMS-dependent –Stand-alone

4. E-course in algebra - overview Content structure –Lecture notes –Interactive exercises (Java-driven) – over 120 problem types supported –Practice tests (Java-driven) –Graded exams (Java-driven) Functionality of assessment tools –Random generation of data –Controlling the difficulty level at design time –Flexible grading (partial credit) –Step-by-step solution presentation –Completeness check –Initial correctness check

5. Lecture notes (1) Studying… … and exploring math

6. Lecture notes (2) Studying… …and self-testing

7. Exercises Exercise…...and solution on demand

8. Graphing problems and related tools Functionality of tests (1)

9. Functionality of tests (2) Dedicated tools for Gauss elimination

10. Functionality of tests (3) Correct solution immediately available

11. Completeness check A student can go back to test as many times as he/she wishes.

12. Initial correctness check Chances to make corrections (enthusiastically greeted by students) – teachers decides on # of chances

13. E-course - implementation Spring 2005 - 55 students Fall 2005 – 400 students Fall 2006 – almost 1000 students Hybrid instruction –Classroom meetings (3 or 4 hours weekly) –(Fall 2006) grading based on 5 online e-exams (play the role of homework) Final in-lab e-exam (proctored) In-class activity (not much credit though)

14. Administrative grading system Available credit –5 online e-exams: 5 problems, 1 hour, 25 points available; best four exams count –In-class activity – 4 points available –In-lab final exam: 8 problems, 90 minutes, 40 points available Grading procedure based on student’s systematic work –A minimum of 20 on the final necessary (but not sufficient) to pass –20% of online e-exams total score added to final (available maximum = 20) –In-class credit added (maximum = 4) –Necessary total to pass = 31

15. Grading system in practice Required minimum on final Required total to pass

16. Fall 2005 – Final grades

17. E-course results vs. traditional course (data after make-up exams) Grade Percentage of students Fail48.72% Satisfactory6.41% Satisfactory +17.95% Good3.85% Good +8.97% Very good14.10% E-course Grade Percentage of students Fail84.11% Pass15.89% Traditional course (Fall 2005 results)

18. Question 1 To what extent did you find the new form of the course (in particular, automatic tests and exams) helpful in your mastering the course material (as confronted with traditional way of learning)?

19. Question 2 How willing would you be to sign up for other math courses, were they offered in a similar form?

20. Question 6 How fairly, in your opinion, was your knowledge assessed by the system of automatic exams? Note. The exam questions were of similar type to those given on traditional exams. Presently, no statistical comparison is possible between the new and the traditional form in terms of how students judge the fairness (no survey was given to „traditionally assessed students”. The majority of students (74%) judged the system as fair (grades 4 to 6).

21. Students’ comments The course is a new appealing form of study to me. It is stress-free. (…) Giving tests and exams that way implies that the grading system is fair. The course is a great thing, and studying algebra that way is truly easy. Thank you !

22. Student’s comments (2) At the beginning I was terrified at the perspective of taking a course that was delivered in that way, but now I would like to wholeheartedly thank the instructors for the opportunity of being a part of this “experiment”. I can definitely say that thanks to this course I understood things that had been all Greek to me before (...) I truly believe that more courses should be taught that way. I wish to thank again for the opportunity to have been in this course.

23. Automatic grading (single problem) Step 1: initial grading function (hard-coded and problem-specific) assigns raw score p to the submitted solution –p – real number varying from 0 to 100 –Raw score measures „percentage” of solution completeness/correctness Step2: final score P for a problem is computed –2-state grading: if p < 50 then P :=0 else P := 100 –3-state grading: if p < 50 then P := 0 else if p < 100 then P := ½ else P := 1 –Score grading: natural formula applied

24. Inside a test (Java-driven only) Hard-coded –Problem data generation –Solution presentation –Initial grading –Completeness check, initial correctness check –Database-related operations Parameter-driven –Difficulty level, problem sub-type –Scores for individual problems –Number of problems –Time assigned for solution –Specific test behaviour (completeness warning, # of tries to make corrections etc.) –Training vs. administrative purpose of test

25. Java+webMathematica (1) Java: –Formulates problem (generates problem data) –Accepts student’s entry(ies) –Sends data and solution to webMathematica webMathematica: –Compares solution with correct result –Sends outcome back to Java –Generates and sends back other elements (e.g. graphs or expressions needed for solution presentation)

26. Java+webMathematica (solution presentation)

27. Java+webMathematica (2)

28. Java+webMathematica (2 - solution)

29. Java+webMathematica (3)

30. Java+webMathematica (3 - solution)

31. Xml: –Holds problem formulation (parametrized, so many problems of a common type are supported) –Holds solution (parametrized) –Holds problem data sets Java: –Reads xml and behaves correspondingly Mathematica: –Encodes formulation and solution in xml, provides necessary graphics etc. –Generates and encodes data sets in xml Java+xml+Mathematica – version 1 Space-consuming (especially if graphics involved)

32. Xml: –Holds problem formulation (parametrized, so many problems of a common type are supported) –Holds solution (parametrized) Java: –Reads xml –Generates data sets Mathematica: –Encodes formulation and solution in xml, provides necessary graphics etc. –NO data sets generated by Mathematica Java+xml+Mathematica – version 2 Quick and space-saving

33. Java+xml+Mathematica (formulation) 8 2 OY 8 -1/8 2 -1/8 2 Data sets - xml (including correct result) Formulation - xml

34. Java+xml+Mathematica (solution) Solution (one of steps)

35. Java+xml+Mathematica (v. 2) Data sets generation - Java Formulation & solution - xml

36. Java+xml+Mathematica (v. 2) - solution Solution (parametrized) - xml Specific values - Java

37. Pros and cons of automatic assessment Automatic assessment – main highlights –Proves far more effective (see the following slides). –Gives a student more opportunity to demonstrate skills. E-course: 25 problems on online e-exams + 8 on the final Traditional: 2 x 3 problems on mid-term „paper” exams + 5 on the final –Motivates a student to systematic work. –Every student gets different problems. –Saves time. Automatic assessment – downsides –Certain problem types (e.g. „prove that…”) cannot be supported („as yet” - as dr. Wierzejewski says…). –Some students focus on solution algorithms rather than on math concepts and methods.

38. Forthcoming future E-course in Calculus Remedial course in secondary-school math Continuation of teaching the e-course in algebra online Completion of e-course in linear algebra

39. Thank you for your attention