Presentation on theme: "Kumar-Wang/TAMU Assessment Conference/2-23-2009 ASSESSING STUDENT LEARNING AT THE COURSE LEVEL IN SCIENCE COURSES CAN WE RECONCILE LARGE CLASSES, MULTIPLE."— Presentation transcript:
Kumar-Wang/TAMU Assessment Conference/ ASSESSING STUDENT LEARNING AT THE COURSE LEVEL IN SCIENCE COURSES CAN WE RECONCILE LARGE CLASSES, MULTIPLE CHOICE TESTS AND TRUE LEARNING ASSESSMENT? A. Anil Kumar, Ph.D. Head, Department of Physics & Professor, Electrical and Computer Engineering & Fa-Chung (Fred) Wang, Ph.D. Professor, Department of Physics PRAIRIE VIEW A&M UNIVERSITY 9th Annual Texas A&M Assessment Conference The Hilton, College Station, TX February 22-24, 2009
Kumar-Wang/TAMU Assessment Conference/ Assessment is new to physicists. No national or regional accreditations. For physicists assessment is based almost entirely on “peer endorsement.” In a standards based world, that does not work. PHYSICS (PHYSICISTS) & ASSESSMENT
Kumar-Wang/TAMU Assessment Conference/ Our emphasis is on how to define and implement assessment in physics courses. More importantly, how to utilize these results in improving our instruction so that the student is better positioned for learning. In particular, we want to focus on freshman and sophomore courses, where multiple-choice tests (MCTs) are almost the norm. So, the question is: How does one reconcile MCTs, assessment and true student learning? In other words, how do we know that they know? Unfortunately, assessment seems to have diverse definitions. WHAT WE ARE UP TO TODAY
Kumar-Wang/TAMU Assessment Conference/ "Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning.” Huba, M. E. and Freed, J. E. (2000). Learner- Centered Assessment on College Campuses -- Shifting the Focus from Teaching to Learning. Boston, Allyn and Bacon. DEFINITION OF ASSESSMENT
Kumar-Wang/TAMU Assessment Conference/ Discuss the environment regarding MCTs – typical posing of the questions, accepted approaches to answers, myths and practices Preliminary results from our own assessment efforts – pre- and post tests, combining MCTs with Fill-in-the-Blanks, and other techniques Recommendations for future MCTs based on studies of cognitive learning theories OUR PRESENTATION IS IN THREE STAGES
Kumar-Wang/TAMU Assessment Conference/ Pros/Advantages Less time to administer More comprehensive evaluation of extent of knowledge. Increase in efficiency Graded purely on their selections Lower likelihood of teacher bias in the results. Graded purely on their knowledge of the topic MULTIPLE CHOICE TESTING – PROS & CONS Cons/Disadvantages Ambiguity Limited feedback, if ever Learning reduced to “correct” answers Process of analysis difficult to ascertain, at best No training for faculty or students in MCTs Acquired “tricks” from schools to answer MCTs
Kumar-Wang/TAMU Assessment Conference/ If item writers are well trained and if items are quality assured, it can be a very effective assessment technique. If students are instructed on the way in which the item format works and myths surrounding the tests are corrected, they will perform better on the test. If there are large numbers of items on a test, and if good sampling and care over case specificity are employed, overall test reliability can be further increased. … MCTs - IF, IF, IF, …
Kumar-Wang/TAMU Assessment Conference/ EXAMPLES OF “CORRECT” ANSWERS BUT WRONG PROCESS The fraction 64 divided by 16 yields: A)8 B)2 C)4 D)Other ______ Correct Answer: 4 Process: The fraction 95 divided by 19 yields: A)7 B)5 C)9 D)Other ______ Correct Answer: 5 Process:
Kumar-Wang/TAMU Assessment Conference/ Look to see if any of the answers is much longer than the others. The person making the test wants the right answer to be completely true. To make it completely true, they sometimes add more details to the correct answer, making it much longer than the others. The freezing point of water is A)32 degrees Kelvin B)32 degrees centigrade C)0 degrees Fahrenheit D)0 degrees centigrade for pure water at sea level (D) is probably the correct answer. TYPICAL INSTRUCTIONS FOR ANSWERING MCTS
Kumar-Wang/TAMU Assessment Conference/ See if one of the answers sticks out as the "odd one out". Since there is only one correct answer to a question, that answer has to be different from the other answers. So if you see two or three answers that all mean the same thing, they must be the wrong answers. A)Boy B)Girl C)Son D)Lad Since "Boy", "Son", and "Lad" all mean roughly the same thing, chances are the answer is "Girl", even if we do not know what the question was. This trick is dangerous, however. Sometimes the "odd one out" can fool you. Maybe the answer was "Son" because it is the only answer about family relationships. But this trick might help if you are guessing blind anyway. TYPICAL INSTRUCTIONS FOR ANSWERING MCTS
Kumar-Wang/TAMU Assessment Conference/ Which of the following is a mathematician: (A) Ann Richards (B) Susan B Anthony (C) Karen Uhlenbeck (D) Hillary Rodham Clinton THIS EXAMPLE IS FROM AN ACTUAL BOOK ON HOW TO PREPARE FOR TAKS EXAM
Kumar-Wang/TAMU Assessment Conference/ A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is (A)3 miles/hr (B)60 miles/hr (C)180 miles/hr (D)Impossible to determine because the raindrop accelerates due to gravity. SOMETIMES, A STUDENT WHO THINKS DEEPER MAY NOT GET THE “CORRECT” ANSWER!
Kumar-Wang/TAMU Assessment Conference/ “Like so much of our exams and testing industry, the questions have far more to do with helping adults rank and sort children for their own ends than with deep learning.” EDUCATION QUANDARY: 'MULTIPLE-CHOICE QUESTIONS CAN BE HARDER IF A PUPIL READS THE QUESTION CAREFULLY OR HAS A DEEPER UNDERSTANDING. SURELY THIS IS WRONG?’
Kumar-Wang/TAMU Assessment Conference/ Outcomes Assessment for Calculus-Based Physics I (Mechanics)
Kumar-Wang/TAMU Assessment Conference/ Goal To find another form of test-format, other than Multiple-Choice Type. –This other format should have the same advantages of Multiple-Choice: Easy to grade, and little ambiguity in assigning grades. –This other format will offer a better measure of Students’ Quantitative and Transferrable Skills.
Kumar-Wang/TAMU Assessment Conference/ Approaches Giving Test in two versions, one in Multiple-Choice & –another in Fill-in-the-Blanks Format. Compare Students’ performance in these two different formats. Using the results and analysis to modify or improve Teaching-Methods and format for later tests.
Kumar-Wang/TAMU Assessment Conference/ Test Design Started in the fall of 2007, pre- and post-test were given in two types of question: Multiple-Choice type to test simple concept, Fill-in-the-Blanks type to test quantitative skills. the wording and style were similar to questions & problems in text-books. (Giancoli, Serway, etc.) Feedbacks: We were not happy. (a) MC type tends to be on the easy side & FB type tends to be harder. Can we have something in the middle? (b) Students spent a large portion of the test-time in reading 9 problems, some have two or three parts. (c) The style of Pre- and Post-tests were not the same. True Measure Of Outcome?
Kumar-Wang/TAMU Assessment Conference/ We started to try a new format in a Post-Test, Dec –Old: 10 to12 problems, each relates to materials covered in one chapter, or even in one section; –New (Goal): 3 to 5 problems, each problem has several sub-problems. Each sub-problem tests a concept, when combined it relates to materials covered in several chapters; Two versions of test. Part (a) of problem 1 is given in MC format in version Gn, and in FB format in version Pk. Part (b) of problem 1 is in FB format in version Gn and in MC format in version Pk. The two versions are the same problem with different numbers.
Kumar-Wang/TAMU Assessment Conference/ A 1,000-kg car accelerates uniformly from 4.00-m/s to m/s in 4.00-s. The car’s acceleration, in m/s 2, is (a) 1.0; (b) 2.0; (c) 3.0; (d) 4.0; (e) 8.0. The car’s displacement is _________________m. The net force on the car is _________________N The car’s final momentum, in N-s, is (a) 4,000; (b) 8,000; (c) 12,000; (d) 16,000; (e) 20,000.. The car’s final kinetic energy is _____________J. The net work done on the car is _____________J. The net average power to the car is __________W. PROBLEM 1A-G - Pk
Kumar-Wang/TAMU Assessment Conference/ A 1,000-kg car accelerates uniformly from 4.00-m/s to m/s in 4.00-s. The car’s acceleration is __________________m/s 2. The car’s displacement, in m, is (a) 12.0; (b) 16.0; (c) 24.0; (d) 32.0; (e) The net force on the car, in N, is (a) 1,000; (b) 2,000; (c) 3,000; (d) 4,000; (e) 5,000. The car’s final momentum is ________________N-s. The car’s final kinetic energy is _____________J. The net work done on the car is _____________J. The net average power to the car is __________W. PROBLEM 1A-G - Gn
Kumar-Wang/TAMU Assessment Conference/ There were 91 students, 43 took version Gn & 48 took version Pk. Each student worked 4 sub-problems in MC format & 4 in FB format. The 8 sub-problems are grouped into three levels: –Level 1, easiest, such as “Find the net force on an object when the mass & the acceleration of the object are given”. –Level 2, a little more involved, such as “ Find the acceleration or displacement when the initial velocity, final velocity & time are given”. –Level 3, more involved, such as exchange of elastic potential energy & kinetic energy, or exchange of kinetic & gravitational potential energy.
Kumar-Wang/TAMU Assessment Conference/ Advantages of the New Format Much less time will be spent by students reading into the physical situations. Physical quantities covered in different chapters are shown to be inter-connected.
Kumar-Wang/TAMU Assessment Conference/ Pre- and Post-Test
Kumar-Wang/TAMU Assessment Conference/ Multiple-Choice vs Fill-in-the-Blanks in Post-Test
Kumar-Wang/TAMU Assessment Conference/ Multiple-Choice vs Fill-in-the-Blanks Level-3, Problem-2a
Kumar-Wang/TAMU Assessment Conference/ Multiple-Choices vs Filling-Blanks Level-3, Problem 2b
Kumar-Wang/TAMU Assessment Conference/ Multiple-Choice vs Fill-in-the-Blanks Level-3 Problems
Kumar-Wang/TAMU Assessment Conference/ Repeat the tests and correlate student’s prior preparation in mathematics and physics with test performance. Conduct interviews with students in groups with similar performance levels. Assess the level of remediation a student needs to improve his/her learning during the semester. NEXT STEPS
Kumar-Wang/TAMU Assessment Conference/
Kumar-Wang/TAMU Assessment Conference/ In each pair, the second picture was altered from the original either in: (i) changes that did not alter the physics depicted, or (ii) changes that made a meaningful change to the physics depicted. Novices perform equally well on both types of image pairs, but experts perform significantly better on image pairs that contain a meaningful change in the physics depicted than they do on image pairs that do not contain a meaningful change to the physics depicted. Implication: Conceptual physics knowledge plays a role in the way physics experts form representations of physics pictures. Moreover, for some image pairs, novices were much more able to detect the change in the picture than were experts. This suggests that the memory representations that experts create may lack the surface-feature details that novices’ memory representations contain. EXPERT-NOVICE DIFFERENCES IN MEMORY
Kumar-Wang/TAMU Assessment Conference/ EXAMPLES OF PROPOSED CHANGES
Kumar-Wang/TAMU Assessment Conference/ Our recommendation is that tests be prepared such that: They offer an environment in which the student can receive partial credit; The student be given an opportunity to explain the rationale for his/her choice; The student’s choice must provide an insight into the depth of his/her understanding. Knowing something about the way appropriate process was used to obtain wrong answers and inappropriate strategy was used to obtain correct responses helps to provide a more valid interpretation of test results. Such knowledge is also essential to distinguish adequate from inadequate procedures so that suitable remedial work can be assigned. Effective remediation usually depends on having detailed knowledge of the types of errors that are being made and these have to be inferred from the alternatives selected by the student.
Kumar-Wang/TAMU Assessment Conference/ Build a degree of ambiguity into the question. Design the distractors to reflect the dominant types of misconceptions. Design a “nested” series of problems that allows us to assess true student learning: Ability to model different parts of the problem and integrate into a “whole.” Ability to categorize problems based on similarities - surface and deep. Ability to transfer skills from one type of problem to another. Ability to transfer skills from one subject to another. SPECIFICALLY
Kumar-Wang/TAMU Assessment Conference/ A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is (A)3 miles/hr (B)60 miles/hr (C)180 miles/hr (D)Impossible to determine because the raindrop accelerates due to gravity. Provide the rationale for your choice: ____________________________________________________________ ____________________________________________________________ EXAMPLE
Kumar-Wang/TAMU Assessment Conference/ A raindrop on a drizzly day takes one minute from the time it leaves the cloud until it hits your umbrella, 3 miles below. The average velocity of the raindrop on its journey is (A)3 miles/hr – CLEARLY WRONG (B)60 miles/hr – CLEARLY WRONG (C)180 miles/hr – MOST LIKELY CORRECT (D)Impossible to determine because the raindrop accelerates due to gravity. – POSSIBLY CORRECT EXAMPLE (continued)
Kumar-Wang/TAMU Assessment Conference/ IF CHOICE (C) Rationale: The raindrop would have acquired the terminal velocity by the time it reaches the umbrella.
Kumar-Wang/TAMU Assessment Conference/ Rationale: There are a few assumptions implied in this problem. The raindrop accelerates due to gravity and may experience other forces as well due to – friction upward, wind sideways, … This may cause the raindrop to traverse a longer path on its way to the umbrella. While the “vertical” displacement may be 3 miles, the actual path length may be longer. Too little information provided to make an accurate determination. IF CHOICE D
Kumar-Wang/TAMU Assessment Conference/ Examine the figure below. Assume a frictionless surface. Answer the following questions The acceleration of the two blocks is observed to be 1.8 m/s 2.What is the magnitude of the tension T in the string connecting the two blocks? Assume a frictionless surface. (a) 5.4 N (b) 9.0 N (c) 14.4 N EXAMPLE
Kumar-Wang/TAMU Assessment Conference/ What is the magnitude of the force F pulling the 5 kg block to the right? Assume a frictionless surface. (a) 5.4 N (b) 9.0 N (c) 14.4 N EXAMPLE (continued)
Kumar-Wang/TAMU Assessment Conference/ What is the magnitude of the net force on the 5 kg block? Assume a frictionless surface. (a) 9.0 N (b) 14.4 N (c) 18.8 N EXAMPLE (continued)
Kumar-Wang/TAMU Assessment Conference/ Assume now that the friction of the surface cannot be ignored. What is the magnitude of the net force on the 5 kg block? (a) 9.0 N (b) 14.4 N (c) 18.8 N (d) Other ________ EXAMPLE (continued) Friction
Kumar-Wang/TAMU Assessment Conference/ How would your answer to each of the previous questions change if: a.The masses are doubled? b.The two blocks are reversed? (see figure below) c.The surface is inclined to the right? d.The surface is inclined to the left? e.In (c) and (d) above, what more information would you need? f.Can you identify a real-life situation for which this problem might be a good model? EXAMPLE (continued) Follow-Up Questions 3 kg 5 kg F T=? a=?
Kumar-Wang/TAMU Assessment Conference/ CONCLUSIONS Revisit our entire approach to testing, evaluation and assessment. Incorporate modern thinking about learning. Employ problem-based learning approach.
Kumar-Wang/TAMU Assessment Conference/ THANK YOU FOR YOUR ATTENTION AND PATIENCE. ANY QUESTIONS?