1. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 1 1.- Introduction 2.- Theory and hypotheses 3.- Methodology 4.- Results 5.- Conclusion References Bresnock, A. E., Graves, P. E. and White, N. 1989. “Multiple-Choice Testing: Question and Response Position.” Journal of Economic Education, Summer. Carlson, J. L. and Ostrosky, A. L. 1992. “Item Sequence and Student Performance on Multiple-Choice Exams: Further Evidence.” Journal of Economic Education, Summer. Doerner, W. M. and Calhoun, J. P. 2009. “The Impact of the Order of Test Questions in Introductory Economics”. Economics Educator: Courses, Cases & Teaching, April 20, Economics Research Network (ERN) Working Paper. Gohmann, S. F. and Spector, L. C. 1989. “Test Scrambling and Student Performance”. Journal of Economic Education, Summer. Marín, C. and Rosa-García, A. 2012. "Gender Bias in Risk Aversion: Evidence from Multiple Choice Exams". MPRA Paper nº 39987, Julio 2012. Online at http://mpra.ub.ini-muenchen.de/39987/."http://mpra.ub.ini-muenchen.de/39987/ Sue, D. L. 2006. “The Effect of Test Scrambling on Student Performance.” NBEA Annual ConferenceProceedings. Sue, D. L. 2009. “The effect of Scrambling test questions on student performance in a small class setting”. Journal for Economic Educators,vol. 9, n. 1. Taub, A. J., and Bell, E. B. 1975. “A Bias in Scores on Multiple-Form Exams.” Journal of Economic Education, Fall.

2. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 2 1. INTRODUCTION The development of new technologies have facilitated many the roles of teachers It has made possible the realization of test with inmediate feeback and the ability to shuffle the order of questions and establish different types The effects on the student shuffle question has been a hotly debat This technique used especially when the number students is high The risk behind this technique is the orden in which the questions are asked It necessary to investigate: Whether the performance of a test that another can influence student performance What are the variables that can influence more in the grades

3. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 3 2.- THEORY AND HYPOTHESES One method of counteracting cheating is to use multiple forms of the same exam by scrambling test questions This procedure produces one content-ordere exam that matches the order in which the material was present in class and one exam that has a random scrambled order Previous studes do affirm that students may perform better on a content orderer exam Taub and Bell (1975) affirm that students who took the form which questions were randomly arranged have more test anxiety and they also lose concentration Other studies showed opposite results (Gohmann and Spector, 1989; Bresnock et al., 1989) Marín and Rosa-Garcia evidencing that males tend to answer a higher amount of questions than females and evidencing a higher degree of risk in males anwer Hypothesis 1: Student who perform an examination of the type 02 do worse tah those doing the type 01 Hypothesis 2: The response procedure (right questions, erroneous and white) of student taking the test type 01 differs from those doing the exam type 02

4. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 4 3.- METHODOLOGY Sample and data. In order to empirically test the effect of scrambling the content order of multiple- choice questions on a student’s performance and exploring the differences in outputs, we use a sample of 331 type 01 and 02’s exams done by undergraduate students from the University of Murcia (Spain) belonging to “Work and Labour Organization” subject of the 2012-13 academic year. Variables. We have use dependent, independent and moderating variables. Dependent variable is student performance, measures in three ways: 1.Quantitative variable in a 0-10 points’ scale, which indicate the score obtained by the student in the exam; 2.Dummy variable indicating if the students obtain a score below or above average; 3.Qualitative variable indicating the procedure of answer, measurers as the number of correct, incorrect and blank responses doing by the student in the exam. Independent variable is the type of exam, measures through a dummy variable indicating type 01 exam (with ordered questions) and type 02 (with not ordered questions). We also have used two segmentations, which lead us to generate two moderating variables: 1.experience of the students, proxies by two variables: the age of students (more or less of 19 years old) and the number of exam’ call student facing (first, second or more); 2.risk taking by the student doing the exam, proxies by the genre of student as a qualitative variable indicating male or female.

5. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 5 4. RESULTS ResultsExamtypeRight answersWrong answersBlank answersScores General comparison Absolute score 0-10 019,047,283,68**3,34 028,826,764,42**3,34 Best grade 0111,594,394,025,06 0211,594,483,925,05 Worse grade 016,749,89**3,37***1,78 026,248,87**4,89***1,75 Students experience A. Students average age Older students 019,107,203,70*3,38 028,936,794,28*3,39 Younger students 018,677,833,503,06 028,156,605,253,01 B. Students number of exam’ call facing Firstcall 019,317,583,11**3,42 029,166,913,93**3,49 Second call or more 018,436,604,983,13 028,076,425,512,99 Students risk taking by genre Female 019,316,823,873,56 028,577,064,373,20 Male 018,647,96***3,39**3,00 029,166,34***4,5**3,52 Table 1. Analysis of variance (ANOVA) results

6. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 6 4. RESULTS Hypothesis 1: Students with a scramble (type 02) exam have a lower performance than those doing the unscramble exam (type 01). HYPOTHESIS 1 -> REJECTED Hypothesis 2: The response procedure (right questions, erroneous and white) of students taking the unscramble (type 02) test differs from those doing the scramble exam (type 01). HYPOTHESIS 2 -> ACCEPTED

7. DOES CONTENT ORDER AFFECT PERFORMANCE ON MULTIPLE CHOICE EXAMS? 7 5. CONCLUSIONS There are differences in the procedure of response, but this procedure does not affect final performance. Type 02 exams’ students left more blank answers and have fewer wrong answers. There aren’t significant differences between the type 01 and 02 exams performance. Trying to explain and understand the origin of such differences, we have segmented the study sample in terms of: 1)Student experience in relation to test examination In case of older students: type 02 exams’ student left more blank answers. Second call or more students: No differences in type 01 and 02 exams performance. First call students: type 02 exams’ students left more blank answers. 2)The student risk taking doing the exam. Female group of students: No differences in type 01 and 02 performance exams Male group of students: In 02 type exams’ students left more blank answers and have less wrong answers, not affecting final performance. General conclusion: older students, first call students, and male student doing type 02 exams, although they show the same performance than the others, left more blank answers and have less wrong answers, trying to secure more their responses, and avoiding penalties (because of wrong answers), becoming in a more risk adverse students.