Assessing Student Achievement in the Core Curriculum Office of Assessment and Program Improvement Report to the University Community Fall 2003

Office of the Provost Office of the Provost 2 The Assessment Process ÜDefine specific learning outcomes that should generally be evident in students who complete a program of study ÜIdentify sources of evidence about how well students are typically achieving these outcomes ÜSet performance targets as criteria for program success ÜConduct studies and use the results to improve achievement

Office of the Provost Office of the Provost 3 Assessing a Major vs. Assessing the Core C O R E What any AU student should know and be able to do no matter what major M A J O R What an AU student majoring in a given subject should know and be able to do Defined by specialists Defined Cooperatively

Office of the Provost Office of the Provost 4 Working Assumptions ÜWe can collect reasonably good evidence about how well students are achieving the desired “core outcomes” for an AU graduate, but it is difficult to know what led to this achievement ssessing students’ core competencies is more feasible than determining the exact contribution of the Core Curriculum itself ÜAssessing students’ core competencies is more feasible than determining the exact contribution of the Core Curriculum itself ÜUsing several methods and measurement points will provide fuller information than relying on any one “snapshot” method ÜThe “core outcomes” are everybody’s business

Office of the Provost Office of the Provost 5 Core Curriculum Oversight Committee  Created together with the Core Curriculum Original charge was to oversee implementation of Core Began assessment in 1995 with reviews of social science and Great Books courses Reviews focused on teaching – syllabi, assignments, textbooks, grade distributions Assessment best practice now focuses on student learning

Office of the Provost Office of the Provost 6 Assessment Process ÜDefine intended outcomes

Office of the Provost Office of the Provost 7 From Broad Purpose to Specific Plans In 2001 the Core Curriculum Oversight Committee ÜStudied the broad purpose statement for the Core Curriculum ÜFormulated 12 more specific intended learning outcomes ÜSelected 4 outcomes for assessment in 2002 and 2003

Office of the Provost Office of the Provost 8 Starting Points for Assessment  Read critically and analytically  Communicate in writing at a sophisticated level  Comprehend the basic concepts of algebra  Gather, interpret, and synthesize information in accordance with contemporary scholarly standards Students who have completed the Core Curriculum will be able to

Office of the Provost Office of the Provost 9 Future Assessment  Scientific Reasoning*  Art Appreciation  History, Culture, Values  Social Sciences  Oral Communication  Awareness of Issues  Critical Thinking  Problem Solving Outcomes for Future Study

Office of the Provost Office of the Provost 10 Assessment Process ÜDefine intended outcomes ÜIdentify means of assessment

Office of the Provost Office of the Provost 11 Two Methods Used ÜLocal methods to estimate the abilities of students who were still taking Core courses  Analysis of papers and exams  Pre- and post-testing ÜACT’s Collegiate Assessment of Academic Proficiency (CAAP) tests to estimate the abilities of students who had fulfilled all Core Curriculum requirements  Writing, Reading, Math, Critical Thinking, Science Reasoning

Office of the Provost Office of the Provost 12 Assessment Methods S T A N D A R D ACT CAAP Tests Project SAILS L O C A L Review of work samples Pre- and Post-Testing Taking CoreFulfilled Core

Office of the Provost Office of the Provost 13 Assessment Process ÜDefine intended outcomes ÜIdentify means of assessment ÜSet targets for program success

Office of the Provost Office of the Provost 14 Standards Chosen: Local Methods ÜCritical Reading 70% of portfolios reviewed should rate satisfactory or better when evaluated by two independent judges ÜWritten Communication 70% of portfolios reviewed should rate satisfactory or better when evaluated by two independent judges ÜCollege Algebra Improvement should be evident on post-test ÜInformation Literacy Both standards – 70% satisfactory and post-test improvement

Office of the Provost Office of the Provost 15 Standards Chosen: CAAP Test ÜAU students who have completed the Core Curriculum should score at or above the 70 th percentile nationally on each CAAP module ÜNo more than 20% of AU students tested should perform below the 50 th percentile nationally on any CAAP module ÜNational comparison group was sophomores at about 80 public 4-year colleges

Office of the Provost Office of the Provost 16 Assessment Process ÜDefine intended outcomes ÜIdentify means of assessment ÜSet targets for program success ÜCollect and analyze assessment data

Outcome 1 Critical Reading Students who have completed the Core Curriculum will be able to read critically and analytically at a sophisticated level

Office of the Provost Office of the Provost 18 Critical Reading Analysis of papers and exams from Great Books II showed ÜStudents were able to find, remember, and interpret specific details from their readings ÜThey were able to discuss themes found in their readings ÜAssessment could not determine whether students were able to discuss formal literary features of the works they had read or place them in their historical and social contexts

Office of the Provost Office of the Provost 19 Critical Reading Percent of Great Books II Portfolios Rated Excellent/Satisfactory

Office of the Provost Office of the Provost 20 Critical Reading In formal papers for Great Books II, students met but did not exceed the performance target for three aspects writing about texts that may imply critical reading ability ÜFormulating a thesis about a text they had read ÜUsing textual evidence to support that thesis ÜReasoning soundly to connect evidence and thesis

Office of the Provost Office of the Provost 21 Critical Reading GB2 Essays rated Excellent or Satisfactory

Office of the Provost Office of the Provost 22 Critical Reading Results from the two CAAP tests for Critical Reading were mixed ÜReading Median score was at the 73 rd percentile nationally But 22% of those tested scored below the 50 th percentile ÜCritical Thinking Median score was at the 82 nd percentile nationally Only 11% scored below the 50 th percentile

Office of the Provost Office of the Provost 23 Critical Reading Median Percentile Scores and “Failure Rates” CAAP Reading and Critical Thinking Tests

Outcome 2 Written Communication Students who have completed the Core Curriculum will be able to communicate in writing at a sophisticated level

Office of the Provost Office of the Provost 25 Written Communication Analysis of argument papers from Composition II showed ÜStudents were clearly able to formulate a thesis to use language that was appropriate to the writing situation ÜStudents had more trouble Supporting their thesis with sound evidence Organizing their arguments effectively

Office of the Provost Office of the Provost 26 Written Communication Four Aspects of Rhetorical Effectiveness ENGL1120 Papers with Satisfactory Ratings

Office of the Provost Office of the Provost 27 Written Communication The two CAAP tests for writing were given in 2002 only ÜStudents “maxed out” the Essay Writing test – nearly all scored at or above the 70 th national percentile ÜScores on the multiple-choice Writing Skills test were also strongly positive ÜIt is not clear that there is good alignment between this test and the goals of Composition I and II

Office of the Provost Office of the Provost 28 Written Communication Median Percentile Scores and “Failure Rates” CAAP Writing Skills and Essay Writing Tests

Outcome 3 College Algebra Students who have completed the Core Curriculum will comprehend the basic concepts of algebra

Office of the Provost Office of the Provost 30 College Algebra Local Assessment (2003 Only) ÜStudents took Math Placement Exam at Camp War Eagle ÜSome items from this exam were embedded in the final exams for pre-calculus Core courses (not including MATH 1100) ÜOn average, the 916 students tested scored 30% higher on these items on the final exam than they had done on the placement test

Office of the Provost Office of the Provost 31 College Algebra Results for the CAAP Mathematics test must be treated separately for 2002 and 2003 because of a change in the test design Ü2002 version (4 calculus items) students met the performance targets for the algebra subscore Ü2003 version (no calculus items) students met the performance targets for the whole test ÜFor both years, students’ performance just cleared the bar of the 70 th national percentile, but concerns remain

Office of the Provost Office of the Provost 32 College Algebra Median Percentile Scores for CAAP Mathematics Test

Office of the Provost Office of the Provost 33 College Algebra The CAAP results are less positive when students are grouped ÜThose who had taken Math courses beyond the Core met performance targets... But... ÜThose who had taken only one Math course failed to meet performance targets

Office of the Provost Office of the Provost 34 College Algebra Median Percentile Scores and “Failure Rates” “More Math” vs. “Core Math” Students

Experimental Outcome Science Reasoning

Office of the Provost Office of the Provost 36 Science Reasoning The CAAP results for Science Reasoning (not formally assessed in either year) resemble those for Mathematics ÜThose who had taken only one Math course scored poorly in Science Reasoning ÜThose who had taken Math courses in addition to the Core did much better in Science Reasoning

Office of the Provost Office of the Provost 37 Science Reasoning Median Percentile Scores and “Failure Rates” CAAP Science Reasoning Test

Outcome 4 Information Literacy Students who have completed the Core Curriculum will be able to gather, synthesize, and interpret information in accordance with contemporary scholarly standards

Office of the Provost Office of the Provost 39 Information Literacy ÜBefore receiving bibliographic instruction (BI), ENGL 1120 students were tested on how to conduct library searches ÜSample: “You wish to find current articles dealing with drug use on U.S. campuses. Your best “place” to begin? ÜStudents were retested after BI and had higher scores

Office of the Provost Office of the Provost 40 Information Literacy Ü165 UNIV1000 students used the online Texas Information Literacy Tutorial (TILT) and took the accompanying quizzes ÜThe overall score on the TILT was 96% ÜTheir lowest score was on an item that tested whether they thought the library’s whole collection was available online

Office of the Provost Office of the Provost 41 Information Literacy Project SAILS  In 2003, 247 ENGL1120 students took part in Project SAILS, a national information literacy test recognized by the Association of Research Libraries  Multiple choice questions yielded information about 4 key information literacy skills  On each skill, AU students did somewhat better than the average for all 10 participating institutions

Office of the Provost Office of the Provost 42 Information Literacy Work Samples Aspects of Using Sources Effectively ENGL1120 Papers with Satisfactory Ratings

Office of the Provost Office of the Provost 43 Assessment Process ÜDefine intended outcomes ÜIdentify means of assessment ÜSet targets for program success ÜCollect and analyze assessment data ÜAct on the results to improve student achievement

Office of the Provost Office of the Provost 44 Planned Actions ÜRemedy focused deficiencies by additional instruction and practice in appropriate courses – e.g., documenting sources ÜGuide instruction, assessment, and improvement by clarifying some broad intended outcomes – e.g., critical reading ability ÜContinue to get good evidence about our students’ core competencies through dialogue and collaboration ÜBenefit from experience with four outcomes to guide assessment planning for the intended learning outcomes that have not been studied so far

Further Information Office of Assessment and Program Improvement Drew Clark, Director 209 Samford Hall Office of the Provost Office of the Provost