2 Webinar Agenda CAAP Overview CAAP and the VSA CAAP Usage Strategies Plan for Success CAAP Resources
3 Collegiate Assessment of Academic Proficiency
4 The standardized assessment program from ACT that enables postsecondary institutions to assess, evaluate, and enhance the outcomes of their general education programs.
5 Selected as one of the options for measuring student learning outcomes for the Voluntary System of Accountability.
6 CAAP Benefits Systematically measure your students achievement levels and growth in your general education program
7 CAAP Benefits Institution and individual student level data to help you identify opportunities to enhance student learning
8 CAAP Benefits Help satisfy the need for greater accountability in higher education
9 CAAP Benefits Help meet accreditation self-study requirements
10 CAAP Benefits Help evaluate individual student readiness for upper division coursework
11 CAAP Benefits Not to evaluate faculty. Not as a sole evaluation in high stakes environments.
12 CAAP Features Modular format (6 content areas) Flexible administration Comprehensive reports Extensive national user norms Value-added measurement through linkage to COMPASS TM and the ACT ® test Identification of action items with Content Analysis Reports
14 CAAP Administration Convenient in-class administration 40 minutes per test Pencil and paper Proctored Can be embedded in required freshman or capstone course 2-week test window Nine local questions can be added (except for VSA testing)
15 CAAP Standard Reporting Package Institutional Summary Report Mean scores for each demographic and subscores in math, reading, & writing skills Student Score Reports – 2 copies One for institution and one for student Certificate of Achievement For students scoring at or above the national mean Student Roster Report Listing of each student with their CAAP results
16 CAAP User Norms to
17 CAAP User Norms Norms for each test module (and subscore) Broken out by: Institution type Year of students tested Institution ownership Three-year rolling norms Custom groupings available
18 CAAP and the VSA Voluntary System of Accountability A joint initiative developed in partnership between NASULGC and AASCU Designed to help institutions Demonstrate accountability and stewardship to the public Measure educational outcomes to identify effective educational practices Assemble information that is accessible, understandable, and comparable Currently over 60 NASULGC/AASCU institutions use CAAP for their outcomes assessment
19 CAAP and the VSA VSA College Portrait Consumer Information Student Experiences and Perceptions Student Learning Outcomes VSA participants measure critical thinking, analytical reasoning, and written communication CAAP Measures critical thinking and written communication using two test modules CRITICAL THINKING WRITING ESSAY
20 VSA College Portrait Pilot VSA Web Site College Portrait data elements College Portrait prototype VSA Background/development VSA Enrollment
21 CAAP and the VSA Student Learning Outcomes Freshman to Senior Learning Gains Critical Thinking Writing Essay The increase in learning on the performance task is _____ what would be expected at an institution with students of similar academic abilities. well above, above, below or well below what would be expected
22 College Portrait Learning Outcomes Reporting College Portrait Prototype
23 CAAP Writing Essay Two independent prompts 20 minutes each Essays scored at ACT by two trained raters ACT scoring rubric 1-6
24 CAAP Writing Essay Formulating an assertion about a given issue Supporting that assertion with evidence appropriate to the issue, position taken, and a given audience Organizing and connecting major ideas Expressing those ideas in clear, effective language
25 Writing Essay Sample Your college administration is considering whether or not there should be a physical education requirement for undergraduates. The administration has asked students for their views on the issue and has announced that its final decision will be based on how such a requirement would affect the overall educational mission of the college. Write a letter to the administration arguing whether or not there should be a physical education requirement for undergraduates at your college.
26 CAAP Critical Thinking 32 items/4 passages Sample questions on-line Clarifying, analyzing, evaluating, and extending and argument Subject matterSocial Sciences, Natural Sciences, Humanities Formats include: Case Studies Dialogues Debates Editorials Statistical Arguments Experimental Results Overlapping Position
27 CAAP and the VSA Sample Size ACT requires a minimum 200 freshman and 200 seniors tested on each module Test Dates Freshmen must be tested early in the fall semester Seniors must be tested within 6 months of graduation Both groups tested during the same academic year (September-May ) ACT (or SAT) Score Required Institutions must provide data file with CAAP tested students ACT or SAT scores – see CAAP VSA Guidelines for more details
28 CAAP and the VSA Random selection Institution determines what constitutes a random representative sample Value-added VSA Model Cross-sectional Longitudinal Model –Consider the minimum number of records needed at the time of post-testing and then calculate back to establish the number for pre-test, and test 3-4 times that number
29 CAAP Scoring Return materials to ACT for scoring and reporting Critical Thinking ACT proprietary scanning Writing Essay Two ACT-trained scorers Three-week turn-around time for standard reporting package
30 CAAP VSA Report Provided upon completion of all freshmen and senior testing – late summer Provided upon receipt of your institution data file (with student matched ACT or SAT and CAAP scores) Report will include: Average scores for both freshmen and seniors on both test modules Student learning gains on each test module Average ACT (or SAT) for students tested
32 CAAP Administration for VSA Administer at least once every 3 years Report within 4 years of becoming VSA participant Updated at least every 3 years
33 Additional CAAP Test Modules CAAP VSA Materials Order Form provides option for additional modules Administer to a different group of students than those tested on the VSA modules Option for local questions for only those who are not testing for VSA
39 TEST MODULEC.A.R. DOMAINS Writing Skills -Usage/Mechanics -Rhetorical Skills Punctuation Grammar Sentence Structure Strategy Organization Style Mathematics -Basic Algebra -College Algebra Prealgebra Elementary Algebra Intermediate Algebra Coordinate Geometry College Algebra Trigonometry Reading - Arts/Literature - Social/Nat. Sciences Reasoning Skills Referring Skills Science Analysis Generalization Understanding Critical Thinking Analysis of Elements of Arguments Evaluation of Arguments Extension of Arguments Content Analysis Report
41 Content Analysis Report (Math Sample)
43 Value-Added Measurement
44 Value-Added Measurement
45 Value-Added Measurement Longitudinal (CAAP to CAAP) Longitudinal (Linkage) ACT to CAAP COMPASS to CAAP VSA Model – Cross-Sectional – Freshman to Senior Learning Gains
46 Value-Added Measurement Longitudinal (CAAP to CAAP) Test cohort with CAAP upon program entry Test same cohort with CAAP upon completion of program Change in scores is a direct measure of value-added performance gain Evaluate results in terms of expectations Investigate opportunities for improvement
47 Value-Added Measurement Longitudinal (ACT or COMPASS to CAAP) Use cohorts ACT or COMPASS scores as a benchmark Test cohort with CAAP upon completion of general education coursework Linkage report evaluates cohorts results in terms of expected gain Evaluate results in terms of expectations Investigate opportunities for improvement
48 Value-Added Measurement Cross-sectional - VSA Model Test a cohort of incoming freshmen at the beginning of the fall term Test a similar group of seniors within 6 months of graduation Differences between the two group mean scores may indicate performance gain Challenge of this strategy is matching student characteristics across groups
52 Plan for Success #1 Confirm the fundamentals Review your institutional mission statement. Evaluate the objectives of your institutions general education program. Evaluate your institutional general education core curriculum.
53 Plan for Success #2 Specify usage of results Who will review and analyze the data? How will the results be used to influence program evaluation and planning? Will results be tracked on a longitudinal basis?
54 Plan for Success #3 Define population Specify characteristics of the student body that is the focus of the study. Specify characteristics of subgroups.
55 Plan for Success #4 Sampling Test entire population or sample? How will sample be selected? How will subgroup sample(s) be selected? Is sample random? Is sample representative of student population?
56 Plan for Success # 5 Faculty motivation Specify plans to communicate with faculty. Specify plans to enlist faculty support. Specify plans to share results with faculty.
57 Plan for Success #6 Student motivation Specify plans to communicate with students. Specify plans to motivate students. Specify plans to share results with students.
58 Plan for Success Submit CAAP-VSA Participation Agreement Confirm test modules, if additional* Confirm optional local questions* Confirm reporting requirements Confirm test period (two-week window) Confirm test venues Confirm students to be tested Confirm proctors Schedule training for proctors Order test materials at least one month prior to test dates *For non-VSA test modules only #7 CAAP Test Administration