1. MAPP and VSA The Role of the Measure of Academic Proficiency and Progress in the Voluntary System of Accountability Bill Wynne Solutions Implementation Specialist ETS Higher Education Division

2. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 2 What are “outcomes?” Outcomes are the effects or changes brought about by the curriculum or activities provided by an institution or program. In the process of specifying outcomes, a program/institution typically establishes measurable criteria for each outcome. What are “outcomes assessments?” Outcomes assessments answer the question "Did we, in fact, accomplish what we intended for our students, institution, local community, or other stakeholders?" It is a formal process for comparing actual results to the established outcomes criteria. Changes implemented as a result of these analyses can be documented and outcomes can be measured again in a following cycle to verify improvement. Measurements of outcomes show the degree of an institution’s or program’s effectiveness.

3. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 3 Why Should We Assess Student Learning Outcomes? Accreditation –Institutional (General Education) –Program (Discipline-specific) Strategic planning Renewed focus on institutional/program effectiveness and continuous improvement –Trend analysis (longitudinal/cross-sectional studies) –Benchmarking –Comparison with other institutions/programs Measure instructional effectiveness –Curriculum alignment –Teaching methods Facilitate student feedback/counseling/advising Performance funding/making effective resource allocation decisions Marketing Demonstrate performance to stakeholders demanding accountability and continuous improvement

4. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 4 Why Use Standardized Outcomes Assessments? National or other broad comparison group Longitudinal data Diagnostic feedback Test design –Validity –Reliability –Fairness –Representativeness

5. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 5 Voluntary System of Accountability –A voluntary initiative for four-year public colleges and universities –A collaborative effort of the American Association of State Colleges and Universities (AASCU) and the National Association of State Universities and Land-Grant Colleges (NASULGC) –Designed to improve public understanding of how public colleges and universities operate College Portrait –An attempt at providing consistent, comparable, and transparent information on the characteristics of institutions and their students –Intended to be used by students, families, policy makers, campus faculty and staff, the general public, and higher education stakeholders More information available at www.voluntarysystem.org What is the VSA?

6. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 6 A common reporting template only five (5) pages in length –MS Excel workbook format uses visual basic macros to convert data into worksheets, graphs, and tables Majority of the data elements selected are from currently available data sources with established definitions and reporting conventions (e.g. the Common Data Set) Data elements divided into three standardized sections –Consumer Information –Student Experiences and Perceptions –Student Learning Outcomes More information available at www.voluntarysystem.org The VSA College Portrait

7. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 7 Costs of attendance Degree offerings Living arrangements Student characteristics Graduate rates Transfer rates Post-graduate plans The VSA College Portrait Consumer Information Data Elements What information would be most helpful to prospective students and their families in deciding which university best fits their educational wants and needs?

8. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 8 Results reported within six specific constructs shown to be correlated with greater student learning and development –Group learning –Active learning –Experiences with diverse groups and ideas –Student satisfaction –Institution commitment to student learning and success –Student interaction with faculty and staff The VSA College Portrait Student Experiences and Perceptions Data Elements A snapshot of student experiences and activities and their perceptions of the college or university

9. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 9 Institution description of how they evaluate student learning –Links to institution-specific outcomes (e.g. professional licensure) Results from one of three standardized instruments which measure broad cognitive skills –MAPP is one of these assessments –Results reported two ways Value added (learning gains between freshman and senior years) Actual average test scores for freshmen and seniors The VSA College Portrait Student Learning Outcomes Data Elements An attempt at standardizing evidence of student learning

10. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 10 Timeline –Institutions can begin collecting and reporting information any time after February 15, 2008 –Initially, the College Portrait does not have to be fully populated Institutions can continue to add data as it is gathered This includes data on a standardized outcomes assessment Measurement of learning outcomes is considered a pilot –Many institutions had not previously administered standardized outcomes assessments on a large scale –Institutions may take up to four years to choose their assessment and begin reporting results –Standardized testing vendors have been permitted these four years to arrive at the common method for reporting value added The VSA College Portrait Getting The Facts Straight

11. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 11 Take into account the academic ability of the students using a control variable (such as ACT/SAT scores) in a regression algorithm Y = p0 + p1*x - p2* x Requires data from large groups to develop the “formula” “Levels the playing field” based on expectations Permits all VSA-approved outcomes measures to report on a common scale across institutions of varying levels of selectivity Each testing organization will use the same method to calculate and characterize value added What is value added? The VSA Methodology Learning gains between freshman and senior year Not the actual regression algorithm

12. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 12 Well Above Expected Above Expected At Expected Below Expected Well Below Expected NOTICE… The scale is Descriptive – NOT Numeric!!! What is value added? The Common Value Added Scale

13. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 13 Value added score is descriptive – not numeric –There is no one qualifying score that determines whether an institution is above, at, or below expectations –The VSA value added methodology measures expected growth specific to an institution – not across institutions –Data from a large number of institutions will be used to create the regression algorithm Most of the VSA-approved tests currently cannot report value added in this way (VSA methodology) –Studies currently underway –Test vendors committed to reporting value added using the same methodology in the same timeframe –MAPP users will be able to use data from current administrations to determine value added retroactively Retain the ACT/SAT scores of each MAPP cohort tested The VSA College Portrait Getting The Facts Straight

14. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 14 Study design –Cross-sectional (different cohorts of students) Faster, simpler, less costly Freshmen and seniors May be tested in the same year Frequency –At the conclusion of four year pilot, institutions will report standardized test results at least once every three years ACT/SAT scores may not be used as a proxy for freshmen testing –VSA feels that value added cannot be measured accurately using two different instruments -High correlations between admissions test and outcomes test do not mean that they are measuring the same thing in the same manner The VSA College Portrait Getting The Facts Straight

15. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 15 Sampling –Recommendations from testing organizations are minimums –Minimums are for reporting institution-level scores (value added) –Oversampling required to obtain scores for specific subgroups –Random stratified sampling recommended to ensure appropriate representation –ETS experts will consult with your institution on the sample size and methodology best suited to your needs The VSA College Portrait Getting The Facts Straight

16. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 16 The VSA College Portrait Reporting MAPP on the College Portrait Is that all there is?

17. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 17 “New kid on the block?” Measure of Academic Proficiency and Progress (MAPP™) 20 year history as Academic Profile (jointly created and operated by ETS and College Board)

18. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 18 Single, integrated measure of general education skills Focuses on academic skills rather than knowledge Provides invaluable data for… –Accreditation –Strategic planning –Curriculum improvement –Performance-based funding –Benchmarking –Determining “value added” or learning gains –Voluntary System of Accountability Measure of Academic Proficiency and Progress (MAPP™)

19. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 19 Measure of Academic Proficiency and Progress (MAPP™) Available in two administration lengths - Standard and Abbreviated Standard Form for more individual student data Abbreviated Form for the shortest available assessment of this kind Multiple forms for pre-/post-testingEliminates concerns about exposure or “test-retest effect” Norm-referenced and criterion-referenced scores Results can be used for benchmarking, determining “value added,” and curriculum improvement Paper-pencil or Internet-based deliveryChoices for any institution regardless of extent of technical infrastructure Proctored or unproctored versionsUnproctored versions are perfect for distance learning students and facilitating sampling of off-campus students Features Benefits

20. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 20 Measure of Academic Proficiency and Progress (MAPP™) All forms and versions equatedResults from Standard/Abbreviated, Pre- /Post, Paper-Pencil/Online, and Proctored/Unproctored can be compared or combined Customizable demographic questionsAllows analysis of subgroups of importance to each institution Ability to add multiple-choice questionsAllows further examination of content, skills, or attitudes not covered by the core test Comprehensive and customizable comparative data Allows comparisons using a reference group of interest to each institution Optional essay using e-Rater™Use constructive-response module to gauge students’ writing progress Features Benefits

21. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 21 MAPP Assesses Academic Skills College-Level Reading25% Critical Thinking25% College-Level Writing25% College-Level Mathematics25%

22. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 22 MAPP Assesses Academic Skills Humanities –Selections of poetry, fiction, and nonfiction prose –Discussions of painting, music, or philosophical issues Social Sciences –History, Economics, Political Science, Psychology, Anthropology, or Sociology Natural Sciences –Biology, Chemistry, or Physics In The Context Of…

23. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 23 Two Versions of MAPP To Meet Different Needs Standard Form Covers all skills in one test 108 multiple-choice questions Administered in 120 minutes* Generates all scaled scores and proficiency classifications for both individuals and groups Provides demographic/ subgroup breakdowns *Can be administered over two class periods Abbreviated Form Covers all skills in one test 36 multiple-choice questions Administered in 40 minutes Generates all scores for groups but only total scores reported for individual students** Provides demographic/ subgroup breakdowns **Additional data can be downloaded for institutional research and analysis but without certain student identifiers

24. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 24 Two Versions of MAPP To Meet Different Needs Standard Form Abbreviated Form Standard Form A 108 questions 27 questions per skill area 2 hours Abbreviated Form A1 36 questions 9 questions per skill area 40 minutes Abbreviated Form A2 36 questions 9 questions per skill area 40 minutes Abbreviated Form A3 36 questions 9 questions per skill area 40 minutes Delivered in matrix administration mode (or “spiraled”) So few questions in each skill area = no skill subscores for individual students (reliability) Indicates that Abbreviated Forms are equated back to the Standard Form. Notice that Abbreviated Forms are not equated to each other.

25. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 25 What Information Can We Get From MAPP? Norm-Referenced Scaled Scores –“Norm-referenced” means “a score that takes its meaning from comparison against the scores of others –Such scores can be used easily for benchmarking or for determining “value added” or learning gains by conducting a longitudinal or cross- sectional study Eight (8) Norm-Referenced Scaled scores reported –Total Score –Skills Subscores (Critical Thinking, Reading, Writing, and Mathematics) –Context Subscores (Humanities, Social Sciences, and Natural Sciences) MAPP Score Types

26. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 26 What Information Can We Get From MAPP? Criterion-Referenced Proficiency Classifications –“Criterion-referenced” means “a score that takes its meaning from comparisons against specific learning objectives such as a set of skills or abilities or specific content from a content domain” –Such scores can be used easily for curriculum improvement Nine (9) Criterion-Referenced Proficiency Classifications reported –Mathematics (Level 1, Level 2, Level 3) –Writing (Level 1, Level 2, Level 3) –Reading (Level 1, Level 2) –Critical Thinking MAPP Score Types

27. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 27 MAPP Scaled Score Report

28. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 28 MAPP Comparative Data

29. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 29 MAPP Comparative Data Reported by… Class Level/Credit Hours Carnegie Classification Visit the MAPP website at www.ets.org/mapp to view comparative data.

30. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 30 MAPP Proficiency Classifications Report

31. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 31 MAPP Proficiency Level Definitions Proficiency levels are defined by a specific set of skills, abilities, or competencies mapped to each level in the design of the test. Reading/Critical Thinking To be considered Proficient at level 1 a student should be able to… - Recognize factual material explicitly presented in a reading passage - Understand the meaning of particular words or phrases in the context of a reading passage To be considered Proficient at level 2 a student should be able to… - Synthesize material from different sections of a passage - Recognize valid inferences derived from material in the passage - Identify accurate summaries of a passage or of significant sections of the passage - Understand and interpret figurative language - Discern the main idea, purpose, or focus of a passage or a significant portion of the passage To be considered Proficient at level 3 a student should be able to… - Evaluate competing causal explanations - Evaluate hypotheses for consistency with known facts - Determine the relevance of information for evaluating an argument or conclusion - Determine whether an artistic interpretation is supported by evidence contained in a work - Recognize the salient features or themes in a work of art - Evaluate the appropriateness of procedures for investigating a question of causation - Evaluate data for consistency with known facts, hypotheses or methods - Recognize flaws and inconsistencies in an argument

32. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 32 Standard MAPP Score Reports Summary of Scaled Scores (Total/Skills/Context) Roster of Scaled Scores Summary of Proficiency Classifications (numeric and graphical) Roster of Proficiency Classifications Scaled Score Distributions (Total/Skills/Context) (graphical) Individual Student Reports Subgroup Reports Demographic Reports Additional Questions Report Data download –Useful for analyzing smaller groups of interest

33. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 33 Data Presentation is Half The Battle From Standard MAPP Score Report…

34. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 34 Data Presentation is Half The Battle …to MS Excel spreadsheet…

35. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 35 Data Presentation is Half The Battle …to easy-to-understand chart… …in less than 1 minute.

36. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 36 MAPP Demographic Reports Fourteen Demographic Categories Class Level Age Gender Ethnicity Language Ability Enrollment Status Hours Worked Credit Hours Completed Employment status Overall GPA Program Intended/current major Core requirements completed Hours transferred

37. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 37 Sample MAPP Demographic Reports MAPP Subgroup sample size must be large enough to provide meaningful data

38. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 38 Other MAPP Reporting Features All reporting via the web – no paper score reports Reports downloadable as MS Word and MS Excel files Ability to combine scores… –…across cohorts –…across multiple administrations –…across paper-pencil and online –…across proctored and unproctored Data downloadable for analysis or import Future enhancements… Custom Comparative Data report –Allows institution to select reference group from MAPP user list Item Information Report –Analysis of group performance on each question by skill area

39. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 39 MAPP is customizable with Institution-Specific Questions Additional Demographics (Subgroups) – Create up to 72 subgroups that the user defines Facilitates analysis of subgroups of importance to each institution Nine (9) demographic questions with up to eight (8) possible student responses Locally-Written Questions – Author and include up to 50 additional multiple-choice questions Allows further examination of content, skills, or attitudes not covered by the core test Reported as percent correct for each response choice Individual responses to each locally-written question also reported in the data download

40. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 40 Generalizing to the entire institution population –Sample mean vs. true population mean (Can you say anything meaningful or unequivocal about 4,000 seniors based on a sample of 200 seniors?) –There is no “magic number” Analysis of subgroups –Gender, ethnicity, major, etc. –Minimum = 50 students per subgroup* (# Subgroups X 50) *For the abbreviated form. (There are no minimums for the Standard Form.) –Ideal = based on statistical power Curriculum improvement –Estimates of proficiency classifications become more accurate with larger groups Sample should be arrived at consultatively and should be based on institutions needs and statistical properties like “power” and SEMean Sampling Revisited When you need more than the minimum

41. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 41 More skills assessed than just critical thinking and written communications –Reading –Math Maximizes data collection while minimizing testing time –Improves student motivation and willingness to participate –Reduces effects of testing fatigue Integrated skills design produces fair, accurate, and reliable results – All students sampled must be assessed in all skill areas Choosing MAPP for VSA Beyond “Value Added”

42. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 42 More uses for data than just value added –Curriculum improvement Criterion-referenced scores make analyzing curriculum easier than ever –Benchmarking Compare to peer institutions in a meaningful way –…by Carnegie classification –…by customized peer reference group Ability to customize –Addition of demographic subgroup questions of local interest –Addition of up to 50 skills/content questions Choosing MAPP for VSA Beyond “Value Added”

43. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 43 Ease and flexibility of administration –MAPP offers both paper-pencil and online versions –MAPP offers unproctored versions Sample distance learning students with no inconvenience to you or students –Online versions offer immediate score reporting –Scores from all versions can be combined for aggregate reporting –MAPP allows continuous on demand testing 365 days a year 24 hours a day, 7 days a week with unproctored testing Choosing MAPP for VSA Beyond “Value Added”

44. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 44 Administer MAPP to cohorts of 200 freshmen and 200 seniors –ETS scores the tests –Scores available online Request the MAPP Learning Gains Report –Institution identifies cohorts to be included –ETS provides institution with a spreadsheet for providing SAT/ACT scores of MAPP test takers Complete the spreadsheet with SAT/ACT scores Receive the report within four weeks MAPP users will be able to use data from current administrations to determine value added retroactively Retain the ACT/SAT scores of each MAPP cohort tested Using MAPP for the VSA Practical Matters

45. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 45 FAQs – Using MAPP for the VSA Sample MAPP Learning Gains Report Sample SAT/ACT Scores Spreadsheet MAPP Learning Gains Report Request Form White Paper Measuring Learning Outcomes In Higher Education Using the Measure of Academic Proficiency and Progress (MAPP) www.ets.org/mappinfo Using MAPP for the VSA Resources

46. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 46 Creating Assessment Instruments - Important Considerations Reflect the mission and values of the institution Link assessment data to improvement of student learning Measure explicit learning goals and objectives Gather data that is meaningful and useful

47. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 47 Choosing Assessment Instruments - Important Considerations What is the purpose of this test? What are the intended uses of the test? Who are the users of this test? What aspects of the department/ program/institution do we want to measure? What scores will be reported?

48. Confidential and Proprietary. Copyright ©2006 Educational Testing Service. 48 Questions and Comments ? Visit the ETS Higher Education Assessment web pages at www.ets.org/mapp to learn more about the MAPP test Contact a MAPP expert toll free at 1-800-745-0269 If you wish to participate in the MAPP value added study, contact Bill Wynne at 609-683-2006 or wwynne@ets.org Voluntary System of Accountability website www.voluntarysystem.org Overview of VSA Overview of College Portrait and its data elements