A Quantitative Literacy Assessment Rubric Development & Lessons Learned - Stuart Boersma, Central Washington Univ. - Caren Diefenderfer, Hollins University - Shannon Dingman, Univ. of Arkansas - Bernie Madison, Univ. of Arkansas Supported by the National Science Foundation DUE Numeracy Article: Quantitative Reasoning in the Contemporary World 3: Assessing Student Learning

Our Course Uses case studies centered on one or more media articles Each case study contains study questions focusing on the quantitative content of the article Student responses often involve critiquing, analyzing, decision making, and explaining.  Making Assessment Difficult!! QRCW Casebook: Case Studies for Quantitative Reasoning: A Casebook of Media Articles, 2 nd edition, Pearson Custom Publishing. ISBN13:

Need: Instrument that we could apply to these non-standard student responses. AAC&U VALUE Project: Developed for institutional level assessment (not for grading student work) 15 areas, 10 of which are termed Intellectual and Practical skills: Inquiry and analysis, critical thinking, written communication, oral communication, reading, quantitative literacy, information literacy, teamwork, problem solving A Little History

Interpretation Representation Calculation Application/Analysis Assumptions Communication

Alignment of QL VALUE Rubric w/ QRCW course Missing Competencies?: – Critical Reading – Number sense – Synthesis Scale – No “zero” – “4” designated capstone achievement; not a consideration for student work in a QL course Folded into “Interpretation” Changed “application/analysis” to “analysis/synthesis” New Scale: 0-3

Achievement Level Quantitative Literacy Core Competency 3210 Interpretation Ability to glean and explain mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words) Correctly identifies all relevant information. Correctly identifies some, but not all, relevant information. Some relevant information is identified, but none is correct. No relevant information identified. Representation Ability to convert information from one mathematical form (e.g. equations, graphs, diagrams, tables, words) into another. All relevant conversions are present and correct. Some correct and relevant conversions are present but some conversions are incorrect or not present. Some information is converted, but it is irrelevant or incorrect. No conversion is attempted. Calculation Ability to perform arithmetical and mathematical calculations. Calculations related to the problem are correct and lead to a successful completion of the problem. Calculations related to the problem are attempted but either contain errors or are not complete enough to solve the problem. Calculations related to the problem are attempted but contain errors and are not complete enough to solve the problem. Calculations given are not related to the problem, or no work is present. Analysis/Synthesis Ability to make and draw conclusions based on quantitative analysis. Uses correct and complete quantitative analysis to make relevant and correct conclusions. Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or partially complete (e.g. there are logical errors or unsubstantiated claims). An incorrect quantitative analysis is given to support a conclusion. Either no reasonable conclusion is made or, if present, is not based on quantitative analysis. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis. All assumptions needed are present and justified when necessary. At least one correct and relevant assumption is given (perhaps coupled with erroneous assumptions), yet some important assumptions are not present. Attempts to describe assumptions, but none of the assumptions described are relevant. No assumptions present. Communication Ability to explain thoughts and processes in terms of what evidence is used, how it is organized, presented, and contextualized. A correct and complete explanation is clearly presented. A partially correct relevant explanation is present, but incomplete or poorly presented. A relevant explanation is present, but is illogical, incorrect, illegible, or incoherent. No relevant explanation is provided.

Use of QL VALUE Rubric in QRCW course Descriptions of achievement levels not objective enough for our uses: Uses quantitative analysis of data as the basis for competent judgments, drawing reasonable conclusions… Uses quantitative analysis of data as the basis for workmanlike judgments, drawing plausible conclusions… v 3 2 v Skillfully converts relevant information into insightful mathematical portrayal… Competently converts relevant information into an appropriate and desired mathematical portrayal 4 3

Achievement Level Quantitative Literacy Core Competency 3210 Interpretation Ability to glean and explain mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words) Correctly identifies all relevant information. Correctly identifies some, but not all, relevant information. Some relevant information is identified, but none is correct. No relevant information identified. Representation Ability to convert information from one mathematical form (e.g. equations, graphs, diagrams, tables, words) into another. All relevant conversions are present and correct. Some correct and relevant conversions are present but some conversions are incorrect or not present. Some information is converted, but it is irrelevant or incorrect. No conversion is attempted. Calculation Ability to perform arithmetical and mathematical calculations. Calculations related to the problem are correct and lead to a successful completion of the problem. Calculations related to the problem are attempted but either contain errors or are not complete enough to solve the problem. Calculations related to the problem are attempted but contain errors and are not complete enough to solve the problem. Calculations given are not related to the problem, or no work is present. Analysis/Synthesis Ability to make and draw conclusions based on quantitative analysis. Uses correct and complete quantitative analysis to make relevant and correct conclusions. Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or partially complete (e.g. there are logical errors or unsubstantiated claims). An incorrect quantitative analysis is given to support a conclusion. Either no reasonable conclusion is made or, if present, is not based on quantitative analysis. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis. All assumptions needed are present and justified when necessary. At least one correct and relevant assumption is given (perhaps coupled with erroneous assumptions), yet some important assumptions are not present. Attempts to describe assumptions, but none of the assumptions described are relevant. No assumptions present. Communication Ability to explain thoughts and processes in terms of what evidence is used, how it is organized, presented, and contextualized. A correct and complete explanation is clearly presented. A partially correct relevant explanation is present, but incomplete or poorly presented. A relevant explanation is present, but is illogical, incorrect, illegible, or incoherent. No relevant explanation is provided.

Competency Number of questions Percent of questions Number of case studies (N=24) which have at least one question mapped to competency Number of case studies which have 50% or more questions mapped to competency Interpretation 15265% Representation 6829%196 Calculation 10143% 21 9 Analysis/ Synthesis 7934%207 Assumptions 146% 10 0 Communication5323%184 Frequency and Prevalence of Core Competencies in QRCW Casebook

Reliability of QLAR Focus on Case Study 4.3 Readers A and B scored questions 3a, 3b, 3c, and 3d on 24 student papers (section 1) Readers C and D scored 2b, 2c, 2d, 3a, and 3d from 24 papers (section 2)

QuestionCompetencies 2bAssumptions 2cAssumptions 2dAssumptions 3aRepresentation, calculation, analysis/synthesis, communication 3bInterpretation, analysis/synthesis 3cInterpretation, analysis/synthesis 3dRepresentation, calculation Core Competency for Study Questions

Reliability Test 1: Readers A and B QL Competency # of disagreements N% disagreement Interpretation % Representation % Calculation % Analysis/ Synthesis % Communication 0 240% Disagreement Level Per Core Competency Overall: Readers A and B disagreed on 7/238 = 2.94% of assessments. Average of 7 differences: 1.23

Reliability Test 2: Readers C and D Disagreement Level Per Core Competency QL Competency# of disagreements N% disagreement Assumptions % Representation % Calculation % Analysis/ Synthesis 0140% Communication % Overall: Readers C and D disagreed on 15/126 = 11.9% of assessments. Average of 7 differences: (just over) 1

Added Value of QLAR 1. Allows one to align student prompts with an organized thought process as represented by the rubric To solve a complex problem a student needs to read and glean information, convert the information into a usable form, calculate, make or draw a conclusion, and then communicate the result via explanation. Scaffold student learning by sorting case studies by competency: Start with Interpretation & Representation and build up to more complex prompts. Students often do not recognize these as components of a strong response. Guiding students through mastery of these competencies is crucial!

Added Value of QLAR 2. Improved student prompts. Find out how the Standard & Poor’s 500 stock index is computed. Describe how the Standard & Poor’s 500 stock index is computed. Explain your reasoning! Be sure to differentiate between quantitative information taken directly from the article and extra assumptions you made. Justify any extra assumptions. Use a quantitative argument to…[answer some question]

QLAR Achievement Level Quantitative Literacy Core Competency 3210 Interpretation Ability to glean and explain mathematical information presented in various forms (e.g. equations, graphs, diagrams, tables, words) Correctly identifies all relevant information. Correctly identifies some, but not all, relevant information. Some relevant information is identified, but none is correct. No relevant information identified. Representation Ability to convert information from one mathematical form (e.g. equations, graphs, diagrams, tables, words) into another. All relevant conversions are present and correct. Some correct and relevant conversions are present but some conversions are incorrect or not present. Some information is converted, but it is irrelevant or incorrect. No conversion is attempted. Calculation Ability to perform arithmetical and mathematical calculations. Calculations related to the problem are correct and lead to a successful completion of the problem. Calculations related to the problem are attempted but either contain errors or are not complete enough to solve the problem. Calculations related to the problem are attempted but contain errors and are not complete enough to solve the problem. Calculations given are not related to the problem, or no work is present. Analysis/Synthesis Ability to make and draw conclusions based on quantitative analysis. Uses correct and complete quantitative analysis to make relevant and correct conclusions. Quantitative analysis is given to support a relevant conclusion but it is either only partially correct or partially complete (e.g. there are logical errors or unsubstantiated claims). An incorrect quantitative analysis is given to support a conclusion. Either no reasonable conclusion is made or, if present, is not based on quantitative analysis. Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis. All assumptions needed are present and justified when necessary. At least one correct and relevant assumption is given (perhaps coupled with erroneous assumptions), yet some important assumptions are not present. Attempts to describe assumptions, but none of the assumptions described are relevant. No assumptions present. Communication Ability to explain thoughts and processes in terms of what evidence is used, how it is organized, presented, and contextualized. A correct and complete explanation is clearly presented. A partially correct relevant explanation is present, but incomplete or poorly presented. A relevant explanation is present, but is illogical, incorrect, illegible, or incoherent. No relevant explanation is provided.