Two Levels of Course Assessment to Identify Student Learning Carol Lerch, Ph.D S 52 Friday 11:45 – 12:35

Background Assess new format – College Algebra – Introduced fall 2000 – Previous course failure rate +40% AMATYC Assess new format – College Algebra – Introduced fall 2000 – Previous course failure rate +40% Looking at students in one specific major Assess new format – College Algebra – Introduced fall 2000 – Previous course failure rate +40% Looking at students in one specific major Course description – Purpose of this course is to provide students with a proficiency in algebraic skills necessary for success in future coursework …

Collect Data Specifics Fall 2001 – Summer 2007 – Pre % pass (anecdotal) Fall average 85% Spring average 64% Summer 80% Includes Withdraws AMATYC 20073

Analysis Improvement over previous College Algebra course AMATYC Improvement over previous College Algebra course Fall pass rates acceptable – Continuing to improve Improvement over previous College Algebra course Fall pass rates acceptable – Continuing to improve Spring pass rates much lower – Needs analysis & new plan Improvement over previous College Algebra course Fall pass rates acceptable – Continuing to improve Spring pass rates much lower – Needs analysis & new plan Summer needs more data – Online first time summer 07

Changes to Course Added text related course management system – online homework – Interactive textbook – Instant help – Video, animation AMATYC Added text related course management system – online homework – Interactive textbook – Instant help – Video, animation Added section JITTs – Just-in-time-tests

Purpose of AssessmentPlan AMATYC Pass rates – Pass rates for particular course Pass rates – Pass rates for particular course Course sequence – Semesters to complete math sequence Thanks to Jack Rotman Pass rates – Pass rates for particular course Course sequence – Semesters to complete math sequence Thanks to Jack Rotman Concept assessment – Learning goals/objectives

Semesters to Complete Semesters to pass through the math sequence, passing MA152 for each semester since fall 03: – 82% of those who started in MA098 passed MA152 in 4 semesters (4 courses) – 93% of those who started in MA122 passed MA152 in 3 semesters (3 courses) AMATYC 20077

Levels of Assessment Gathering information about students’ achievement (Marzano, 2001, p. 115) AMATYC Level I Assessment – General Course information Pass rates Success in subsequent math courses Semesters to complete sequence

Levels of Assessment AMATYC Level II Assessment – Content specific – Concepts Learning Map o Summative assessment  Overall course content  Final exam Item Analysis o Formative assessment o Individual concepts tested

Cycle of Assessment PlanCollect DataAnalyze AMATYC

Online homework required as of spring 2006 – Introduced as option for fall 2005 Daily quiz – 2-3 questions on concepts in each section Read text Come to class prepared Carrot – replace test grade with quiz average AMATYC Plan

Collect Data Fall pass rate improved overall – 90% Spring stayed same – 63% AMATYC

Analysis Fall pass rates acceptable – Online homework helped Anecdotal evidence – Students like all options available with online homework – Have more opportunities to learn Spring rates need improvement AMATYC

Plan Assesses student learning – Concept assessment Learning goals/objectives Develop “Learning Map” – Identify major concepts Identify teaching objectives Identify student learning objectives Means of assessment Set success standard AMATYC Plan

Learning Map AMATYC

Learning Map 1 st step – Major topics of course Linear equations, quadratic equations, inequalities, absolute value, exponential & log equations Modified as further developed AMATYC st step – Major topics of course Linear equations, quadratic equations, inequalities, absolute value, exponential & log equations Modified as further developed 2 nd step – What do I want them to learn Teaching objectives

Learning Map Knowledge level Cognitive Process – Action verbs AMATYC

Levels of Knowledge AMATYC Factual – Basic elements of the discipline – Facts specific to topic Parts of linear or quadratic equation Methods to solve quadratic equations Factual – Basic elements of the discipline – Facts specific to topic Parts of linear or quadratic equation Methods to solve quadratic equations Procedural – Processes of mathematics – Algorithms, techniques, methods Complete the square Quadratic formula

Levels of Knowledge Conceptual – Interrelationships – Classifications – Principles, generalizations Pythagorean Theorem Properties AMATYC Conceptual – Interrelationships – Classifications – Principles, generalizations Pythagorean Theorem Properties Metacognitive – Self knowledge – Students need to reflect to learn

Teaching vs. Learning Objectives Teaching objective – What I will do – What I want the students to know AMATYC Student learning experience What they will do Student learning objective – “Students will be able to …”

Assessment on Map Each objective has generic question Indicates how student is assessed AMATYC

Learning Map Gave to students as part of syllabus – Type of questions included – ‘Truth in advertizing’ Matched final exam to map – Redesigned map Concepts tested but not part of map – Redesigned final exam Some concepts not tested but important Some concepts tested multiple times AMATYC

Level II Assessment Assess concepts learned Allow overall picture of student progress Shows strengths Identifies weaknesses – Areas for further development AMATYC

Collect Data Collected chapter test data – Right/wrong only Collected final exam item results 4 th semester of data collection Stats: Percent correct AMATYC

Collect Data Identified concept tested by semester – Shows what is not tested (over-tested) AMATYC Results of chapter tests for Fall 06, Spring 07, Summer 07

Analysis - Tests More Student Learning Objectives – Needed to add very specific objectives – Dropped some that I thought I wanted but don’t test for AMATYC More Student Learning Objectives – Needed to add very specific objectives – Dropped some that I thought I wanted but don’t test for Success measure – 65% or better correct More Student Learning Objectives – Needed to add very specific objectives – Dropped some that I thought I wanted but don’t test for Success measure – 65% or better correct Results below need attention More Student Learning Objectives – Needed to add very specific objectives – Dropped some that I thought I wanted but don’t test for Success measure – 65% or better correct Results below need attention % indicates problem with question as well as topic

Analysis – Final exam Final used by all sections – Data just my sections Shows problems with online course on specific concepts AMATYC

Asides Only list what is being assessed – If can’t assess, don’t put as objective AMATYC Have a thick skin Data show changes made in course – Provides longitudinal look at course Allows comparison of student results by semester or year Data show where more emphasis needed in teaching

Before Beginning Assessment Purpose of assessment – What do you want to know? – Why do assessment? Action leading from assessment – What do you do with results? AMATYC

5 Basic Questions AMATYC

References Angelo, T. A. (1991). Introduction and overview: From classroom assessment to classroom research. Classroom research: Early lessons from success. T. A. Angelo and K. P. Cross. San Francisco, Jossey-Bass: Brockbank, A. and I. McGill (1998). Facilitating reflective learning in higher education. Philadelphia, Society for Research into Higher Education & Open University Press. Carson, T. (1990). "What kind of knowing is critical action research?" Theory Into Practice XXIX(3): Cochran-Smith, M. and S. L. Lytle (1993). Inside outside: Teacher research and knowledge. New York, Teachers College. Elliott, J. (1991). Action research for educational change. Philadelphia, PA, Open University Press. Leibowitz, M. (1997). Instruction for process learning. Supporting the spirit of learning: When process is content. A. Costa and R. M. Liebmann. Thousand Oaks, CA, Corwin Press, Inc.: Marzano, R. J. (2001). Designing a new taxonomy of educational objectives. Thousand Oaks, CA, Corwin Press. Mezirow, J. (1990). Fostering critical reflection in adulthood: A guide to transformative and emancipatory learning. San Francisco, Jossey-Bass. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA, Harvard University. AMATYC

Participation Gather into groups Pick one course you teach List the major concepts – Very few topics Pick one or two to use Identify Teaching Objective Identify Student Objective Identify knowledge levels – Can be more then one Identify possible means of assessment Identify success measures AMATYC

SHARE AMATYC

Two Levels of Course Assessment to Identify Student Learning Carol Lerch, Ph.D S 52 Friday 11:45 – 12:35