Presentation on theme: "Don’t Guess... Know When (and How) to Assess William F. McComas, Ph.D. Director of Science Education Programs Fellow, Center for Excellence in Teaching."— Presentation transcript:
Don’t Guess... Know When (and How) to Assess William F. McComas, Ph.D. Director of Science Education Programs Fellow, Center for Excellence in Teaching Rossier School of Education University of Southern California Los Angeles, CA
What is Assessment? Assessment comes from the Latin meaning to “sit with” a learner and verify that the responses really mean what they seem to mean + In most assessment plans such assurance is lacking + Much assessment is suspect in terms of validity and reliability
What Can We Assess? Knowledge Attitudes Skills
Assessment Plans Must: Be free of bias Reflect what is (or should have been) taught Provide information to enhance instruction (teaching or curriculum) Reveal misconceptions Adapted from Champagne, Lovitts and Calinger (1990). Assessment in the Service of Instruction. Washington, AAAS.
Assessment should: Be a learning experience for both students and teachers Involve the evaluation of reasonable standards or benchmarks known in advance to the learners Be based on a well-designed table of specifications The table of specifications lists the learning goals associated with the number, kind, and nature of the individual assessment items
Timing is Everything Diagnostic Evaluation (pretest) Used for gauging prior conceptions on which to base instructional strategies Formative Evaluation (in process) Useful for midcourse corrections Summative Evaluation (posttest) A way of seeing how well you did as a teacher
Sorry, Wrong Number! Norm-referenced; measures of performance against that of others Scores are typically reported in percentiles “Jane was in the 95th percentile on the SAT” Criterion-referenced; absolute measures Scores are typically reported in percents “ Bob got 82% of the questions correct” Problems occur when instructors confuse the two types of assessment goals
Major Assessment Issues High Stakes Test; a single assessment the result of which (positive or negative) will have a large impact on the future of the examinee Certification, licensure, entrance exams, etc. Low Stakes Test; a single assessment the result of which does not have a large impact on the future of the examinee Most teacher-made tests
Major Issues in Assessment Design Reliability ; a measure of the consistency of the results of a given instrument (mathematically determined) Validity ; a measure of the degree to which the instrument measures what it is designed to measure (qualitatively determined)
What is Validity? Validity refers to the accuracy of a measure A measurement is valid when it measures what it is designed to measure and performs the functions that it purports to perform. Does an indicator accurately measure the variable that it is intended to measure?
What is Reliability? Reliability refers to the dependability and consistency of the results provided by an instrument or technique. A measurement is reliable when it produces the same result repeatedly with the same examinees or those from the same sample. Does an indicator repeatedly report the same measure that it is intended to measure?
Forms of Assessment Enhanced: new forms of style, substance and goals Traditional: typical forms of style and substance
Traditional Assessment Goals; assignment of grades, student progress reporting and fault finding Target; learners Timing; summative Methods; objective exams at the recall level of the knowledge domain
Enhanced Assessment Goals; assignment of grades, student and program assessment Target; learners, instructors and the curriculum Timing; formative and summative Methods; objective exams and expanded methods targeting KSA at all levels
Enhanced Assessment Includes More data points (not just summative assessment) More domains (not just knowledge) but attitudes, skills, creativity, etc., Higher levels of all domains (such as synthesis rather than memorization) More techniques such as portfolio, authentic assessment and empirically derived exams
Why Expand Assessment? The nature and focus of assessment tell students (and teachers) what is important Enhanced assessment encourages students to achieve in nontraditional realms (ex; creativity) and in nontraditional ways Expanded assessment results provides the data necessary to make more thoughtful decisions about the curriculum, instruction, and student progress
Multi-Domain Assessment Knowing and Understanding Exploring and Discovering (process skills -- particularly in math and science) Using and Applying Knowledge Imagining and Creating Feeling and Valuing (attitudes) Understanding the nature of the discipline (such as science)
Domain: Knowledge & Understanding Facts Information Concepts Laws (principles) Explanations and theories Processes inherent to the discipline
Discipline-Based Skills and Processes Primary Science Process Skills Observing Using Space/Time Relationships Classifying (Grouping and Ordering) Using Numbers (Quantifying) Measuring Communicating Inferring Predicting
Integrated Science Process Skills Controlling and Identifying Variables Interpreting Data Formulating Hypotheses Defining Operationally Experimenting Discipline-Based Skills and Processes
Domain: Using & Applying Seeing learned concepts in everyday life Applying learned concepts to everyday life Evaluating reports of scientific developments Making personal decisions based on legitimate knowledge Becoming involved in science-related pursuits Taking actions based on what has been learned
Domain: Imagining & Creating Visualizing by producing mental images Combining objects and ideas in unusual or useful new ways Producing alternate uses for objects or ideas Pretending Converging, Diverging and Synthesizing
Domain: Attitudes Values and Feelings Developing positive attitudes toward science in general and school science Exploring and discussing both sides of an issue Expressing personal feelings in a constructive fashion Making decisions based on values
Domain: Understanding the Nature of the Discipline Appraising the strengths and limitations of the discipline and its methods Evaluating and applying appropriate knowledge-production modes Using criteria to place boundaries on the discipline (what is in and what is out) Understanding the social aspects of work in the field
Bloom’s Taxonomy of the Cognitive Domain Knowledge (Recall of data) Comprehension (Understanding) Application (Using data) Analysis (Separating ideas into their parts) Synthesis (Building something out of parts) Evaluation (Making judgments)
Technique: Authentic Assessment These are modes of assessment in which examinees are asked to perform in ways highly related to “real” situations (or situations in which the learning originally occurred) “Real” situations are those that exist in study, work, or life itself The best authentic assessments are both faithful (life-like) and comprehensive (wide- ranging)
Authentic Assessment Involves: Worthy problems or questions of importance Faithful representations of real situations Options, constraints, and access to resources that are legitimate and not arbitrary Problems that require a combination of knowledge, judgment and creativity Judgment standards that are clearly stated in advance Adapted from Wiggins (Nov / 93) Phi Delta Kappan
Technique: Empirically- Derived Exams 1 Determine the standards or bench- marks for achievement 2 Create an open-ended assessment tool 3 Collect data and collapse similar responses into typical response items 4 Now, use the typical responses on a multiple choice type examination
Defining science is difficult because science is complex and does many things. But, MAINLY science is: Athe study of fields like biology, physics, etc. Ba body of knowledge (laws and theories) which explain the world around us Cexploring the unknown and making discoveries Dcarrying out experiments to solve problems Einventing or designing things like computers Fan organization of people (scientists) who have ideas for discovering new knowledge GNone of these choices fits my basic viewpoint
Technique: Performance Assessment Practical examinations, Performances, Exhibits The task must be legitimate and contextualized, not artificial and remote in physical science - students measure the temperature of a liquid vs. in ecology - students determine the chemicals in a sludge sample
Technique: Portfolio Assessment “... a systematic and organized collection of evidence used by the teacher and student to monitor growth of the student’s knowledge, skills, and attitudes in a specific subject area.” Vavrus, L. (August, 1990). Put portfolios to the test. Instructor. pp
What Can Portfolios Contain? Artifacts; tests, lab reports, photographs, meaningful journal entries Reproductions; documents about typical events not usually captured (videos of presentations, photos of projects, etc.) Attestations; something produced by someone else (i.e. notes from the teacher) Productions; documents prepared for the portfolio (goal statement or reflection)
The Portfolio Process Collect materials for the portfolio Select materials to be in the portfolio Reflect on why those materials were selected Project make a presentation based on the portfolio contents Respect the contents and presentation Show the Effect of the portfolio
Issues in Enhanced Assessment How do we evaluate the results of the new schemes? How do we report the results? What do we do with the results in terms of advisement and promotion? What are the solutions to questions of time and expense?
Typical Small Toolbox of Assessment Methods Midterms and Finals Multiple Choice Items True False Items Summative Assessment Philosophy
Larger Toolbox of Assessment Methods Summative Formative True/False Multiple Choice Term Papers Labs / Practical exams Not graded Perhaps on-line Feedback to all concerned Immediate feedback Allow for real time adjustments Graded End of a Unit Feedback to students Minute Paper Empty Outlines Approximate Analogies Etc.
Don’t Guess... Know you Know How to Assess William F. McComas, Ph.D. Director of Science Education Programs Fellow, Center for Excellence in Teaching Rossier School of Education University of Southern California Los Angeles, CA