Presentation on theme: "Why I don’t understand the assessment results (and what I can do about it)"— Presentation transcript:
Why I don’t understand the assessment results (and what I can do about it)
Hi…I’m the Assessment Director
Reasons for Assessment Accreditors demand it Institutional leaders ought to care Faculty members probably do care Students should know what we think of them
What Could Go Wrong???
Take Action: Now what?
Take Action: Now What?
Assessment: The Process 1. Design 2. Implementation 3. Reporting 4. Follow-up
Design: Validity and Reliability The emphasis [on validity] is not on the instrument itself; rather, it is on the interpretation of the scores yielded by a test. -- College BASE Technical Manual The CLA measures were designed by nationally recognized experts in psychometrics and assessment, and field tested in order to ensure the highest levels of validity and reliability. -- Collegiate Learning Assessment Advertisement
Design: Epistemology We say that a sentence is factually significant to any given person, if and only if, [she or] he knows how to verify the proposition which it purports to express—that is, if [she or] he knows what observations would lead [her or him], under certain conditions, to accept the proposition as being true, or reject it as being false. – A. J. Ayer, Language, Truth, and Logic [T]he meaning of a word is its usage in the language. – L. Wittgenstein
Design: Complexity and Validity Collegiate learning is complex Assessment should be: Authentic Contextual Rigorous Reliable Understandable
Design: Measurement Smeasurement Measurement requires: Units Ability to aggregate What we actually do is estimation
Analysis: Using the Data Pieper, Fulcher, and Erwin – 1. Differences 2. Relationships 3. Change 4. Competency
Analysis: Tools MS Access Spreadsheet software Pivot Tables Statistics packages: Excel SPSS SAS Logistic Regression:
CIRP Self-Assessed Mathematical ability Frequen cyPercent Valid Percent Cumulative Percent ValidLowest 10% Below average Average Above average Highest 10% Total MissingSystem 1.5 Total
Conclusions At strategic level, averages are fine At tactical level, proportions are easier to act on Abstraction is the enemy Don’t sacrifice validity for reliability without a good reason For complex skills, subjectivity is your friend