1. Assessing When Numbers Don’t Count Binghamton University March 23, 2007

2. Today’s Objectives Define what a discourse-based, or qualitative assessment method is Define what a discourse-based, or qualitative assessment method is Review a few of these methods Review a few of these methods Work with some case studies to better understand how these methods might be used Work with some case studies to better understand how these methods might be used Discuss the uses and limitations of these methods, especially with regard to assessment Discuss the uses and limitations of these methods, especially with regard to assessment

3. The Pressure to Assess This week—Spellings Commission meeting to discuss several issues. One of them is student learning outcomes This week—Spellings Commission meeting to discuss several issues. One of them is student learning outcomes There is consistent pressure to use standardized tests and surveys; we are currently using one of them, the NSSE There is consistent pressure to use standardized tests and surveys; we are currently using one of them, the NSSE Through the use of standardized tests and surveys, we gain perspective on how we compare to other institutions Through the use of standardized tests and surveys, we gain perspective on how we compare to other institutions We might be able to gain a “value added” perspective We might be able to gain a “value added” perspective

4. Weaknesses of the Standardized Test/Survey Approach We obtain a “macro” perspective, but may not gain a “micro” perspective We obtain a “macro” perspective, but may not gain a “micro” perspective We have little control over the questions We have little control over the questions “Value added” is still difficult to establish with standardized tests “Value added” is still difficult to establish with standardized tests Faculty see little value in using standardized tests, especially in interdisciplinary studies and the liberal arts Faculty see little value in using standardized tests, especially in interdisciplinary studies and the liberal arts Standardized tests/surveys often assume they can measure most of what concerns faculty in respect to teaching and learning Standardized tests/surveys often assume they can measure most of what concerns faculty in respect to teaching and learning

5. Forging the Middle Ground: Discourse-Based Assessment Methods Allow for the discovery of the unanticipated Allow for the discovery of the unanticipated Relevant to interdisciplinary study Relevant to interdisciplinary study Maximizes faculty/staff input when discourse is highly valued Maximizes faculty/staff input when discourse is highly valued Good to use when the number of objectives outweigh the amount of time available to assess student learning Good to use when the number of objectives outweigh the amount of time available to assess student learning Great contextualizer Great contextualizer When performed carefully and compared with other methods, great return on investment When performed carefully and compared with other methods, great return on investment

6. Types of Qualitative Assessment Focus groups Focus groups Expert panels Expert panels Open-ended surveys Open-ended surveys Ethnographic studies (participant observations) Ethnographic studies (participant observations) Portfolio reviews Portfolio reviews Primary trait scoring Primary trait scoring Delphi panels Delphi panels

7. Qualitative Assessment is an Inductive Process Defined Focus Observations Analysis Summary Report Comparison with Other Assessments Action

8. Expert Panel Type of focus group Type of focus group Focus is on (for assessment purposes) a particular assignment or performance Focus is on (for assessment purposes) a particular assignment or performance Not a simple conversation; is done methodically, with precision, and is systematic Not a simple conversation; is done methodically, with precision, and is systematic Often involves convenience or random samples of a homogeneous population Often involves convenience or random samples of a homogeneous population Must include carefully written questions Must include carefully written questions Might include a survey or other assessment technique as part of the process Might include a survey or other assessment technique as part of the process

9. Krueger’s 10 Quality Factors in Focus Group (Expert Panel) Research Clarity of purpose Clarity of purpose Appropriate environment Appropriate environment Sufficient resources Sufficient resources Appropriate participants Appropriate participants Skillful moderator Skillful moderator Effective questions Effective questions Careful data handling Careful data handling Systematic and verifiable analysis Systematic and verifiable analysis Appropriate presentation Appropriate presentation Honoring the participant, clarity, and method Honoring the participant, clarity, and method

10. Expert Panel Procedure Select FocusSelect Experts Write Questions Use Additional Method? Logistics Conduct Expert Panel— How to Assess Agreement? Report Results

11. Final Thoughts on Expert Panels Good method of assessing “ineffable outcomes” Good method of assessing “ineffable outcomes” Better when expert panel questions and conversations are grounded in standards and/or student learning objectives Better when expert panel questions and conversations are grounded in standards and/or student learning objectives Also good when specific focus is on a particular assignment or performance Also good when specific focus is on a particular assignment or performance It is advisable to use a secondary method either prior to or during expert panel It is advisable to use a secondary method either prior to or during expert panel Does not control for anonymity among respondents Does not control for anonymity among respondents

12. Primary Trait Scoring Focus is upon one particular assignment, performance, etc., that is reflective of several aggregate student learning outcomes Focus is upon one particular assignment, performance, etc., that is reflective of several aggregate student learning outcomes Rate each outcome according to a scale— ex., proficient, satisfactory, unsatisfactory Rate each outcome according to a scale— ex., proficient, satisfactory, unsatisfactory Idea is to look at trends, not numbers, that spark discussion Idea is to look at trends, not numbers, that spark discussion

13. Primary Trait Scoring--Procedure Choose an assignment in which students demonstrate summative knowledge, skills, or competencies Choose an assignment in which students demonstrate summative knowledge, skills, or competencies Carefully rate student performance according to the scale Carefully rate student performance according to the scale Place checkmarks in each column Place checkmarks in each column Look for visual trends Look for visual trends Discuss why these trends occur, what basis these rating occurred, and what specific issues are revealed through the analysis Discuss why these trends occur, what basis these rating occurred, and what specific issues are revealed through the analysis Combine with other findings, or make plans for action Combine with other findings, or make plans for action

14. Example of Primary Trait Analysis ElementExcellentSatisfactoryUnsatisfactory Demonstrate an ability to research the financial integrity of a business plan through accounting/financ ial analysis √√√√ √√√√√ √ √√ Demonstrate an ability to assess a plan’s organizational integrity; that is, a business plan demonstrates how an organization will be built and sustained to under gird the success of the business being organized √√√√√√ √√√√√ √

15. Primary Trait Analysis—Final Thoughts Method to get faculty or staff to talk about what assessment results mean Method to get faculty or staff to talk about what assessment results mean A good starting point toward developing a rubric A good starting point toward developing a rubric Enables discussion, which can lead to further discovery Enables discussion, which can lead to further discovery “Simple, stress free, and easy” “Simple, stress free, and easy”

16. Delphi Panel Introductory Exercise Divide into 3 groups Divide into 3 groups Get out piece of paper and individually write down, “what do students have the most difficulty with when they first come to college (as first-year students)?” Get out piece of paper and individually write down, “what do students have the most difficulty with when they first come to college (as first-year students)?” Try to create frequency counts—combine like answers and tally them Try to create frequency counts—combine like answers and tally them Discuss Discuss

17. Questions for Groups What do these say about the difficulty students might have when they start? What do these say about the difficulty students might have when they start? Take a look at the most “popular” answer—do these ordinarily achieve “majority vote status? Take a look at the most “popular” answer—do these ordinarily achieve “majority vote status? Even in cases where “majority vote status” is achieved, might less popular answers indicate group consensus? Even in cases where “majority vote status” is achieved, might less popular answers indicate group consensus?

18. Introduction to the Delphi Method Combination of at least 3 methods—open-ended survey, closed-ended survey, and expert panel Combination of at least 3 methods—open-ended survey, closed-ended survey, and expert panel Unlike expert panel, attempts to maximize anonymity of respondents to control for power dynamics among these respondents Unlike expert panel, attempts to maximize anonymity of respondents to control for power dynamics among these respondents Assumes highly motivated groups of experts (faculty or staff) willing to participate in more than one round of questions Assumes highly motivated groups of experts (faculty or staff) willing to participate in more than one round of questions

19. Introduction to the Method 1. Find homogeneous group of experts who can comment either on one assignment or specific student learning outcomes 2. Create an open-ended survey in which respondents are asked to identify strengths and weaknesses in student performance in reference to specific standard or student learning outcome 3. Content analyze responses by combining like responses, placing how many times each was mentioned in parentheses 4. “Cut and paste” these onto a survey, and ask respondents to indicate to what extent they agree with each on a 4 or 5 point scale 5. Report those responses that indicate consensus 6. If needed, move to 3 rd round, in which respondents rank these consensus items

20. Strengths of Delphi Method A way of addressing “ineffable outcomes” A way of addressing “ineffable outcomes” Can be used to designate the most agreed-upon student learning objectives that faculty have communicated Can be used to designate the most agreed-upon student learning objectives that faculty have communicated Can be used to gather information from employers, internship supervisors, alumni, etc. about specific items of interest Can be used to gather information from employers, internship supervisors, alumni, etc. about specific items of interest

21. Limitations of Delphi Method Can be time consuming Can be time consuming Takes some knowledge of statistics Takes some knowledge of statistics Not a method that can be used by itself; usually results need to be compared with direct assessments of student learning Not a method that can be used by itself; usually results need to be compared with direct assessments of student learning

22. Today’s Activities Separate into three groups; select group note taker Separate into three groups; select group note taker If you have not already, read the case study packets If you have not already, read the case study packets As a group, discuss questions at end of case study—debate, applaud, etc.—do something active As a group, discuss questions at end of case study—debate, applaud, etc.—do something active Write answers on provided sheet of paper Write answers on provided sheet of paper