1. Assessing for Critical Thinking
Narrowing the Gulf Conference
March 2009

2. Narrowing the Gulf Conference
St. Petersburg College
Presenters
Dr. Carol Weideman, Mathematics Professor, and Assessment Coordinator for the QEP
Maggie Tymms, Assessment Coordinator for Academic Programs
Dr. James Coraggio, Director of Institutional Research and Effectiveness
March 27, 2009
Narrowing the Gulf Conference

3. Narrowing the Gulf Conference
St. Petersburg College
SPC, established in 1927, is the oldest
2-year college in Florida
First Community College in Florida to offer 4 year degrees (2002)
9 Campuses throughout the county
FTE: 16,086 (LD), 1,415 (UD)
Opening Fall 2008 credited enrollment: 27,076
Annual headcount: 62,200
March 27, 2009
Narrowing the Gulf Conference

4. Narrowing the Gulf Conference
SACS
SPC was recently reaffirmed in 2008 as part of the SACS affirmation process. Keys to the process included…
Compliance with all SACS requirements and standards
Assessment of all academic programs and support processes
Credentialing of all faculty and key staff
Continuous improvement of student learning through a Quality Enhancement Plan (QEP)
March 27, 2009
Narrowing the Gulf Conference

5. The Quality Enhancement Plan (QEP)
Second major requirement by SACS (after reaffirmation). Requirements include…
Focus: Significant issue related to student learning
Faculty-driven
Broad-based involvement
Institution demonstrates ability to carry out plan
Includes methods of assessment
Report results to SACS after five years
March 27, 2009
Narrowing the Gulf Conference

6. Broad-based involvement
Selected ‘improve student learning through the enhancement of CT skills’
First step after deciding on a topic was the development of a standardized definition of CT
Involved the entire college including a web blog review process to gather information
March 27, 2009
Narrowing the Gulf Conference

7. Narrowing the Gulf Conference
St. Petersburg College
SPC’s definition of Critical Thinking:
“Critical thinking is the active and systematic process of communication, problem-solving, evaluation, analysis, synthesis, and reflection, both individually and in community, to foster understanding, support sound decision-making, and guide action.”
March 27, 2009
Narrowing the Gulf Conference

8. Measureable Learning Outcomes
In order to link specific and measureable student learning outcomes, SPC’s definition of critical thinking was operationalized in order to provide a more concrete and less abstract linkage or bridge between the original SPC definition of critical thinking and the student learning outcomes.
March 27, 2009
Narrowing the Gulf Conference

9. Measureable Learning Outcomes
March 27, 2009
Narrowing the Gulf Conference

10. Narrowing the Gulf Conference
Learning Outcomes
March 27, 2009
Narrowing the Gulf Conference

11. Narrowing the Gulf Conference
Learning Outcomes
March 27, 2009
Narrowing the Gulf Conference

12. Linkage between Student Learning Outcomes and Assessments
SPC’s QEP contained a number of direct measures and indirect measures for use in assessing student learning in the area of critical thinking.
These assessments have been linked to the specific Core Operational Elements (KSAs) and student learning outcomes.
Provides the relationship between the Core Operational Elements (KSAs), the student learning outcomes, and the specific assessment measures.
March 27, 2009
Narrowing the Gulf Conference

13. Linkage between Student Learning Outcomes and Assessments
March 27, 2009
Narrowing the Gulf Conference

14. Linkage between Student Learning Outcomes and Assessments
SPC will determine improvement in students’ critical thinking skills using the multiple measures.
These include standardized instruments, authentic assessments, and indirect methods.
Before we can measure Critical Thinking…
we need to understand what it looks like.
March 27, 2009
Narrowing the Gulf Conference

15. Narrowing the Gulf Conference
Class Without CT
Didactic instruction where students are presented with factual information from a text book
Assessment is primarily multiple choice items where students are expected to regurgitate factual information
March 27, 2009
Narrowing the Gulf Conference

16. Narrowing the Gulf Conference
Class With CT
Interactive learning environment where students not only learn facts but the relationship between the facts and the application of that information
Authentic assessment where students are able to model the applications of the discipline through simulations, projects, etc.
March 27, 2009
Narrowing the Gulf Conference

17. Linkage between Student Learning Outcomes and Assessments
How do you measure Critical Thinking in your classroom?
March 27, 2009
Narrowing the Gulf Conference

18. Authentic Assessments
Authentic assessments serve dual purposes of
encouraging students to think critically and
providing assessment data for measuring improved student learning.
These assessment techniques fall into three general categories:
criterion-referenced rubrics,
student reports (reflection or self-assessments), and
student portfolios.
March 27, 2009
Narrowing the Gulf Conference

19. Assessment Rubric for Critical Thinking
SPC developed a multi-measure model to assess critical thinking skills, which includes a critical thinking rubric.
Assessment Rubric for Critical Thinking (ARC)
A global rubric template developed to provide a snapshot view of how student learning is being affected by the critical thinking initiative.
March 27, 2009
Narrowing the Gulf Conference

20. Assessment Rubric for Critical Thinking
Designed to be flexible enough to address a number of student project modalities including written and oral communications.
Will evaluate the student’s use of critical thinking skills in the development of the paper as opposed to specifically evaluating the quality of student’s writing skills.
March 27, 2009
Narrowing the Gulf Conference

21. Assessment Rubric for Critical Thinking
ARC was designed by the QEP staff and the Faculty Champions to…
Enhance the QEP
Align with the College’s definition of critical thinking
Be flexible for use in multi-disciplines
March 27, 2009
Narrowing the Gulf Conference

22. Assessment Rubric for Critical Thinking
Performance
Element
Exemplary
(4)
Proficient
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
I. Communication
Define problem in your own words.
Identifies the main idea or problem with numerous supporting details and examples which are organized logically and coherently.
Identifies the main idea or problem with some supporting details and examples in an organized manner.
Identifies the main idea or problem with few details or examples in a somewhat organized manner.
Identifies the main idea or problem poorly with few or no details or states the main idea or problem verbatim from the text.
Does not identify the main idea or problem.
II. Analysis
Compare & contrast the available solutions.
Uses specific inductive or deductive reasoning to make inferences regarding premises; addresses implications and consequences; identifies facts and relevant information correctly.
Uses logical reasoning to make inferences regarding solutions; addresses implications and consequences; Identifies facts and relevant information correctly.
Uses superficial reasoning to make inferences regarding solutions; Shows some confusion regarding facts, opinions, and relevant, evidence, data, or information.
Makes unexplained, unsupported, or unreasonable inferences regarding solutions; makes multiple errors in distinguishing fact from fiction or in selecting relevant evidence.
Does not analyze multiple solutions.
III. Problem Solving
Select & defend your final solution.
Thoroughly identifies and addresses key aspects of the problem and insightfully uses facts and relevant evidence from analysis to support and defend potentially valid solutions.
Identifies and addresses key aspects of the problem and uses facts and relevant evidence from analysis to develop potentially valid conclusions or solutions.
Identifies and addresses some aspects of the problem; develops possible conclusions or solutions using some inappropriate opinions and irrelevant information from analysis.
Identifies and addresses only one aspect of the problem but develops untestable hypothesis; or develops invalid conclusions or solutions based on opinion or irrelevant information.
Does not select and defend a solution.
March 27, 2009
Narrowing the Gulf Conference

23. Assessment Rubric for Critical Thinking
Performance
Element
Exemplary
(4)
Proficient
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
IV. Evaluation
Identify weaknesses in your final solution.
Insightfully interprets data or information;
identifies obvious as well as hidden assumptions, establishes credibility of sources on points other than authority alone, avoids fallacies in reasoning; distinguishes appropriate arguments from extraneous elements; provides sufficient logical support.
Accurately interprets data or information;
identifies obvious assumptions, establishes credibility of sources on points other than authority alone, avoids fallacies in reasoning; distinguishes appropriate arguments from extraneous elements; provides sufficient logical support.
Makes some errors in data or information interpretation; makes arguments using weak evidence; provides superficial support for conclusions or solutions.
Interprets data or information incorrectly;
Supports conclusions or solutions without evidence or logic; uses data, information, or evidence skewed by invalid assumptions; uses poor sources of information; uses fallacious arguments.
Does not evaluate data, information, or evidence related to final solution.
V. Synthesis
Suggest ways to improve/strengthen your final solution.
Insightfully relates concepts and ideas from multiple sources; uses new information to enhance final solution; recognizes missing information; correctly identifies potential effects of new information.
Accurately relates concepts and ideas from multiple sources; uses new information to enhance final solution; correctly identifies potential effects of new information.
Inaccurately or incompletely relates concepts and ideas from multiple sources; shallow determination of effect of new information on final solution.
Poorly integrates information from more than one source to support final solution; Incorrectly predicts the effect of new information on final solution.
Does not identify new information for final solution.
March 27, 2009
Narrowing the Gulf Conference

24. Assessment Rubric for Critical Thinking
Performance
Element
Exemplary
(4)
Proficient
(3)
Developing
(2)
Emerging
(1)
Not Present
(0)
VI. Reflection
Reflect on your own thought process.
“What did you learn from this process?”
“What would you do differently next time to improve?”
Identifies strengths and weaknesses in own thinking: recognizes personal assumptions, values and perspectives, compares to others’, and evaluates them in the context of alternate points of view.
Identifies strengths and weaknesses in own thinking: recognizes personal assumptions, values and perspectives, compares to others’, with some comparisons of alternate points of view.
Identifies some personal assumptions, values, and perspectives;
recognizes some assumptions, values and perspectives of others; shallow comparisons of alternate points of view.
does not consider alternate points of view.
Does not reflect on own thinking.
March 27, 2009
Narrowing the Gulf Conference

25. Sample Scenario (Deer)
Three teenagers were seriously injured in a car accident when swerving to avoid a deer on a two-lane road near a small, rural town in Florida. The residents of the town have seen more and more deer enter the town’s populated areas over recent years. Local law enforcement has been called numerous times this year to remove the animals from backyards and neighborhood streets, and one deer even caused considerable damage as it entered a restaurant in town. The mayor has been charged by the city leaders to keep the town residents safe.
March 27, 2009
Narrowing the Gulf Conference

26. Sample Scenario (Deer)
Local crops have even been damaged by the animals. Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population. One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas. Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.
March 27, 2009
Narrowing the Gulf Conference

27. Narrowing the Gulf Conference
Group Project
Task 1: Communication
Define the main idea or problem in your own words.
March 27, 2009
Narrowing the Gulf Conference

28. Narrowing the Gulf Conference
Small Group Project
Task 2: Analysis
Identify the Three Available Solutions
Compare and Contrast Available Solutions to the Problem.
March 27, 2009
Narrowing the Gulf Conference

29. Sample Scenario (Deer)
Possible Solutions:
Some long time residents have requested that the hunting season and catch limits be extended in order to reduce the deer population.
One city leader even proposed that the city purchase electronic devices to deter the deer from entering populated areas.
Health concerns have recently been elevated as three deer carcasses were found at the edge of town and local law enforcement suspect that the animals had been poisoned.
March 27, 2009
Narrowing the Gulf Conference

30. Narrowing the Gulf Conference
Task 3: Problem-Solving Select the best solution from the possible solutions analyzed.
Hunting season and catch limits
Electronic devices to deter the deer
Poison Animals
0 of 30
March 27, 2009
Narrowing the Gulf Conference

31. Narrowing the Gulf Conference
Group Project
Task 4: Evaluation
Critically evaluate the weaknesses to the solution you chose.
March 27, 2009
Narrowing the Gulf Conference

32. Narrowing the Gulf Conference
Small Group Project
Task 5: Synthesis
Explain ways to improve and strengthen the solution you chose, and improve any weaknesses you identified. (You may introduce new information)
March 27, 2009
Narrowing the Gulf Conference

33. Narrowing the Gulf Conference
Group Project
Task 6: Reflection
What did you learn from this process?
What were the strengths and weaknesses in your thinking process?
What personal assumptions influenced your thinking during the process?
What could you do differently to improve your thinking process?
March 27, 2009
Narrowing the Gulf Conference

34. The Deer Scenario was a Helpful Critical Thinking Exercise.
Strongly Agree
Agree
Disagree
Strongly Disagree
0 of 30
March 27, 2009
Narrowing the Gulf Conference

35. Narrowing the Gulf Conference
Lessons Learned
Provide rubric to students prior to assessment
Anonymous papers
Anchor papers defining levels of proficiency for reference
Use of multiple scorers
Interrater reliability statistics during training and grading
March 27, 2009
Narrowing the Gulf Conference

36. Other Critical Thinking Assessments
The CAT - Critical Thinking Assessment Test is designed to assess and promote the improvement of critical thinking and real-world problem solving skills.  The instrument is the product of extensive development, testing, and refinement with a broad range of institutions, faculty, and students across the country. 
The iSkills™ assessment (former ICT Literacy Assessment), externally developed by ETS, Inc., is a comprehensive test of Information and Communication Technology proficiency that uses scenario-based critical thinking tasks to measure both cognitive and technical skills.
The Measure of Academic Proficiency and Progress (MAPP) test, externally developed by Educational Testing Services, Inc. (ETS), Inc., is a measure of college-level reading, mathematics, writing, and critical thinking in the context of the humanities, social sciences, and natural sciences.
March 27, 2009
Narrowing the Gulf Conference

37. Critical Thinking Assessment Test
“The CAT instrument is a unique tool designed to assess and promote the improvement of critical thinking and real-world problem solving…
The test was designed to be interesting and engaging for students. Most of the questions require short answer essay responses and a detailed scoring guide helps insure good scoring reliability”
(Tennessee Tech University, Critical Thinking Assessment Test Overview).
March 27, 2009
Narrowing the Gulf Conference

38. Critical Thinking Assessment Test
March 27, 2009
Narrowing the Gulf Conference

39. Critical Thinking Assessment Test
March 27, 2009
Narrowing the Gulf Conference

40. Critical Thinking Assessment Test
In collaboration with Tennessee Technological University and with support from the National Science Foundation, St. Petersburg College received a grant to administer the Critical Thinking Assessment Test (CAT) instrument to a representative sample of approximately 100 students enrolled in the College during 2008.
March 27, 2009
Narrowing the Gulf Conference

41. Critical Thinking Assessment Test
SPC administrators attended regional training workshops at Tennessee Technological University in 2008 and 2009.
Eighty-seven CAT assessments were administered during 2008 to SPC students enrolled in five courses.
A CAT Scoring Session was held on July 11th, 2008 to score these assessments.
March 27, 2009
Narrowing the Gulf Conference

42. Critical Thinking Assessment Test
Source:
March 27, 2009
Narrowing the Gulf Conference

43. Narrowing the Gulf Conference
I Will Be Able To Apply The Information I Learned In This Presentation.
Strongly Agree
Agree
Disagree
Strongly Disagree of 30
March 27, 2009
Narrowing the Gulf Conference

44. I Enjoyed This Presentation!of 30
Strongly Agree
Agree
Disagree
Strongly Disagree
March 27, 2009
Narrowing the Gulf Conference

45. Assessing for Critical Thinking
Narrowing the Gulf Conference
March 2009