Presentation on theme: "Intelligence, Creativity, and Wisdom: Ability Assessments for the New Millennium Robert J. Sternberg Dean of the School of Arts and Sciences Professor."— Presentation transcript:
Intelligence, Creativity, and Wisdom: Ability Assessments for the New Millennium Robert J. Sternberg Dean of the School of Arts and Sciences Professor of Psychology Tufts University
Main Message Current tests used for assessing abilities are narrow and do an injustice to certain groups We can create and have created broader, more equitable, better assessments
Organization of Talk Context Theory of Successful Intelligence The Rainbow Project The Tufts Kaleidoscope Project The Aurora Project Related Projects Conclusions
Context How have technologies changed since one century ago: –Computing –Telecommunications –Photography –Medical (medicines and diagnostic equipment)
Context –Standardized tests!
Closed Systems in Higher Education Test scores Socioeconomic status Gender Religion Socially-defined race Caste
Closed Systems in Higher Education Height!
Self-Fulfilling Prophecies Person X is a member of Group A People in Group A appear generally not to succeed Person X will not succeed Person X does not succeed Therefore, people in Group A really are losers
Theoretical Background: The Augmented Theory of Successful Intelligence People are SUCCESSFULLY INTELLIGENT, according to the theory, when they have the abilities to succeed in life according to their own standards within their sociocultural context, by: 1. Recognizing and capitalizing on strengths; 2. Recognizing and correcting or compensating for weaknesses.
Theoretical Background Capitalization and compensation enable people to balance: 1. Adaptation to existing environments; 2. Shaping environments to improve them; 3. Selecting new environments.
Theoretical Background To succeed, a person needs –Creative skills and attitudes flexibly to generate new ideas to adapt to a changing environment –Analytical skills and attitudes to ascertain whether his/her or others’ ideas are good ones –Practical skills and attitudes to implement the ideas and persuade others of their value –Wisdom-based skills and attitudes to ensure that the ideas help foster a common good by balancing intrapersonal, interpersonal, and extrapersonal interests over the long- and short-terms through the infusion of positive ethical values
Theoretical Background Analytical skills and attitudes are evoked when we analyze compare and contrast evaluate explain critique
Theoretical Background Creative skills and attitudes are evoked when we: create design invent imagine suppose
Theoretical Background It is hard to think creatively because –We are, for the most part, socialized not to think creatively, or to think creatively only within narrow boundaries, lest we be punished –Creative thinking involves defying the crowd, which is hard to do because of External pressure to conform Internal pressure to conform
Theoretical Background Practical skills and attitudes are involved when we: use apply implement employ contextualize
Theoretical Background Wisdom-based skills and attitudes are involved when we: think dialogically think dialectically judge according to our ethical values balance competing interests over the long-term as well as short-term seek a solution representing a common good
The Rainbow Project: A Plan for Augmentation of the SAT Robert J. Sternberg And the Collaborators of the Rainbow Project
The Context of the Rainbow Project Early Admissions Procedures—The Days of the “Aristocracy” The New Wave in Admissions—The Days of the “Meritocracy” What Went Wrong: What Meritocracy? Some Proposed Fixes –Affirmative Action –Compensatory Scoring –The End of Testing The Rainbow Mission
The “Meritocracy”: Who Benefits and Who Doesn’t? A Winner: Alice A Loser: Barbara A Loser: Celia
Overarching Goal To develop a battery assessing analytical, creative, and practical intellectual skills that is reliable and construct valid, increasing prediction of academic and life performance beyond that obtained from conventional “g-based” measures
Specific Project Goals To develop and preliminarily validate a set of based instruments to augment the prediction of the SAT –Measuring analytical, creative, and practical thinking skills in college settings –Via paper-and-pencil and performance assessments
Method: Materials Outcome Variable (the criterion) –College GPA Baseline Materials –SAT-Verbal –SAT-Math –SAT-Total –ACT or PSAT scores if SAT scores were not available
Method: Materials Analytical 15 multiple choice items Including verbal, quantitative, and figural content 4 response options per item
Method: Materials Practical 15 multiple choice items Including verbal, quantitative, and figural 4 response options per item Performance tasks – Tacit-knowledge inventories College Life (15 vignettes) General Workplace/Common Sense (15 vignettes) Everyday Situational Judgment - Movies (7 vignettes)
Vignette Sample Item College Life Tacit-Knowledge Inventory Not at all NeitherExtremely Characteristic Characteristic nor Characteristic Uncharacteristic You are enrolled in a large introductory lecture course. Requirements consist of three term-time exams and a final. Please indicate how characteristic it is of your behavior to spend time doing the following, if your goal is getting an A in the course. ___Attending class regularly. ___Attending optional weekly review sessions, if there are any, with the T.A. ___Reading assigned text chapters thoroughly. ___Taking comprehensive class notes. ___Speaking with the Professor after class and during office hours. ___Talking to students who took the course last year. ___Studying regularly instead of cramming in the night before exams. ___Doing the extra credit or optional reading assignments. ___Skimming the required reading in the morning before class.
Vignette Sample Item General Workplace/Common Sense Tacit-Knowledge Inventory Extremely Very Somewhat Neither Bad Somewhat Very Extremely Bad Bad Bad Nor Good Good Good Good You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do? ___Tell the worker that you think he is worthless. ___Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor. ___Avoid all conversation and eye contact with the other worker. ___Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day. ___Tell your supervisor that you refuse to work with this man. ___The project is going to be impossible to accomplish with this worker, so you may as well not even try--you can always blame your bad work partner. ___See if you can convince one of your friends to take your place and work with this employee. ___Demand a raise from your supervisor; you should not have to tolerate these conditions.
Everyday Situational Judgment - Movies Examinees see seven digitized movies depicting various real-life situations that college students confront or may confront: The Party: Entering a party where one does not know anyone A Fair Portion: Discussing shares of rental payments for a flat Professor’s Dilemma: Asking for a letter of recommendation from a professor who does not know you very well
Everyday Situational Judgment – Movies, Contd. No Free Lunch: Having eaten a lunch and discovering that you do not have the money to pay for it The Unwanted Guest: Dealing with a friend in need of help at a time when you are just seeing your significant other for the first time in a long time Pressing Corporate Matter: Making a decision regarding proactive actions that can be taken before a wave of firings commences in your company Jerry’s “Beauty-rest Sleeper”: Organizing your friends to move your furniture to a new flat
Method: Materials Creative 15 multiple choice items (STAT – Level H) Including verbal, quantitative, and figural 4 response options per item Performance tasks 2 written stories 2 oral stories 3 cartoon captions
Written Stories SHORT STORY TASK: TITLES “A Fifth Chance” “2983” “Beyond the Edge” “The Octopus’s Sneakers” “It’s Moving Backwards” “Not Enough Time”
Oral Stories SHORT STORY TASK The following task may at first seem somewhat novel; however, I am confident that you will have no difficulty doing it. You will be presented with several sheets of paper, each containing several images. You are to choose two of these pages and to formulate a short story for each. In order to expedite the process, rather than write out each of the stories, you will dictate them to a cassette recorder. When recording: please speak clearly; state your name; spell your name; and state the page number of each story. There is no limitation in the content of the story or the manner in which you decide to present it. Do you have any questions?…. So, if there are no [more] questions as to what we are asking you to do, please get started.
Cartoon Titles Examinees see five cartoons and need to provide titles for three of the five.
Predicting GPA: All measures (practical before creative)* Step 1: SAT-M SAT-V HSGPA Step 2: + Analytic Step 3 : + Practical Step 4 : + Creative *Controlling for school quality in dependent variable
Regressions: In sum In the Rainbow sample, –Adding Rainbow measures over SAT roughly doubles prediction of college success –Adding Rainbow measures over SAT + High School GPA increases prediction by roughly half
Amount of Each Measure That Is Predicted by Racial / Ethnic Differences (ω²)
Effect Sizes: Cohen’s d with Whites as Reference Group Blacks Latinos Asians Nat.Am. SAT-M SAT-V SAT-T
Effect Sizes: Cohen’s d with Whites as Reference Group Blacks Latinos Asians Nat.Am. STAT-A STAT-C STAT-P
Effect Sizes: Cohen’s d with Whites as Reference Group Blacks Latinos Asians Nat.Am. Movies Common Sense College Life
Effect Sizes: Cohen’s d with Whites as Reference Group Blacks Latinos Asians Nat.Am. Cartoons Oral Stories Written Stories
Group Difference Analyses: In Sum In the Rainbow sample: –Rainbow measures reduce ethnic-group differences relative to the SAT alone –The new measures reduce differences because different ethnic groups show different average patterns –Differences are not eliminated, however
The Tufts Kaleidoscope Project Preliminary Results (’06-’08) Robert J. Sternberg and the Kaleidoscope Collaborators Class of 2011 With special thanks to Tzur Karelitz
Goal Insert analytical, creative, practical as well as wisdom-based essays as part of the Tufts-specific admissions application in order to broaden the way we think about applicants Change categories for rating system of applicants
Tufts Admissions Tufts applicants complete the Common Application and (if they choose to) the Supplemental Application. Applications are rated on 4 dimensions: –Academic (AC), –Personal Quality (PQ), –Extracurricular (EX), and –Overall (OV). Final admission decision is made by a committee based on multiple criteria.
The Kaleidoscope Framework Kaleidoscope (Kscope) is a framework for evaluating applicants’ –Creative (C), –Practical (P), and –Wisdom (W) skills. Applicants submitted optional essays. Admission officers were trained and asked to use a Kscope rubric to evaluate applicants.
Essay prompts (Year 1) 1. The late scholar James O. Freedman referred to libraries as "essential harbors on the voyage toward understanding ourselves." What work of fiction or non-fiction would you include in a personal library? Why? 2. An American adage states that "curiosity killed the cat." If that is correct, why do we celebrate people like Galileo, Lincoln, and Gandhi, individuals who thought about longstanding problems in new ways or who defied conventional thinking to achieve great results? 3. History's great events often turn on small moments. For example, what if Rosa Parks had given up her seat on that Montgomery bus in 1955? What if Pope John Paul I had not died in 1978 after a month in office? What if Gore had beaten Bush in Florida and won the 2000 U.S. Presidential Election? Using your knowledge of American or world history, choose a defining moment and imagine an alternative historical scenario if that key event had played out differently.
Essay prompts (cont.) 4. Create a short story using one of the following topics: a. The end of MTVc. The Professor Disappeared b. Confessions of a Middle School Bullyd. The Mysterious Lab 5. Describe a moment in which you took a risk and achieved an unexpected goal. How did you persuade others to follow your lead? What lessons do you draw from this experience? You may reflect on examples from your academic, extracurricular or athletic experiences. 6. A high school curriculum does not always afford much intellectual freedom. Describe one of your unsatisfied intellectual passions. How might you apply this interest to serve the common good and make a difference in society? 7. Using an 8.5x11 inch sheet of paper, create an ad for a movie, design a house, make an object better, illustrate an ad for an object.
Creative Essay: “What if…” If the Trojans had heeded Laocoon’s advice and thrown Odysseus’ wooden horse into the sea, they would have defeated the Greeks at Troy. Aeneas would then never have had reason to flee the city, and he would never have ventured to Italy to found Rome. Without Rome, neither the Roman Republic nor a Roman Empire would have existed. Concrete, the arch, plumbing, and the sauna might never have been invented. The modern implications of Rome never having existed are indeed drastic. Lacking even concrete floors, people would resort to sleeping in the mud, and, without plumbing or saunas, they would be perpetually filthy and, generally, quite chilly. France could not have built the base of the Eiffel Tower without arches, so tourists would be unable to purchase miniature collectible Towers in Parisian convenience stores.
Good but Uncreative Essay: “What if…” What if the ratification of the nineteenth amendment did not pass and women were never given the right to vote? What would life for women, like me, be like in the United States? For one thing, I probably would not be writing this essay. If women were not given their right to vote, I probably would stop going to school after this year and it would be unlikely that I would receive a college education. Without suffrage, my career options would be limited, if a career were a possibility at all. My accepted practices would be limited to staying home and taking care of the family. Rather than being equals, women would be subservient to men. I might not drive, I might not dress in the way in which I choose to, and I might not be able to live my life the way that I can in the twenty- first century.
Kscope Pilot Study : Research Questions How do Kscope measures relate to other application information? –Academic, personal quality, extracurricular activity How do Kscope measure relate to admissions decision?
Data - Class of % of the applicants received at least one Kscope rating (Creative, Practical and Wisdom) K Group- combined C, P, W scores C = only low scores B = only medium scores A = one high score A+ = two or three high scores
Correlations between admission measures CreativePracticalWisdomK Group Practical0.41* Wisdom0.34*0.32* Academic0.17* * Personal0.18*0.22*0.21*0.25* Extracurricular0.45*0.44*0.41*0.49* SAT-V0.14* * SAT-M GPA
Post-hoc Prediction of Admission Decision Academics explain 28% of the variance in admission. Kscope explains an additional 6%. Kscope significantly improves prediction over Personal Quality score.
Pilot Study Data Kscope correlates with extracurricular and personal quality, but not with conventional academic measures. Kscope significantly improves post-hoc prediction of admission decisions over academic and personal quality measures. Interviewed admission officers and students supported the Kscope framework.
Pilot Study Data Number of applications rose Bottom third of old application pool greatly diminished; many more top applicants Average SATs rose slightly African-American applications up 25%, acceptances up 30% Hispanic-American applications and acceptances up 15%
Pilot Study Data There were no significant ethnic-group differences on Kaleidoscope Kaleidoscope correlated weakly with a composite academic rating (.11) Kaleidoscope correlated moderately with rated leadership/extracurricular activities (.44) First-semester GPAs did not differ significantly between Kaleidoscope A’s and academically comparable students admitted for other reasons
Pilot Study Data Greater customer satisfaction Message to students, parents, teachers, and counselors that Tufts is looking for more than just the high-SAT, high-GPA student
Larger Program at Tufts WICS –Admissions –Instruction and Course Assessment –Value Added –Leadership Minor
The Aurora Project Exploring the Entire Spectrum of Giftedness
Collaborators Hilary Chart Elena Grigorenko Linda Jarvin Bob Sternberg
Acknowledgment Work on the Aurora Project has been funded by a private donation from Karen Jensen.
Why Aurora? The Need for a Closer Look… Traditional overemphasis on analytical skills Under-representation of minority students in gifted programming 99.9% of special-education funding allocated to the lower end of the ability spectrum
FAMOUS “FAILURES” Thomas Edison *Of himself: “I have not failed. I’ve just found 10,000 ways that won’t work.” Albert Einstein *Got poor grades and failed his entrance exams to Zurich’s Polytechnic Institute Leonardo da Vinci *Lack of aptitude for learning languages, a horrible speller with poor grammar Winston Churchill *Twice failed the entrance exam to Sandhurst British military college
FAMOUS “FAILURES” Steven Spielberg *Dropped out of high school after being placed in a learning- disabled class Sydney Poitier *Dropped out of his elementary school Agatha Christie *Thought to have had a learning disability Walt Disney *Fired by a newspaper editor because he “had no good ideas”
Capturing ALL the Variables… MemoryAnalyticCreativePractical Images Reception Production Words Reception Production Numbers Reception Production *With SENSITIVITY and SPECIFICITY*
The Aurora Project, at a glance A standardized paper-and-pencil test, with analytical, creative, and practical components A parent interview for further qualification An observation schedule for use by clinicians
Analytical Skills using Words Homonym Fill-in-the-Blanks: Each sentence below is missing two words which sound the same but have different meanings, and sometimes different spellings too. Fill in the blanks with the same-sounding words that make the sentence make sense! 1. Go to the ___________ for some money and meet me by the river ___________. 2. I have a cut on my ___________ and will wear a bandage to help it ___________. 3. You may use an __________ to row across the lake, ___________ take a motorboat. 4. She __________________ heavily and rolled onto her ________________. Answers: bank/bank heel/healoar/or sighed/side
Analytical Skills using Numbers Thinking Math! 1.I am a three digit number. My tens digit is five more than my ones digit. My hundreds digit is eight less than my tens digit. What number am I? ___ ___ ___ Answer: Jordan is twice as old as Kaya. Kaya is three years older than Max. Max Is ten years younger than Jordan. How old is each of them? Jordan’s Age: Kaya’s Age: Max’s Age: Answer: 14, 7, 4
Analytical Skills using Images Geometry Boats: Pattern A and Pattern B show two ways to tie four boats together with four strings. Can you find three brand new ways to do this? Remember, just moving boats, or crossing or moving strings does not create a new pattern (all three examples to the right are identical to Pattern B, NOT new patterns). Answers: Pattern APattern B Identical to Pattern B:
Judging Creativity: Originality Complexity Emotional Expressiveness Descriptiveness Cleverness Humor Task Appropriateness (Rated Scales using Criteria set from Pilot Study Results)
Make room for Creativity! Cartoons:
Make room for Creativity! Sticker Task Examples:
Make room for Creativity! Math Stories and Mystery Paintings:
Creativity using Words Impossible Conversations: Imagine that the world has changed so that almost everything can speak. Write a little conversation describing what the two things listed might say to each other if they could talk. Each thing must say at least one thing. Before you write what they will say, circle who is speaking. Be creative! A Tree and the Tree-House that is in it TreeTree-House : TreeTree-House :
Creativity using Numbers Math Stories: Tell a short and interesting story about how the number 3 became the number 3/4ths. Be as creative as you can, but in your story you must include a blue piece of paper and an ice cream cone! Add a picture if you like! _________________________________ 3 --> 3/4ths
Creative Skills using Images Children’s Book Covers: Imagine that each of the following pictures is the cover of a children’s book. In a couple of sentences, tell what the story in each book might be about. Be as creative as you can be!
Making room for Practicality Concrete Problem Solving: Alex lost his cat two weeks ago and he is desperately trying to find her. Which of the following is the smartest thing to do? 1) Ask his parents if they have seen the cat. 2) Wait for the cat in the last place he has seen her. 3) Take out an ad asking for people that have seen her to call him. 4) Wait for the cat to come home on its own. Intuitive Comprehension: The teacher said, “We will go on the trip tomorrow only if the weather permits.” The teacher meant that the trip will take place only if the weather: 1) is perfect. 2) improves. 3) does not change. 4) is good for traveling. Tacit-Knowledge Examples:
Practicality using Words Applied Riddles: 1.You are standing in a room with three light switches that can be turned either on or off. In another room, which you cannot see, is a single light bulb. The light bulb is controlled by one of the switches in your room, and you must find out which one. You may flick the switches however you like, but you may leave the room and go to the other room only once. How will you find the right switch? 2.Laurie and Mariko want to go to the movies. The only way for them to get to the theater is to drive, and they want to take only one car, so Laurie must drive and pick up Mariko, or Mariko must drive and pick up Laurie. They are each at their own houses. Decide which person should drive if they want to get to the theater as quickly as possible and you know that: It takes more time to drive to the theater from Laurie’s house than it takes to drive to the theater from Mariko’s house. It takes longer to drive between the two houses than from Laurie’s house to the theater. Answer: Turn switch 1 on for a few minutes, then off. Turn switch 2 on, leave switch 3 off. If the light is on, it’s 2; if it’s off and cold, it’s 3; if it’s off and warm, it’s 1. Answer: Laurie should drive Mariko (longest route + shortest route).
Practicality using Numbers Applied Math: 1.Which is a better deal. In other words, if you had to buy a lot of chocolate for a party, would you spend less money buying lots of the first kind, or even more of the second kind? Circle the better deal: 12 ounces of chocolate for $3.605 ounces of chocolate for $ The ages of a father and son add up to 55. If you reverse the two digits in the father’s age, you have the age of the son. How old is each of them? Father’s Age:_______________Son’s Age:______________ Answer: 5 ounces for $1.25 because here each ounce is.25 (rather than.30). Answer: The father is 41, and the son is 14.
Practicality using Images Real-World Thinking: Below are pictures of a girl sitting in her airplane seat and holding a bottle Of grape juice. The first picture shows what she looks like when the plane Is still on the ground. In the second, the plane is either taking off (and going Up), or about to land (going down). Which is it? Circle the answer. Plane on the ground:Taking Off About to Land Answer: Taking Off
Social Problem Solving: Mother baked a cake for her children, Eric and Kathy. The children are looking for a way to split the cake between them, such that each receives the same amount of cake. Which of the following arrangements is most likely to result in a fair split of the cake between Eric and Kathy? 1) Eric cuts the cake, then Kathy chooses her piece. 2) Eric cuts the cake, then he chooses his piece. 3) A friend cuts the cake and Kathy chooses Eric’s piece. 4) Eric and Kathy eat whatever they want. Logical Comprehension: Aaron found a note from his mother saying: “You should not watch TV until you have eaten dinner, or it is after 8 PM.” It is 8:35 PM. Can Aaron watch TV now? 1) Yes, but only if he has eaten dinner. 2) Yes, he definitely can. 3) No, he definitely can not. 4) Yes, but only if he will have dinner later.
Capturing ALL the Variables… MemoryAnalyticCreativePractical Images Reception Production Words Reception Production Numbers Reception Production *With SENSITIVITY and SPECIFICITY*
Comments Aurora-g appears to act just like a conventional test of abilities Aurora-a appears to be different from a conventional test of abilities Practical “facet” of Aurora-a appears to be particularly relevant to school achievements
Comments Years of practice taught teachers to pay attention to conventional indicators of abilities It is important for teachers to consider and pay attention to “other” (non-conventional?) abilities For creativity to develop in classrooms teachers need to “see” it and know how to develop it
To Conclude Aurora-a/g are designed to be “laboring” in academic settings They are expected to relate to things that matter in school They are also expected, however, to provide new information NOT discernable either through conventional tests of abilities or through tests of achievement Aurora is looking good…. (cautious optimism)
Related Projects University of Michigan Business School Choate Rosemary Hall Advanced Placement –Statistics –Psychology –Physics
Conclusions Traditional abilities tests are narrow and limited Our new measures can –Broaden the range of skills tested for educational purposes –Increase predictive validity –Decrease ethnic-group differences –Increase customer satisfaction
Conclusions WICS can serve as a basis for an integrated program of transformation in university admissions, instruction, and assessment, as well as a sharpening or clarification of the purpose of university educatoin