1. Copyright CCR Consulting Group © 2010. All rights reserved. From Unidimensional to Multidimensional Cognitive Indicators of College and Career Readiness: Where Are We in the Journey? David T Conley, PhD University of Oregon Educational Policy Improvement Center

2. Copyright CCR Consulting Group © 2010. All rights reserved. Why Do We Need a New Conception of College and Career Readiness? Historical model is aptitude based – College readiness was conceived of as unidimensional, consisting of an unspecified amalgam of content knowledge and “critical thinking.” – Goal was determining general eligibility more than readiness. – Ignored was the inherent complexity of content knowledge and cognitive strategies and skills. – Also overlooked entirely were a host of other factors because they could not be conveniently measured with existing technologies. Net result is a model of readiness that is overly narrow and not particularly actionable – It is difficult to know what to do to be college ready beyond “take challenging classes.” – HS grades are wildly inflated; admissions tests are supposedly immune to test-prep. – Tremendous bias exists in favor of individuals and schools with access to the “privileged knowledge” of what it really takes to be college ready. 2

3. Copyright CCR Consulting Group © 2010. All rights reserved. The Equivocal Role of Prior Content Knowledge High school preparation focuses primarily on content knowledge acquisition. Many entry-level college courses assume little or no prior knowledge in the subject area. Even students with well-developed content knowledge bases struggle in entry-level college courses. College courses begin to expect students to apply content knowledge in non-routine ways. Many college courses require skills that are not tested on traditional placement tests or well developed in high school. College students can effectively avoid whole content areas throughout college. 3

4. Copyright CCR Consulting Group © 2010. All rights reserved. What Do We Need from a More Comprehensive Model of College and Career Readiness? Must be multidimensional. Must be actionable. Must be transparent. Must be valid. Must be more class and race neutral. Must result in (many) more students being well prepared for postsecondary opportunities. 4

5. Copyright CCR Consulting Group © 2010. All rights reserved. The Four Dimensions of College Readiness Key Cognitive Strategies Problem formulation, research, interpretation, communication, precision and accuracy. Key Content Knowledge Key foundational content and “big ideas” from core subjects. Academic Behaviors Self-management skills: time management, study skills, goal setting, self-awareness, and persistence. College Knowledge Admissions requirements, college types and missions, affording college, college culture, and relations with professors. 5

6. Copyright CCR Consulting Group © 2010. All rights reserved. Tracking Readiness Along Four Dimensions for an Individual Student 6 This Line shows an optimal trajectory This line shows the student’s growth on key cognitive strategies, which levels off in high school This line shows academic behaviors, which dip precipitously in 10 th grade, but show a comeback by 12 th grade This line shows student college knowledge strengthens late in high school but remains below optimal This line shows the student’s content knowledge, which is stronger than the optimal This profile describes a student whose content knowledge is strong, but for whom some concerns remain: cognitive strategies are not developing, college knowledge is below optimal and academic behaviors are somewhat erratic. This profile could be used to diagnose and prescribe in high school or to link the student with support services in college. 6

7. Copyright CCR Consulting Group © 2010. All rights reserved. What Is Key Content Knowledge? Components: – Key terms and terminology – Factual information – Linking ideas – Organizing concepts The brain retains this type of information to the degree to which it can: – generate connections or links among the pieces to create an internal structure for the information – associate emotions, positive or negative, with the information – find the information meaningful, relevant, or useful – apply or use the information in a variety of authentic situations – receive timely feedback on how useful the information was to achieve a specific purpose or general goal. 7

8. Copyright CCR Consulting Group © 2010. All rights reserved. What Are Key Cognitive Strategies? What is a strategy? – A systematic approach or plan to achieving an objective – An elaborate plan of action that chooses among alternative approaches and anticipates potential problems that must be addressed for the objective to be achieved successfully. What are cognitive strategies? – Systematic approaches to achieve key learning goals that take into account the rules and methods of the academic disciplines that are necessary to achieving the goal – Elaborate plans of action that choose among alternative approaches and anticipate potential problems that must be addressed to solve a problem or complete a complex task. 8

9. Copyright CCR Consulting Group © 2010. All rights reserved. Key Cognitive Strategies for Postsecondary Readiness and Success 9

10. Copyright CCR Consulting Group © 2010. All rights reserved. 10 Adults’ Use of Cognitive Strategies Have you ever purchased anything at Ikea? Was “some assembly required”? Consider the strategies you typically use to assemble such merchandise and their effectiveness.

11. Copyright CCR Consulting Group © 2010. All rights reserved. How Novices and Experts Solve Problems are slower and more deliberate know individual facts about topics learn about pieces of systems memorize bits of information and encode the bits superficially tend to focus on discrete knowledge in isolation of the structure of a discipline reason in specific contexts by using recently-acquired information recall information by rote are faster and more accurate organize facts into “chunks” for better recall and application integrate pieces of knowledge into systems frameworks connect new knowledge to existing knowledge learn through example and analogy use analytical skills to apply knowledge and select procedures generalize knowledge to new settings and circumstances create mental cues to facilitate recall 11 Novices:Experts:

12. Copyright CCR Consulting Group © 2010. All rights reserved. The Novice-Expert Framework Based on recent results from brain and cognitive sciences – Brain organizes information into databases, which can be retained through a variety of means – Meaning is enhanced and retention strengthened when the information is used for non-routine tasks and activities Represents a vertical scale built off of a learning or developmental progression – Strategies are developed in the context of the content area, consistent with cognitive science – Repeated practice with a variety of challenging tasks or materials is critical 12

13. Copyright CCR Consulting Group © 2010. All rights reserved. 13 Novice-to-Expert Scale Grades 6-12 Green Green indicates the student is on or above the pathway to college readiness. Gray Gray indicates performance levels not scored at that benchmark level. Blue Blue indicates the student is approaching the pathway to college readiness. Benchmark Level I (~6 th grade) Benchmark Level II (~8 th grade) Benchmark Level III (~10 th grade) Benchmark Level College Ready Emerging Expert Emerging Expert Emerging Expert Emerging Expert Accomplished Strategic Thinker Strategic Thinker Emerging Strategic Thinker Accomplished Novice Novice Emerging Novice Emerging Novice Emerging Novice Emerging Novice

14. Copyright CCR Consulting Group © 2010. All rights reserved. Example of a Cognitively Challenging Task that Requires the Use of Strategies Calculate Incarceration Rates in the United States Recent news has given much coverage to the growing prison population in the United States. Criminal justice organizations need to know the number of people who will be under correctional supervision in the future in order to plan ahead. The numbers will affect their budgets and the need for prison beds and support services and will have implications for society more broadly as well. You have been hired to study the situation and make projections about future incarceration rates. This includes the numbers of people in prisons, jails, and on parole and probation. You will prepare a report that presents your projections and explains how you developed them. Two specifics you have been asked to address in your report are: – Projections of the affected population by 2020, 2025, 2030, and 2050. – A prediction whether, and if so when, the rate of incarceration can be expected to exceed 10 percent of the overall US population. 14

15. Copyright CCR Consulting Group © 2010. All rights reserved. Examples of Decisions Associated with Formulating a Hypothesis HypothesizingEliminate obviously wrong, impractical, or unrealistic hypotheses. Understand the intent of the problem. Consider mathematical constraints. Represent the problem accurately. Define “incarceration.” Define range of years. Identify assumptions. Anticipate complexity of the problem. Consider factors that affect the potential solution. Identify the full nature of the problem: Is it purely mathematical, or might social constraints affect a plausible solution? Be aware that purely quantitative solutions to a social issue or problem have limitations and qualifications. 15

16. Copyright CCR Consulting Group © 2010. All rights reserved. Examples of Decisions Associated with Formulating a Strategy Strategizing Consider appropriateness and efficiency of strategy. Gauge availability of data sources and feasibility of accessing them. Consider mathematical constraints. Consider prior experience with similar problems. Identify previous effective and ineffective strategies. Recognize that problem like this involve functions and graphing. Consider some key principles or analytic issues. Understand that the problem involves plotting the interaction of two variables: population over time. Recognize that the problem likely requires the use of a spreadsheet or graphing calculator at the least. Recognize that there is no one right answer and that a good answer is one that can be defended, and anticipate the need to explain or defend all choices made. 16

17. Copyright CCR Consulting Group © 2010. All rights reserved. College and Career Readiness: Same or Different? EPIC’s research on the content of courses in two-year certificate programs is identifying the overlap between college readiness and career readiness knowledge and skills. Necessary academic content knowledge varies among two- year certificate programs and between certificate programs and general education courses at four-year institutions. However, almost all degree and certificate programs require a strong academic foundation, and students in certificate programs need strong skills in the Academic Behaviors dimension of readiness. 17

18. Copyright CCR Consulting Group © 2010. All rights reserved. Welding Technology Certificate (A.S.): Sample Student Learning Outcomes (from Los Rios Community College) Select the correct electrode classification and parameters for various thickness of material and welding positions on ferrous and nonferrous metals. Define principles of gas metal arc welding. Interpret GMA electrode and classification and specification. Select correct electrode amperage settings for the job application. Interpret graphic welding symbols. Describe shielded metal arc welding operations of various positions using selected electrodes on different joint designs. Explain the reason for the formation of each discontinuity type and distinguish different discontinuities. Interpret fabrication blueprints using a systematic process. Relate the requirements for welding ferrous and nonferrous metals. 18

19. Copyright CCR Consulting Group © 2010. All rights reserved. Highly Aligned Cross-Disciplinary Standards (Top five Performance Expectations) 19

20. Copyright CCR Consulting Group © 2010. All rights reserved. Alignment Analysis Results To what degree are the CCRS cross-disciplinary skills aligned with what is necessary to be prepared to succeed in entry-level CTE college courses in Texas? 20

21. Copyright CCR Consulting Group © 2010. All rights reserved. Comparison of 3 CCRS Validity Studies Results 21

22. Copyright CCR Consulting Group © 2010. All rights reserved. Texas College and Career Readiness Standards Aligned with a Career Pathway To what degree are the Texas CCRS aligned with what is both necessary to succeed in or taught in Nursing or IT Programming pathway courses? 22

23. Copyright CCR Consulting Group © 2010. All rights reserved. Interaction Between Cognition and Behavior Cognition exists in a behavioral context. Thinking is more likely to occur when students and teachers establish a mutually favorable context. Thinking strategies have (or can have) behaviors associated with them. Strategic thinking and strategic behaviors are complementary notions. – Students succeed to the degree to which they marshal a suite of thinking strategies and learning behaviors specific to the knowledge or skill in question. 23

24. Copyright CCR Consulting Group © 2010. All rights reserved. Effort as a Form of Aptitude US students tend to believe aptitude trumps effort. Effort is more than trying hard. – It is strategic and adaptive in nature and includes an element of persistence. – Doing the same thing and getting the same wrong answer six times is not effort or persistence. Thoughtful errors are potentially more instructive than all correct answers. For most people, effort will be far more important than aptitude for academic and career success. – Ability to make strategic adaptations based on experience 24

25. Copyright CCR Consulting Group © 2010. All rights reserved. Where Are We in the Journey? Very little acceptance in practice of the multidimensional nature of college and career readiness. – Ironically, high school grades are a universally accepted multidimensional measure, but few argue they meet standards of psychometric rigor. The consortia common assessment proposals begin to tip-toe into this arena, but they are very limited in what they are proposing beyond content knowledge measurement. Much experimentation ongoing to measure more dimensions of college and career readiness. 25

26. Copyright CCR Consulting Group © 2010. All rights reserved. Closing Comments The reality is that college and career readiness is truly multidimensional regardless of how we choose to measure it. Restricting the information we are willing to use to determine college and career readiness only distorts the behaviors of students and teachers so that they pay less attention to other key dimensions of readiness. 21 st Century measurement systems need to be profile-like in nature and provide more actionable information to students and schools, not just a cut score to a college. 26