1. KERA 1 © Regents of the University of California Verifying Learning and Advanced Skills and Knowledge Eva L. Baker National Center for Research on Evaluation, Standards, & Student Testing Graduate School of Education & Information Studies University of California, Los Angeles Korean Educational Research Association Educational Innovation for the 21 st Century: Sharing Visions and Experiences Hanyang University, Seoul, Korea 29 October 2010

2. KERA 2 © Regents of the University of California Presentation Overview How have we designed, used, and validated assessments in the past? Advanced knowledge and skills call for new methods 21 st century skills require deeper, more research-based analyses related to learning and learners The elastic relationships among practice, research and policy decisions, and research needs

3. KERA 3 © Regents of the University of California Before the Marriage of Learning and Assessment Assessment design Focused on stable findings rather than change Required many items sampled across a broad construct Use of item analyses, etc. to improve item performance Detect the best Uses and validity Reporting of position in normal distributions Technical quality indices were sensitive to numbers of items and wide ranging response Correlations and predictions Quotas and other limits on access

4. KERA 4 © Regents of the University of California Is it Time to Flower? http://www.google.com/imgres?imgurl=http://www.rubythroat.org/images/RoseOfSharon01.jpg&imgrefurl=http://www.lacitycollege.edu/ academic/departments/forlang/Kor/Kcourses.html&h=359&w=250&sz=15&tbnid=ioukMy9nXKdceM:&tbnh=121&tbnw=84&prev=/images %3Fq%3Dsouth%2Bkorean%2Bflower&zoom=1&q=south+korean+flower&hl=en&usg=__VJcjOdfCNSfjONMmSgPpY3LNbMs=&sa=X& ei=SV3ATN2yDY7CsAPku8DxCw&sqi=2&ved=0CCMQ9QEwAw

5. KERA 5 © Regents of the University of California Overview of Assessment of Advanced Skills and Knowledge Needs Goals and strategies Making education more accessible High expectations for all Emphasis on adaptation and flexibility Preparation for global environment Building in validity Different learners, motivations, constraints Emphasis on change Integration/application interpretive, not just “correct” Shared analyses, methods, and technical approaches— convergence and individuality

6. KERA 6 © Regents of the University of California Beyond “Alignment” Metaphor Standards-based systems imply lining up instruction provided and tests used to produce learning (ISD) Too limiting as rarely can standards be fully or completely sampled, almost never do they apply directly to future contexts Good teaching in 21 st century cannot be simply following of approved text and endless repetition to prepare for tests

7. KERA 7 © Regents of the University of California Towards Coherence and Transfer Paradox: to prepare for flexibility takes greater attention to components of learning rather than general constructs Biggest change: transfer must be taught and measured Harder to prepare for accountability exams Both horizontal (subjects) and vertical (grade level) integration

8. KERA 8 © Regents of the University of California What Are the First Steps? Clarifying and developing a reasonable number of standards Making sure the standards are valid—by assessing their quality, commonality of interpretation, and relationships to one another Logical and empirical processes—building the content ontology

9. KERA 9 © Regents of the University of California 1. Content Clarification by Ontology Graphic representation of nodes (what is to be learned) and relationships among them Not strictly a teaching sequence Verify common interpretations among teachers, educators, content experts, and policymakers Determine priorities and fundamentals Create a top-level meta-tag system for managing learning and performance database

10. KERA 10 © Regents of the University of California Common Core Standards Progression

11. KERA 11 © Regents of the University of California 2. Link to Cognitive Demands Cognitive demands are learnable/teachable processes that are embedded in subject matter Individuals differences may apply, but these are not traits They range between very specific to a domain, e.g., trouble shooting, somewhat general, develop a plan for persuasion; to more independent of specific domains, general writing skill

12. KERA 12 © Regents of the University of California 2. Cognitive Demands—21 st Century Skills Adapt problem solving approaches Transfer and apply skills to new situations Seek and manage new learning experiences Take risks in intellectual or entrepreneurial tasks Communicate in a variety of ways Expand skills for global work—teamwork, collaboration, language proficiency, cultural situation awareness

13. KERA 13 © Regents of the University of California Problem Solving Ontology Mayer, S. (in progress). Problem solving ontology (working document). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

14. KERA 14 © Regents of the University of California 3. Integrate Content and Cognition in Model Tasks Format may be of multiple types driven by outcome rather than convenience Support teacher and student initiative in creating similar tasks Assessments used for accountability and formative purposes Create reusable objects or templates for assessments (in online environments) with immediate automated scoring Monitor in database

15. KERA 15 © Regents of the University of California English Language Arts

16. KERA 16 © Regents of the University of California 4. Conduct Ongoing Validity Studies Systems evolve and validity is a continuing process Functional for purposes, accountability, improvement Sensitive to instruction not only to student background Fair to cultural/linguistic differences, disabled, gender Minimal gaming or negative side effects Include in database of individuals and items Timing

17. KERA 17 © Regents of the University of California 5. More Expansive Goals for 21 st Century Agility Intellectual and emotional resilience Persistence Societal and family values: dependability, ethics, caring, and joy

18. KERA 18 © Regents of the University of California Continuing Challenges Performance assessments are back, but need research on making them more comparable to one another—validity and fairness New psychometrics required if feedback to teachers and learners grows in importance— improvement guidance not rank order Costs can only be dealt with by serious use of simulations in design, administration, scoring

19. KERA 19 © Regents of the University of California Continuing Challenges (Cont’d) Teacher evaluation using students’ results very imperfect—but has numbers so may drive out more interpretive models Domains of tasks for configurations of standards need to be developed in advance, tried, and sampled. Problem is short time frame, school year, knowledge dependencies Need theory of situations (Resnick) to improve process

20. KERA 20 © Regents of the University of California Research Dreams Small sample, few—tasks models of validity Integrated databases from standards, through learning, through performance, mediated by individual background and recognition of many paths, with room to grow New theories of design Real investigations of variables, not only evaluation of entire interventions Finding ways to re-motivate students for intrinsic learning in this age of distraction, celebrity, and catastrophe

21. KERA 21 © Regents of the University of California Ready to Fly ??? http://en.wikipedia.org/wiki/File:Black-billed_Magpie.png

22. KERA 22 © Regents of the University of California http://www.cse.ucla.edu Eva L. Baker 310.206.1530 310.267.0152 baker@cse.ucla.edu voice fax email

23. KERA 23 © Regents of the University of California Backup

24. KERA 24 © Regents of the University of California Notes from the U.S. Emphasis on testing expanding with Race to the Top Competition funded two consortia to measure Common Core Standards—each with overlapping majorities of states Involve both formative and accountability assessment Pressure to link teacher evaluation to student work— controversial Rethinking role of commercial test producers No way to provide standard-by-standard reporting

25. KERA 25 © Regents of the University of California Speaking Truth to Power Not easy unless you are friends Policymakers in U.S. have odd mix of research-based ideas and preferences—now almost an overwhelming wave Desire conflicting goals without recognizing conflict