Presentation is loading. Please wait.

Presentation is loading. Please wait.

UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Assessment Potpourri—

Similar presentations


Presentation on theme: "UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Assessment Potpourri—"— Presentation transcript:

1 UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Assessment Potpourri— Accountability and NCLB; CRESST Update Eva L. Baker CRESST/UCLA Testing Seminar for Editors and Reporters Los Angeles, CA - August 10, 2006

2 2/34 Goals for Today Comprehensive picture of assessment for improving learning and accountability approaches General background on CRESST, federal initiatives Primer or review of elements of testing Hot assessment topics Illustrative CRESST work Children with disabilities Focus on California Interaction throughout Any question is OK

3 3/34 CRESST Center for Research on Evaluation, Standards, and Student Testing Nationally competed as institution: consecutive awards since Current mission: Research and development to improve practice in learning, assessment and educational policy Problem-focused, tool producing Assessment, testing, evaluation, learning, technology Set the national agenda and influence international practice New education research focus is 5-year study on formative assessment and questions about ELLs Learners range from 4-year-olds to military and postgraduate (medical training) High-end investment in simulation, training effectiveness of Navy and Marines  2006 Regents of the University of California

4 4/34 History Center for the Study of Evaluation (1966) Mission emphasized evaluation of programs and policies, evaluation theory, bilingual assessment, and measurement and statistical models CRESST (1985) partnership Mission began to emphasize learning-based measurement and assessment and accountability policies. Standards for performance assessment and accountability CRESST has won 6 consecutive 5-year awards to be the National Center  2006 Regents of the University of California

5 5/34 CRESST Center for Research on Evaluation, Standards, and Student Testing (Cont’d) Partners: Colorado, Harvard, USC, Maryland, Stanford, UCSB Support from U.S. Federal Government (NSF, IES, DoD), private foundations, international agencies, business partnerships, school districts or states, state colleges and universities, private companies Multidisciplinary staff including faculty, full-time PhD and MA staff, administrative support, and graduate and undergraduate students: computer science, electrical engineering, psychology, linguistics, learning, evaluation, measurement, sciences, math, humanities Partner with a variety of academic departments: engineering, computer science, chemistry, medicine, biology, classics, architecture, physics, sociology, political science, mathematics, and applied linguistics  2006 Regents of the University of California

6 6/34 CRESST Goals Conduct research and development on assessment, measurement and statistics, accountability (and other) policies, learning and instruction, and technology development and use Address formal and informal educational systems for learners of all ages Focus on needed instructional support to assure high-quality accomplishments (including transfer and generalization of learning) Create knowledge, models, tools, and support systems to transfer our findings into usable and intuitive forms Provide support to the public (and media) on questions about testing, evaluation, accountability, and learning  2006 Regents of the University of California

7 7/34 Talking About Tests Formats Frequency Designers Evidence of validity (test standards) Elements of test Level of content; depth of content Cognitive demands  2006 Regents of the University of California

8 8/34 CRESST Approach Summarize scientific knowledge about learning Find cognitive elements that can be adapted and reused in different topics, subjects and age levels. These elements make a “family” of models Embed model in subject matter like math or history Focus on “Big” content ideas to support learning and application Create templates, scoring schemes, training, and reporting systems (authoring systems available) Conduct research (we do) to assure technical quality and fairness  2006 Regents of the University of California

9 9/34 Model-Based Assessment Families of Cognition Content Understanding Problem Solving Teamwork and Collaboration MetacognitionCommunication Learning  2006 Regents of the University of California

10 10/34 CRESST Approach (Cont’d) Focus on “Big” content ideas to support learning and application Don’t start over every time. Create templates, scoring schemes, training, and reporting systems (authoring systems available) that can be reused Conduct or find research to assure technical quality and fairness  2006 Regents of the University of California

11 11/34 NCLB Background Educational reform—part of a continuing international competition U.S. is alone in its interpretation of standards-based reform Emphasizes secure tests Emphasizes improvement State control Lack of transparency Other countries have curriculum goals or standards, common syllabi (Australia), and assessments that may be released annually Impact on schools and individuals Lack of syllabus serious  2006 Regents of the University of California

12 12/34 NCLB Basics By 2014 all children will be proficient Annual targets (% of students) (AYP) Disaggregation (separation into summaries by subgroup) for results and % taking test ELL issues Sanctions Questions about quality and comparability of standards, assessments and targets among States Evidence-based instruction? Quality teachers Math, Language Arts, Science GAO estimates $1.9 billion a year for testing  2006 Regents of the University of California

13 13/34 State of Testing: NCLB External, varying standards and tests from States Range of targets (AYP) Short timeline to serious sanctions Raised scores only clear evidence of learning Few incentives to measure “high standards” No incentives to create assessments that respond to quality instruction Growing enthusiasm for use of classroom assessment for accountability, but few scalable methods Benchmark tests (test practice) Need for new ways to think about the relationship of accountability, long-term learning and performance  2006 Regents of the University of California

14 14/34 Assessment Purposes Needs sensing System monitoring Evaluation Improvement Accountability Selection/placement Opt out Diagnosis Formative/progress Achievement Certification/career Skill retention Transfer of learning  2006 Regents of the University of California System or Program Individual/Team

15 15/34 Changes in Assessment Policy and Practices From: One purpose, one measure To: Multiple purposes—one measure Difficult to retrofit measure designed for one purpose to serve another New assessment design strategy  2006 Regents of the University of California

16 16/34 NCLB Testing and Assessment Issues Purposes Quality Fairness Utility Underlying reality  2006 Regents of the University of California

17 17/34 Quality Quality has many parts, parts that confuse educators, the policy community and the public Main Quality Issue Do the results provide a measure of the construct or domain (that is, whatever you want to measure)? How do you know? U.S. answer is “alignment” Are the inferences drawn rated for purpose(s) and fair? How do you know?  2006 Regents of the University of California

18 18/34 Quality (Cont’d) Construct Validity Reliability Fairness Inferences about progress Inferences about real learning Are measures sensitive to instruction?  2006 Regents of the University of California

19 19/34 Fairness Fairness as equal understanding of test/task Fairness as equal access to preparation Fairness as representation of the construct Fairness as equal chance to perform (accommodations) Fairness in interpretation and targeting  2006 Regents of the University of California

20 20/34 Utility Do the results provide the information needed for decisions and action? Granularity Knowledge context—teachers, student prior Options Time Individual and group needs  2006 Regents of the University of California

21 21/34 Reality Accountability Alignment of standards (goals), tests, and expectations by grade Sampling of test content and cognitive demands Measures that include more than they should Judgment that the test score = learning Transfer? Push-back New methods: value-added, benchmark, formative assessment Old solutions: performance assessment/multiple choice Coming attractions: serious simulation-based assessment  2006 Regents of the University of California

22 22/34 Useful Questions Who made the test and how long did it take? What is the evidence supporting the quality of the test? For different groups? Evidence that instruction (not test prep) accounted for growth? Was growth sustained? Is the school meeting its targets? What is meant by closing the “gap”? What’s the relationship between drop out and test performance?  2006 Regents of the University of California

23 23/34 Caveats Tests are different Therefore, scores have different meaning No single score should be used to make a decision (Test Standards, 1999) Tests focus attention on some topics and some students Scores appear to give a complex system a bottom line, but which kinds matter?  2006 Regents of the University of California

24 24/34 Back Up

25 25/34 Some CRESST Illustrative Activities New system for addressing mathematics learning, teaching and assessment, blending formative assessment, professional development, job performance aids, and computer support Developing a semi-automated smart assessment development system Quality School Portfolio—K-12 plus university systems for individual and departmental longitudinal databases, graphical interface Validity studies and interventions for English-language measures  2006 Regents of the University of California

26 26/34 Some Illustrative Activities (Cont’d) Designed and conducted assessment of Los Angeles Unified School District individual student performance for multiple years (trained teachers, about 400K children a year) Developed collaboratively automated assessment system for 4-year-olds Partner in NSF Teaching and Learning Center in assessment, focused on tool building for science assessments Conducted validity studies of new measures of knowledge, procedures, and problem solving for international community Conducted workshop on medical simulation and metrics  2006 Regents of the University of California

27 27/34 R&D Opportunities: Environments Simulations Embedded PC games VR technologies Automated design and scoring systems Reporting and profiling Integrated psychomotor, affect and cognition measures for improvement  2006 Regents of the University of California

28 28/34 Emerging Emphases Complex self-assessments High-end knowledge transfer Generalization Unobtrusive measurement Organizational learning to predict school success  2006 Regents of the University of California

29 29/34 The Iowa Tests in Louisiana  2006 Regents of the University of California

30 30/34 Extended Analysis in High School History LILIUOKALANI For many years our sovereigns had welcomed the advice of American residents who had established industries on the Islands. As they became wealthy, their greed and their love of power increased. Although settled among us, and drawing their wealth from resources, they were alien to us in their customs and ideas, and desired above all things to secure their own personal benefit. Kalakaua valued the commercial and industrial prosperity of his kingdom highly. He sought honestly to secure it for every class of people, alien or native. Kalakaua’s highest desire was to be a true sovereign, the chief servant of a happy, prosperous, and progressive people. And now, without any provocation on the part of the king, having matured their plans in secret, the men of foreign birth rose one day en masse, called a public meeting, and forced the king to sign a constitution of their own preparation, a document which deprived [him] of all power and practically took away the franchise from the Hawaiian race. It may be asked, “Why did the king give them his signature?” I answer without hesitation, because he had discovered traitors among his most trusted friends and because the conspirators were ripe for revolution, and had taken measures to have him assassinated if he refused. It has been known ever since that day as “The Bayonet Constitution,” and the name is well- chosen; for the cruel treatment received by the king from the military companies. [text continues] *From Hawaii’s Story by Hawaii’s Queen, Liliuokalani (Boston: Lee and Shepard Publishers, 1898). Explain to your friend who missed class the reasons and differences for the Queen’s and the Senator’s approaches to Hawaii’s future. Scoring Rubric General impression (on task) Principles and themes Prior knowledge Relevant concrete examples Avoidance of misconceptions  2006 Regents of the University of California

31 31/34 Reading and Literature Sample Test The Baobab Tree: The Baobab is Africa’s most fascinating and unusual tree. It is usually leafless, making it look as though its roots, not its branches, are stretching high into the air. For the Dogon people of Mali, it is an essential part of their lives. Every part of the tree is of use to them. Found in the hot, dry areas of Africa know as the savannas, the baobab lives up to a thousand years and grows to a height of 60 feet. The trunk is often 50 feet around. If you can imagine a group of 12 eight-year-olds holding hands in a circle, that is roughly the size of the trunk. Few other trees are as massive or have such a long life. The Dogon people live at the base of the 125-mile-long Bandiagara Cliffs, where baobabs thrive in the rocky ground. The Dogon strip the bark off the trees in yard-long sections and weave the fibers into rope. Tribes in other areas also make paper and cloth from the bark. No matter how much bark is stripped, the baobab is able to heal itself and keep growing. Twice a year, when the rains come, a sparse covering of leaves briefly appears. The leaves are used as medicine and for flavoring soup. At twilight, huge purple-and-white flowers open, hanging down on thick stems. The next morning, the blossoms fall to the ground, and large fruits, or pods, eventually take their place. The hard, oval fruits, called monkey bread, are almost a foot long and hang like lanterns from the trees. They are filled with seeds that grow in a cotton candy-like pulp. The pulp is eaten or used to flavor cool drinks. Unfortunately, it tastes more like cotton balls than cotton candy! Now that tourists occasionally visit the remote Dogon country, another use has been discovered for the baobab fruits. When the dark brown fruit hardens and the pulp inside dries, the pod is made into a rattle. Circular designs are often carved in the pod, exposing the white inner flesh. Many believe that there are no young baobab trees. For the first 40 years, the baobab is shaped like an evergreen tree, its thick branches growing straight out and widening at the base. It looks nothing like the older tree. Although these great trees live to a ripe old age, eventually, they begin to crumble, the fragments loosening and blowing away until nothing is left. Sometimes, the huge base is cut open and used as a cistern, a receptacle for storing water. Regardless of age, the baobab is precious to the Dogon people. Without it, their lives would be very different.  2006 Regents of the University of California

32 32/34 Reading and Literature Sample Test (Cont’d) 1. The baobab is an essential part of the Dogon people’s lives. The word essential most likely means A. Exciting B. Strange C. Unusual D. Necessary 2. Why does the author want the reader to imagine a group of eight-year-olds holding hands in a circle? A. To help young children learn how to read B. To help young readers learn the parts of the tree C. To help the reader understand the fictional story D. To help the reader understand the size of the trunk 3. Why do the Dogon people make rattles from the pods of the fruit? A. To heal wounds B. To sell as souvenirs C. To meet basic needs D. To flavor cold drinks  2006 Regents of the University of California

33 33/34 Science Sample Test 1. Five minutes into the race, what happens in the students’ circulatory system? A. The heart pumps more blood to the legs. B. The muscles start to tighten. C. Fluids pour into the stomach. D. The lungs need more air. 2. Ten minutes into the race, the students are perspiring or sweating. Which human body system is now involved? A. Nervous system B. Respiratory system C. Skeletal system D. Excretory system 3. One of the students falls and sprains an ankle. A part of which system is now injured? A. Circulatory system B. Respiratory system C. Skeletal system D. Nervous system A Road Race: Some students are taking part in a community running race.  2006 Regents of the University of California

34 34/34 next presentation Eva L. Baker voice fax ©2006 Regents of the University of California


Download ppt "UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Assessment Potpourri—"

Similar presentations


Ads by Google