C R E S S T / U C L A Alicia M. Cheak, Gregory K. W. K. Chung, Eva L. Baker, Cecile H. Phan, and Linda F. de Vries American Educational Research Association,

Slides:

Advertisements

Similar presentations

Ed-D 420 Inclusion of Exceptional Learners. CAT time Learner-Centered - Learner-centered techniques focus on strategies and approaches to improve learning.

Advertisements

Project VIABLE: Behavioral Specificity and Wording Impact on DBR Accuracy Teresa J. LeBel 1, Amy M. Briesch 1, Stephen P. Kilgus 1, T. Chris Riley-Tillman.

Listening. Learning. Leading. Using Factor Analysis to Investigate the Impact of Accommodations on the Scores of Students with Disabilities on English-Language.

Test Development.

Test Item Analysis Science Grade 6 Josh Doty John Sevier Middle School Kingsport, TN Grade Level: 6 Content Area: Earth Science Source: 2007 Tennessee.

ASSESSING RESPONSIVENESS OF HEALTH MEASUREMENTS. Link validity & reliability testing to purpose of the measure Some examples: In a diagnostic instrument,

1 H2 Cost Driver Map and Analysi s Table of Contents Cost Driver Map and Analysis 1. Context 2. Cost Driver Map 3. Cost Driver Analysis Appendix A - Replica.

Dental Treatment Needs of Active Duty Military Personnel 1994 and 2003 Andrew K. York, CAPT, DC, USN Susan W. Mongeau, Lt Col, USAF, DC David L. Moss,

Iowa Assessment Update School Administrators of Iowa November 2013 Catherine Welch Iowa Testing Programs.

VALIDITY AND RELIABILITY

Chapter 5 Measurement, Reliability and Validity.

General Information --- What is the purpose of the test? For what population is the designed? Is this population relevant to the people who will take your.

Division of Student Assessment IAA Test Administration Illinois State Board of Education 100 North First Street, E-216 Springfield, Illinois www.isbe.net/assessment.

Chapter 17 Overview of Multivariate Analysis Methods

Multiple Criteria for Evaluating Land Cover Classification Algorithms Summary of a paper by R.S. DeFries and Jonathan Cheung-Wai Chan April, 2000 Remote.

C R E S S T / U C L A Issues and problems in classification of students with limited English proficiency Jamal Abedi UCLA Graduate School of Education.

National Center on Educational Outcomes N C E O Strategies and Tools for Teaching English Language Learners with Disabilities April 9, 2005 Kristi Liu.

CRESST / U C L A Slide 1, Implementing No Child Left Behind: Assessment Issues Joan L. Herman UCLA Graduate School of Education & Information Studies.

Wrapping Up PBL Problems Hal White Dept. of Chemistry and Biochemistry Workshop Wednesday June 28, 2006 about Developed by with who uses Presented on emphasizing.

Geog 458: Map Sources and Errors Uncertainty January 23, 2006.

SOWK 6003 Social Work Research Week 10 Quantitative Data Analysis

Data Mining: A Closer Look Chapter Data Mining Strategies (p35) Moh!

C R E S S T / U C L A Improving the Validity of Measures by Focusing on Learning Eva L. Baker CRESST National Conference: Research Goes to School Los Angeles,

C R E S S T / U C L A Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing (CRESST) New Models.

RATING THE PROFESSOR TA & PFF Workshop– Spring 2011 Artze-Vega.

C R E S S T / U C L A Evaluating the Impact of the Interactive Multimedia Exercises (IMMEX) Program: Measuring the Impact of Problem-Solving Assessment.

Cognitive Interviewing for Question Evaluation Kristen Miller, Ph.D. National Center for Health Statistics

PUBLIC SCHOOLS OF NORTH CAROLINA STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 1 Review of the ABCs Standards SBE Issues Session March 2, 2005.

The Hierarchy of Learning Adapted from Benjamin Bloom’s Taxonomy of Educational Objectives.

Application of Generalizability Theory to Concept-Map Assessment Research Yue Yin & Richard J. Shavelson Stanford Educational Assessment Laboratory (SEAL)

Poetry Assessment Analysis & Critique Krissa Loretto EDUC 340 Spring 2014.

LECTURE 06B BEGINS HERE THIS IS WHERE MATERIAL FOR EXAM 3 BEGINS.

Leslie T. Allen Oral Defense. Introduction Family Andy and I have been married for 3 years! Our son, Ethan, is 15 months old and the joy of our lives!

Naglieri Nonverbal Ability Test (NNAT) Miami-Dade County Public Schools NNAT Workshop March 26, 28, & 29, 2007.

Forum - 1 Assessments for Learning: A Briefing on Performance-Based Assessments Eva L. Baker Director National Center for Research on Evaluation, Standards,

© Regents of University of California 1 Functional Validity: Extending the Utility of State Assessments Eva L. Baker, Li Cai, Kilchan Choi, Ayesha Madni.

Page 1 Ming Ji Department of Computer Science University of Illinois at Urbana-Champaign.

LAS LINKS DATA ANALYSIS. Objectives 1.Analyze the 4 sub-tests in order to understand which academic skills are being tested. 2.Use sample tests to practice.

Andrew J. Mason 1, Brita Nellermoe 1,2 1 Physics Education Research and Development University of Minnesota, Twin Cities, Minneapolis, MN 2 University.

Semantic Map Assessment Project Overview for McREL Dr. Roy B. Clariana Penn State, Great Valley 12/18/02.

Employing Empirical Data in Judgmental Processes Wayne J. Camara National Conference on Student Assessment, San Diego, CA June 23, 2015.

United Nations Statistics Division Registry of national Classifications.

XP New Perspectives on The Internet, Sixth Edition— Comprehensive Tutorial 3 1 Searching the Web Using Search Engines and Directories Effectively Tutorial.

Reliability vs. Validity.  Reliability  the consistency of your measurement, or the degree to which an instrument measures the same way each time it.

CRESST ONR/NETC Meetings, July 2003, v1 ONR Advanced Distributed Learning Impact of Language Factors on the Reliability and Validity of Assessment.

ONR/NSF Technology Assessment of Web-Based Learning, v3 © Regents of the University of California 6 February 2003 ONR/NSF Technology Assessment of Web-Based.

Discriminant Analysis Discriminant analysis is a technique for analyzing data when the criterion or dependent variable is categorical and the predictor.

C R E S S T / U C L A UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation,

1/27 CRESST/UCLA DIAGNOSTIC/PRESCRIPTIVE USES OF COMPUTER- BASED ASSESSMENT OF PROBLEM SOLVING San-hui Sabrina Chuang CRESST Conference 2007 UCLA Graduate.

California Educational Research Association Annual Meeting CERA Session 3 Day 1 – Preparation and Professional Development Rancho Mirage, CA – December.

35th Annual National Conference on Large-Scale Assessment June 18, 2005 How to compare NAEP and State Assessment Results NAEP State Analysis Project Don.

CHAPTER OVERVIEW The Measurement Process Levels of Measurement Reliability and Validity: Why They Are Very, Very Important A Conceptual Definition of Reliability.

Statistical Literacy: Statistics Canada Initiatives DLI National Workshop May 14, 2007 Montreal.

Concerns, Data, Next Steps.  New Administration Software from Scantron  New Academic Senate Policy  New Items/Survey Form (ACE, Iowa Item Pool)  New.

Teaching to the “Big Ideas”: Moving beyond the standards Terry P. Vendlinski UCLA Graduate School of Education & Information Studies National Center for.

Fair and Appropriate Grading

CRESST ONR/NETC Meetings, July 2003, v1 17 July, 2003 ONR Advanced Distributed Learning Greg Chung Bill Bewley UCLA/CRESST Ontologies and Bayesian.

CRESST ONR/NETC Meetings, July July, 2003 ONR Advanced Distributed Learning Greg ChungAllen Munro Bill BewleyQuentin Pizzini Girlie DelacruzUSC/BTL.

1 Science, Learning, and Assessment: (Eats, Shoots, and Leaves) Choices for Comprehensive Assessment Design Eva L. Baker UCLA Graduate School of Education.

UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing V4, 1/18/07 Research.

[School Name]’s Student Perception Survey Results This presentation is a template and should be customized to reflect the needs and context of your school.

© 2009 Pearson Prentice Hall, Salkind. Chapter 5 Measurement, Reliability and Validity.

Harry O'Neil University of Southern California and The Center for Research on Evaluation, Standards, and Student Testing A Theoretical Basis for Assessment.

Presented to West Manor Administration and Faculty By: Felicia Belcher April 16, 2014.

SHOW-ME TASK FORCE ON ACCREDITATION AND ASSESSMENT REPORT MISSOURI ASSOCIATION OF SCHOOL ADMINISTRATORS June 14, 2016.

Product Definition Chapter 4 Paul King.

Classroom Assessment Validity And Bias in Assessment.

Week 3 Class Discussion.

Week 10 Slides.

Word Map Word Maps are DUE on Friday.

Presentation transcript:

C R E S S T / U C L A Alicia M. Cheak, Gregory K. W. K. Chung, Eva L. Baker, Cecile H. Phan, and Linda F. de Vries American Educational Research Association, New Orleans, LA April 2002 UCLA Graduate School of Education & Information Studies Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing An Investigation of the Feasibility of an Automated Approach to Classifying Open-Ended Concept Map Links

C R E S S T / U C L A Overview of Presentation è Concept map definition è Past research è Current research study n Purpose of study, method, and results

C R E S S T / U C L A Concept Maps è Node-Link representation (nodes = concepts, links = relationships) è Typically used for instructional purposes è Sometimes used for assessment è Automated scoring possible with fixed set of concepts and links sunlight surface warming leads to

C R E S S T / U C L A Concept Map Sample

C R E S S T / U C L A Previous CRESST Work è Use concept mapping for assessment purposes n Predefined terms and links allow for automated scoring n Expert maps used as scoring criterion n Tested with 4th-graders to adult learners n Map scores correlate with essay measures of content understanding (r ~.70)

C R E S S T / U C L A Purpose of Our Study è Investigate feasibility of an automated approach to classifying open-ended links n Identify “root” form of links used in open-ended concept maps in two content areas n Develop software to convert links into root forms n Examine quality of conversion n Test approach with existing map data è Long-term goal is automated scoring of open- ended links

C R E S S T / U C L A Method è Analyze distribution of links in two content areas n Highly skewed distribution would be a favorable situation for an automated classification system è Approach n Raters convert links to root form  Example: is a, is an, is one of, are => is a  Example: are part of, are (is) a part of, is a part of the, is part of, is part of the, part of, part of process => part of n Identify most common root form links across two content areas (physiology and marine mammals)

C R E S S T / U C L A Method è Automated approach n Use part-of-speech (POS) tagger and custom-developed software to convert links to root form n Examine quality of conversion  With respect to student-generated links  With respect to rater-converted links n Test overall system with existing map data

C R E S S T / U C L A Results: Distribution of Links è Very skewed distribution of links è Marine Mammals: n 898 links converted to 134 root form links n Most used: has, is, part of, eats (43%); top 10 links account for 58% è Physiology: n 464 links converted to 92 root form links n Most used: part of, is, causes, has (36%); top 10 links account for 60% è Common links n Has, is, part of

C R E S S T / U C L A Results: Quality of Automated Conversion vs. Student-Generated Links

C R E S S T / U C L A Results: Quality of Automated vs. Rater-Converted Links

C R E S S T / U C L A Test of Automated Approach Propositions (57) Exact match (28) No exact match (29) Synonyms Match (8)No match (21) Correct (12)Incorrect (5)Correct (4)Incorrect (3) Could not convert (7) Correct (28) ? (1)? (4)

C R E S S T / U C L A Summary è Limitations n Small sample of links, only 2 content areas, age of sample (4th-graders) n Analysis of mismatches indicates insensitivity to prepositions and other integral parts of speech è Skewed distribution of link types favorable for automated processing è Automated conversion yields links similar to student-generated and rater-converted links è Test of automated approach promising

C R E S S T / U C L A For More Information è Greg Chung è è Web:

C R E S S T / U C L A Categories of Link Types

C R E S S T / U C L A Distribution of Links