Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two.

Slides:



Advertisements
Similar presentations
Chapter 2 The Process of Experimentation
Advertisements

Elaine Gerber & AJ Kelton, Montclair State University & Tracy Chu,
Cross Cultural Research
1 COMM 301: Empirical Research in Communication Kwan M Lee Lect4_1.
Reliability and Validity
The Basics of Experimentation I: Variables and Control
Effect Size and Meta-Analysis
Prevalence of Out-of-Level Testing Martha Thurlow and Jane Minnema National Center on Educational Outcomes
MSc Applied Psychology PYM403 Research Methods Validity and Reliability in Research.
2008Anton McLachlan Workshop on Publishing Scientific Papers Constructing a Paper The final step in a research project. We all stand on the shoulders of.
Statistics Micro Mini Threats to Your Experiment!
Doing Social Psychology Research
Writing Program Assessment Report Fall 2002 through Spring 2004 Laurence Musgrove Writing Program Director Department of English and Foreign Languages.
Agenda for January 25 th Administrative Items/Announcements Attendance Handouts: course enrollment, RPP instructions Course packs available for sale in.
Sex Differences in Visual Field Lateralization: Where are they? Christine Chiarello 1, Laura K. Halderman 1, Suzanne Welcome 1, Janelle Julagay 1 & Christiana.
© 2005 The McGraw-Hill Companies, Inc., All Rights Reserved. Chapter 5 Making Systematic Observations.
Chapter 2 – Tools of Positive Analysis
Chapter 7 Correlational Research Gay, Mills, and Airasian
Reliability and Validity. Criteria of Measurement Quality How do we judge the relative success (or failure) in measuring various concepts? How do we judge.
Chapter 14 Inferential Data Analysis
Comprehensive Assessment System Webinar #6 December 14, 2011.
Questions to check whether or not the test is well designed: 1. How do you know if a test is effective? 2. Can it be given within appropriate administrative.
Methodology: How Social Psychologists Do Research
ICT TEACHERS` COMPETENCIES FOR THE KNOWLEDGE SOCIETY
Technical Issues Two concerns Validity Reliability
EDRS6208 Lecture Three Instruments and Instrumentation Data Collection.
Chapter 4 Principles of Quantitative Research. Answering Questions  Quantitative Research attempts to answer questions by ascribing importance (significance)
© 2013 Cengage Learning. Outline  Types of Cross-Cultural Research  Method validation studies  Indigenous cultural studies  Cross-cultural comparisons.
How to conduct good investigation in social sciences Albu Iulian Alexandru.
Instrumentation.
References : 1- The A-Z of Social Research 2- The Sage Encyclopedia of Qualitative Research Methods Research design /
Evaluation of software engineering. Software engineering research : Research in SE aims to achieve two main goals: 1) To increase the knowledge about.
CSD 5100 Introduction to Research Methods in CSD Observation and Data Collection in CSD Research Strategies Measurement Issues.
Instrumentation (cont.) February 28 Note: Measurement Plan Due Next Week.
Module 4 Notes Research Methods. Let’s Discuss! Why is Research Important?
The Analysis of the quality of learning achievement of the students enrolled in Introduction to Programming with Visual Basic 2010 Present By Thitima Chuangchai.
Educators’ Attitudes about the Accessibility and Integration of Technology into the Secondary Curriculum Dr. Christal C. Pritchett Auburn University
Online Course Evaluations Is there a perfect time? Presenters: Cassandra Jones, Ph.D., Director of Assessment Michael Anuszkiewicz, Research Associate.
MGS3100_04.ppt/Sep 29, 2015/Page 1 Georgia State University - Confidential MGS 3100 Business Analysis Regression Sep 29 and 30, 2015.
For ABA Importance of Individual Subjects Enables applied behavior analysts to discover and refine effective interventions for socially significant behaviors.
Why is Research Important?. Basic Research Pure science or research Research for the sake of finding new information and expanding the knowledge base.
Measurement Validity.
What is Science? or 1.Science is concerned with understanding how nature and the physical world work. 2.Science can prove anything, solve any problem,
Dissertation Defense Machine Assisted Grading of Rare Collectibles through the COINS framework By: Rick Bassett July 24, 2003.
A Comparison of General v. Specific Measures of Achievement Goal Orientation Lisa Baranik, Kenneth Barron, Sara Finney, and Donna Sundre Motivation Research.
Research Methodology and Methods of Social Inquiry Nov 8, 2011 Assessing Measurement Reliability & Validity.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
TEMPLATE DESIGN © Attitudes Towards Science: Demographics, Education, or Knowledge? Christina K. Pikas,
Online students’ perceived self-efficacy: Does it change? Presenter: Jenny Tseng Professor: Ming-Puu Chen Date: July 11, 2007 C. Y. Lee & E. L. Witta (2001).
Reliability and Validity Themes in Psychology. Reliability Reliability of measurement instrument: the extent to which it gives consistent measurements.
Chapter 14: Affective Assessment
University of Wisconsin-Eau Claire Andrea C. Privratsky, M.S.E., William Frankenberger, Ph.D. Teacher Attitudes on the use of the Responsive Classroom.
RESEARCH METHODS IN INDUSTRIAL PSYCHOLOGY & ORGANIZATION Pertemuan Matakuliah: D Sosiologi dan Psikologi Industri Tahun: Sep-2009.
T YPES OF R ELIABILITY AND V ALIDITY Research Methods (cont’d)
PSY 219 – Academic Writing in Psychology Fall Çağ University Faculty of Arts and Sciences Department of Psychology Inst. Nilay Avcı Week 9.
Special Topics in Educational Data Mining HUDK5199 Spring term, 2013 March 6, 2013.
Reliability a measure is reliable if it gives the same information every time it is used. reliability is assessed by a number – typically a correlation.
Outline Variables – definition  Physical dimensions  Abstract dimensions Systematic vs. random variables Scales of measurement Reliability of measurement.
CHAPTER 1 THE FIELD OF SOCIAL PSYCHOLOGY. CHAPTER OBJECTIVES After reading this chapter, you should be able to: Offer a definition of social psychology.
Crystal Reinhart, PhD & Beth Welbes, MSPH Center for Prevention Research and Development, University of Illinois at Urbana-Champaign Social Norms Theory.
1 Innovative Teaching and Learning (ITL) Research Corinne Singleton SRI International.
Participants & Procedure
Jenn Shinaberger Lee Shinaberger Corey Lee Coastal Carolina University
ARE YOU AS SMART AND CREATIVE AS YOU THINK
Jenn Shinaberger Corey Lee Lee Shinaberger Coastal Carolina University
Journalism 614: Reliability and Validity
Reliability & Validity
The Analysis Phase: Gathering Information
Formative Assessments Director, Assessment and Accountability
Erin M. Buchanan, Katherine D. Miller, Emily R. Klug, and the DOOM Lab
Presentation transcript:

Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two. There were no differences in mean claim rates between Spring and Summer studies, yet a significant drop from the Fall to the Winter studies, the latter involving more accountability. Fortunately, overclaiming is typically robust to such changes, so mean Accuracy did not vary, while mean Bias did (also decreasing significantly), suggesting that participant attitude had changed. Nevertheless, individual item claim rates were astonishingly consistent across samples. The fact that the studies very in accountability contexts and item extremity, may also explain the difference in reliabilities observed below. Another standard form of reliability (simultaneous correlations) is called Concurrent Reliability. Winter surveys combined items used in the Fall with 50 new items (the latter having more difficult Reals), both administered in the same session, Results from earlier surveys informed our development of better performing item sets. The table below shows regression results showing the contributions of accuracy and bias in predicting the same general vocabulary measure. The improved items in the Winter study boosted the R-square substantially. Overall, four different item sets and five samples indicated the value of the VOCE items. A multi-measure study confirmed its validity for predicting later course grades. Abstract The Overclaiming (OC) technique can measure knowledge via Accuracy (the ability to distinguish real items from foils) while simultaneously measuring Bias (the tendency to exaggerate knowledge). Selecting optimal items can be challenging. We used psycholinguistic parameters to develop a new instrument titled the Vocabulary Over Claiming of English (VOCE). Motivation Typical overclaiming studies use items covering various knowledge domains, such as science, literature, politics, popular culture, etc. However, performance can be impaired by lack of subject knowledge or interest in the domain, lowering the power of the instrument. To develop a more efficient and robust measure, we selected a core domain of general knowledge (vocabulary) and sought to fine-tune claim rates by using psycholinguistic properties to increase predictive validity. Linguistic Predictors To create plausible Foils, we used special pseudo-word generating software (both available free, online), first Wuggy ( then WordGen ( They can create words that conform to various linguistic/lexical constraints of a given language. For those most consistent with English norms, Google was used to confirm their uniqueness. To measure actual linguistic properties of both Foils and Reals, we used the English Lexicon Project (ELP) database (elexicon.wustl.edu/), which contains both frequency of use, lexical properties, and experimental lexical decision data. Conclusions The VOCE provides a novel vocabulary measure using the overclaiming technique. It is more efficient and less domain specific than earlier OC measures. The improvements resulted from consideration of linguistic properties of both real and foil items. For example, claim rates for Reals were strongly predicted by frequency and recognition rate from the ELP database, yielding an adjusted R 2 of.90. The final VOCE item set shows concurrent validity as well as predictive validity with a relatively small number of familiarity ratings. Developing a VOCE: Using Psycholinguistics to Enhance Vocabulary Overclaiming Measures Patrick J. Dubois & Delroy L. Paulhus The University of British Columbia Studies Participants were students enrolled in undergraduate psychology courses at the University of British Columbia. Students received course credit for participation. Each study (except one done on paper in a classroom) was conducted online, with participants presented with a list of words and asked if they had heard of it, or never heard of it. Spring, Summer and Fall surveys involved 50 items, with the Winter study repeating the 50 Fall items plus 50 new items (to measure test-retest reliability and concurrent validity). The Class study was done on paper with only 30 items. Both the Class and Winter studies included other measures, such as an independent general vocabulary test (no similar items), to measure predictive validity.