Presentation is loading. Please wait.

Presentation is loading. Please wait.

Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two.

Similar presentations


Presentation on theme: "Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two."— Presentation transcript:

1 Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two. There were no differences in mean claim rates between Spring and Summer studies, yet a significant drop from the Fall to the Winter studies, the latter involving more accountability. Fortunately, overclaiming is typically robust to such changes, so mean Accuracy did not vary, while mean Bias did (also decreasing significantly), suggesting that participant attitude had changed. Nevertheless, individual item claim rates were astonishingly consistent across samples. The fact that the studies very in accountability contexts and item extremity, may also explain the difference in reliabilities observed below. Another standard form of reliability (simultaneous correlations) is called Concurrent Reliability. Winter surveys combined items used in the Fall with 50 new items (the latter having more difficult Reals), both administered in the same session, Results from earlier surveys informed our development of better performing item sets. The table below shows regression results showing the contributions of accuracy and bias in predicting the same general vocabulary measure. The improved items in the Winter study boosted the R-square substantially. Overall, four different item sets and five samples indicated the value of the VOCE items. A multi-measure study confirmed its validity for predicting later course grades. Abstract The Overclaiming (OC) technique can measure knowledge via Accuracy (the ability to distinguish real items from foils) while simultaneously measuring Bias (the tendency to exaggerate knowledge). Selecting optimal items can be challenging. We used psycholinguistic parameters to develop a new instrument titled the Vocabulary Over Claiming of English (VOCE). Motivation Typical overclaiming studies use items covering various knowledge domains, such as science, literature, politics, popular culture, etc. However, performance can be impaired by lack of subject knowledge or interest in the domain, lowering the power of the instrument. To develop a more efficient and robust measure, we selected a core domain of general knowledge (vocabulary) and sought to fine-tune claim rates by using psycholinguistic properties to increase predictive validity. Linguistic Predictors To create plausible Foils, we used special pseudo-word generating software (both available free, online), first Wuggy (http://crr.ugent.be/programs-data/wuggy) then WordGen (http://www.wouterduyck.be/?page_id=29): They can create words that conform to various linguistic/lexical constraints of a given language. For those most consistent with English norms, Google was used to confirm their uniqueness. To measure actual linguistic properties of both Foils and Reals, we used the English Lexicon Project (ELP) database (elexicon.wustl.edu/), which contains both frequency of use, lexical properties, and experimental lexical decision data. Conclusions The VOCE provides a novel vocabulary measure using the overclaiming technique. It is more efficient and less domain specific than earlier OC measures. The improvements resulted from consideration of linguistic properties of both real and foil items. For example, claim rates for Reals were strongly predicted by frequency and recognition rate from the ELP database, yielding an adjusted R 2 of.90. The final VOCE item set shows concurrent validity as well as predictive validity with a relatively small number of familiarity ratings. Developing a VOCE: Using Psycholinguistics to Enhance Vocabulary Overclaiming Measures Patrick J. Dubois & Delroy L. Paulhus The University of British Columbia Studies Participants were students enrolled in undergraduate psychology courses at the University of British Columbia. Students received course credit for participation. Each study (except one done on paper in a classroom) was conducted online, with participants presented with a list of words and asked if they had heard of it, or never heard of it. Spring, Summer and Fall surveys involved 50 items, with the Winter study repeating the 50 Fall items plus 50 new items (to measure test-retest reliability and concurrent validity). The Class study was done on paper with only 30 items. Both the Class and Winter studies included other measures, such as an independent general vocabulary test (no similar items), to measure predictive validity.


Download ppt "Results Following Signal Detection Theory, Accuracy is calculated as the difference between Real and Foil claim rates, and Bias is the mean of the two."

Similar presentations


Ads by Google