SPHS 5780, LECTURE 6: PSYCHOMETRICS, “STANDARDIZED ASSESSMENT”, NORM-REFERENCED TESTING

pSYCHOMETRICS

psychometricS “psycho” +“metric” Who is responsible for assessing psychometric adequacy? Is this the case even when a test or procedure is mandated? Who is affected when a psychometrically inadequate measure is used?

psychometricS Discussion: What makes a test or diagnostic procedure “good”?

psychometricS When assessing psychometric adequacy, we are assessing rigor of assessment Validity: Testing actually assesses what you think it is assessing (what it is supposed to be assessing) Reliability: Testing is consistent across time and across different clients

psychometricS Is assessment of psychometric adequacy (reliability and validity) needed for: Formal, norm-referenced tests? Criterion-referenced tests? Behavioral observation?

Psychometrics Rigor of assessment for informal testing Validity: Clear definition of the test domain, on which experts agree, as evidenced through “mastery by masters” and “non-mastery by non-masters” Reliability: Consistency in decision outcome “distance” from cut-off score score on closely related versions of measure Rigor or assessment for formal tests will be addressed in more detail in the current lecture

standardization

Standardization insures that all people taking the test….. receive the same experience are expected to perform the same task with the same materials receive the same amount of assistance from the examiner are evaluated according to a standard set of criteria WELL CONCIEVED STANDARDIZED TESTS ALLOW FOR MEANINGFUL COMPARISON AMONG CHILDREN Ideally, standardized test are developed by giving the test to a large group of children so that they (the test makers) may compute an acceptable range of variation in the scores for the age range covered. If the same materials, procedure, scoring are not used on all children than the results will have limited comparability across children. Example: put together a three pieces puzzle some would show child first some might not show the child first some would impose a 1 minute strategy others would have limitless times. Strict adherence to standardized procedure is fundamentally important Problem doing this with some handicapped children---Informal use of standardized test. …regardless of time or place of administration, regardless of clinician

Standardization “Standardized test” is a term commonly used to refer to norm-referenced tests. Why? Does this mean that other types of tests are not standardized?

Standardization How does standardization support a test’s validity? a test’s reliability?

Review of the formal /informal distinction

Purpose of formal vs. Informal Testing To determine eligibility for services, a norm-referenced test needs to be sensitive to the presence of a disorder (not miss those who have disorder) and specific in determination of referrals (not refer those who do not have a disorder). The same requirements do not hold for informal testing. Do you see why?

Examples of formal vs. Informal Testing Formal: CELF Informal: Augmentative communication evaluation

NORM-REFERENCED MEASURES CRITERION-REFERENCED MEASURES Function Identify clients who need help Plan the nature of that help Normative group performance Establish group of scores against which client is compared Establish performance standard (performance “criterion”) NOTE: Performance standard may be client’s own past performance! Interpreta- tions of client’s performance Relative to range of performance of others Relative to standard (“criterion”): mastery vs. non-mastery When to use When norms are available When broad content (“trait”) is of interest When norms are unavailable or inappropriate When specific skills/behaviors are of interest Specific Dx Focus Tx planning

NORM-REFERENCED MEASURES CRITERION-REFERENCED MEASURES Level of detail for given area Low detail (broad area) High detail (narrow area) Summary scores Converted scores Raw scores Choice of test items Those items on which test-takers perform variably Those items always passed by masters Those items always failed by non-masters Decision to be defended Relative ranking compared to normative group Dichotomous (mastery, non-mastery)

Formal (Norm-referenced) Tests: Nature

Purpose and procedure of norm-referenced tests Fundamental purpose is to rank individuals “to determine if an individual obtains a score similar to the group average or, if not, how far away from average the score is” (H&P, M&P) “(to determine) if there is a problem, or a significant enough difference from standard performance to warrant concern with regard to normalcy” Performance summarized with reference to the “standardization sample” (normative sample), through conversion of raw scores percentile scores standard scores

Advantages of Norm-Referenced Tests Objectivity Replicability Elimination of unwanted / uncontrolled variation

Disadvantages of Norm-Referenced Tests For some domains (e.g. language) it is difficult to find a language test that is: valid reliable The normative sample needs to represent your client NRTs do not accommodate cultural variation NRTs do not accommodate individual variation

Norm referenced tests cannot: Measure treatment progress Guide creation of therapy goals, through analyzing individual test items Richly describe performance Accommodate cultural variation Discussion of WHY this is the case….

Lecture 06a ends here this is where material for exam 2 ends