Technical Adequacy of Tests Dr. Julie Esparza Brown SPED 512: Diagnostic Assessment

Two Essential Concepts Reliability: Test consistency Validity: Test measures what it says it does

Psychometric Properties To create an instrument with sound psychometric properties means that it is reliable.

Reliability Reliability is whether a test or measurement tool measures something consistently.

1.Reliable - Hits same part of the target each time, consistent but not valid because goal is center. 2.Valid – evenly distributed around goal, but not reliable because shots are off the mark and inconsistent. 3.Not reliable – shots not tightly clustered, not consistent and not valid because pattern is not around true center. 4.Reliable – darts are close together and valid because darts clustered around where they were aimed.

Reliability Type of Reliabilit y When You Use ItHow You Do ItEx. Of What You Can Say When You’re Done Test-retest reliability When you want to know whether a test is reliable over time. Correlate the scores from a test given in Time 1 with the same test given in Time 2 The Bonzo test of identity formation for adolescents Is reliable over time Parallel form of reliability When you want to know if several different forms of a test are reliable or equivalent Correlate the scores from one form of the test with the scores from a second form of the same test of the same content (but not the exact same test)

Reliability Type of Reliabilit y When You Use ItHow You Do ItEx. Of What You Can Say When You’re Done Internal consistenc y reliability When you want to know if the items on a test assess one, and only one, dimension. Correlate each individual item score with the total score. All of the items on the SMART Test of Creativity assess the same construct. Interrater reliability When you want to know whether there is consistency in the rating of some outcome. Examine the percent of agreement between raters. The interrater reliability for the best-dressed football player judging was.91, indicating a high degree of agreement between judges.

Interpreting Reliability Coefficients We want two things: o For reliability coefficients to be positive (or direct) and not to be negative (or indirect) o Reliability coefficients that are as large as possible (between.00 and +1.00). o Reliability is a function of how much error contributes to the observed score. The lower the error, the higher the reliability.

To Increase Test Reliability… Ensure instructions are standardized across all settings when the test is administered. Increase the number of items because the larger the sample the more likely it is representative and reliable; this is especially true for achievement tests. Delete unclear items. Minimize the effects of external events.

Validity The property of an assessment tool that indicates the tool does what it says it does. A valid test measure what it is supposed to.

Validity Type of ValidityWhen You Use ItHow You Do ItEx. Of What You Can Say When You’re Done Content validityWhen you want to know whether a sample of items truly reflects an entire universe of items in a certain topic. Ask Mr. or Ms. Expert to make a judgment that the test items reflect the universe of items in the topic being measured. My weekly quiz in my class fairly assesses the chapter’s content. Criterion validity (concur-rent or predictive) When you want to know if test scores are systematically related to other criteria that indicate the test taker is competent in a certain area Correlate the scores from the test with some other measure that is already valid and assesses the same set of abilities The EATS tests (of culinary skills) has been shown to be correlated with being a fine chef 2 years after culinary school (ex. Of predictive validity)

Validity Type of Validity When You Use It How You Do ItEx. Of What You Can Say When You’re Done Construct validity When you want to know if a test measures some underlying psychological construct Correlate the set of test scores with some theorized outcome that reflects the construct for which the test is being designed. It’s true – men who participate in body contact and physically dangerous sports score higher on the TEST(osterone) test of aggression.

Quantifying Validity The maximum level of validity is equal to the square root of the reliability coefficient. Example, if the reliability coefficient of a test is.87, the validity coefficient can be no larger than.93 (the square root of.87).

Reliability and Validity You can have a test that is reliable but not valid. But, you can’t have a valid test without it first being reliable. If a test does what it is supposed to, then it has to do it consistently to work!

Test Scores A raw score or obtained score on a test is the number of points obtained by an aminee. A true score is that part of an examinee’s observed score uninfluenced by random events. The error of measurement or error score is the difference between an obtained score and its theoretical true score counterpart.

Error Score The error score is that part of the obtained score which is unsystematic, random, and due to chance.

Standard Error of Measurement The standard deviation of errors of measurement that is associated with the test scores for a specified group of test takers.” It is a measure of the variability of the errors of measurement. It is used to help us predict true scores based upon knowledge of obtained score. SEM is the standard deviation of errors of measurement associated with test scores for a specific group.

Score Bands Score bands are sometimes called confidence intervals or confidence bands because they allow us to make probabilistic statements of confidence about an unknown value. Score bands have lower and upper limits on the score scale and provide an estimate that is a range or band of possible test scores.

Score Bands An example of a score band or confidence interval is “I am 95 percent confident that the examinee’s obtained score will be between 46 and 54 (given a true score of 50 and an SEM of two). 68 percent confidence intervals are the most commonly used.

Confidence Intervals In a normal distribution: o The area between one SDs below and one SDs above the mean is 68 % of the total area under the curve o The area between two (actually 1.96) SDs below and two SDs above the mean is 95% o The area between 2.58 SDs below and 2.58 SDs above the mean is 99%

Confidence Intervals If we add and subtract one SEM from a person’s test score, we will have an estimate of the true score.

Standard Scores Test developers calculate the statistical average based on the performance of students tested in the norming process of test development. That score is assigned a value. Different performance levels are calculated based on the differences among student scores from the statistical average and are expressed as standard deviations.

Standard Scores These standard deviations are used to determine at what scores fall within the above average, average, and below average ranges. Standard scores and standard deviations are different for different tests. Many of the commonly used tests, such as the Wechsler Intelligence Scales, have an average score of 100 and a standard deviation of 15.

Standard Scores Standardized test scores enable us to compare a student's performance on different types of tests. Although all test scores should be considered estimates, some are more precise than others. Standard scores and percentiles, for example, define a student's performance with more precision than do t-scores, z-scores, or stanines.

Standard Deviation Standard deviation measures how widely spread data points are. If data values are all equal to one another, then the standard deviation is zero. Under a normal distribution, ± one standard deviation encompasses 68% of the measurements and ± two standard deviations encompasses 96% of the measurements.

Standard Deviation If a high proportion of data points lie near the mean (average) value, then the standard deviation is small. o An experiment that yields data with a low standard deviation is said have high precision. If a high proportion of data points lie far from the mean value, then the standard deviation is large. o An experiment that yields data with a high standard deviation is said to have low precision.

Test Scores Observed score – what someone actually gets on a test. True score – the true, 100% accurate reflection of what the someone actually knows. An observed score is usually close to the true score but they are rarely the same. The difference between the two is in the amount of error that is introduced.

Observed Score If someone has 89 on a test, but their true score is 80, that means the 9 points in the difference (the error score) are due to error or the reason why individual test scores vary from being 100% true. What may be the source of the error: o Room is too warm o Person didn’t have time to study o Person has a fever o ????

Observed Score We need to reduce the errors as much as possible. The less error, the more reliable the score.

Reporting Test Scores Score or confidence bands are the best way to report test scores. The score band provides reasonable limits for estimating true score; it is an adequate approximation when the test reliability is reasonably high and the obtained score is not an extreme deviate from the mean of the reference group. You can say, “It is fairly likely your daughter’s true ability lies between 110 and 120.”

Test Selection In selecting a published test: o Read test manual to determine if it has reported the reliability, SEM and norms (including confidence bands) o The above information is reported for a reference group similar to your examinee o Be sure manual explains clearly how the information was gathered and how the confidence bands in the manual were calculated

Reliable and Value If the tools you use to collect data are not reliable nor valid, then the results will be inconclusive.