1. Reliability EDUC 307

2. Reliability  How consistent is our measurement?  the reliability of assessments tells the consistency of observations.  Two or more observations using the same assessment should yield similar results.

3. Reliability and Validity  -reliability in scores does not guarantee validity of the test. It indicates only that the scores are consistent from one assessment to the next.  1. Reliability is verified by the consistency of the observation of an outcome while validity tells me the relevance of an observation for making a decision.  2. Both are true or not true by degrees.

4. Reliability in Educational Testing  -must have reliability or test scores lack credibility and no accurate decisions can be drawn from the scores.  A. True scores and errors of measurement 1. Test scores typically include some error or degree of imprecision; the extent to which a test deviates from the true source is called measurement error. 2. Accidental errors are called random errors.  B. Teachers are test maker and their test must be reliable.

5. How do Test makers Determine Reliability?  Test publishers know that reliability is determined by how well two measurements of a given trait agree with each other in the ranking of each student.  A sample group of people is measured more than once and the results are compared and correlated.

6. Reliability and Correlation -Two basic steps  1. Using two comparable assessment procedures get two assessments of a skill on all persons in a sample group.  2. Using correlation procedures compare position of each student in the group on the first assessment with the student's position on the second of these two assessments.  3. Reliability coefficients-correlation coefficients for reliabilities have the same range as those for validity coefficients. Educators look for reliability coefficients to be at least.80 for commercial tests.

7. Making Use of Reliability Data  -selection of techniques determines the aspects of reliability measured

8. Test-Retest Reliability  Used to refer to a single student's score over time as this method will give us an indicator of stability. The test-retest method involves the following sequence of steps:  1. Administer the test to a group of students  2. After a short period of time, administer the SAME test to the SAME students.  3. Correlate the scores from the two administrations.  The main problem associated with the test-retest method for classroom teachers is that teachers do not typically give the same identical test on two occasions.

9. Alternate Forms Reliability  This method involves the administration of different forms of a test. Specifically, alternate forms of the same test would present students with different items of the same difficulty and covering the same content. In contrast, equivalent forms of the same test include the identical items, simply arranged in different sequence.  The following steps outline the procedure for establishing alternate-forms reliability:  1. Administer Form A of the test to a group of students.  2. Shortly afterward, administer Form B of the test to the same students.  3. Correlate the scores from the two administrations.  The main problem associated with the alternate-forms method of estimating reliability is that it is also not very feasible for classroom teachers; giving two tests that cover the same content.

10. Split-half Method  This method involves dividing one test into two comparable halves and follow these steps:  1. Administer one long test to a group of students.  2. Separate the items on the test into two equal halves (this is typically done by placing all even-numbered items into one subtest and all odd- numbered items into the other.)  3. Correlate the scores from the two shorter tests. This method can be reasonably used by classroom teachers, but it tends to provide an "over-correction" or slightly higher final reliability coefficient that would be obtained using other methods.

11. Using Internal Consistency  if all items in a test presumed to assess a common, unitary trait, internal consistency procedures may be useful indicators of reliability.  Some applications of reliability data 1. Different forms in high school to preclude students sharing items 2. School counselor's use of an aptitude test 3. Class placement-math aptitude for advanced math course

12. Factors that Affect Reliability  A. Length of Test -as tests get longer they tend to become more reliable.  1. Short test tend to be less reliable because they do not take enough samples of the required behavior to adequately represent a person's performance. (however twice as long in NOT twice as reliable)  2. Observations are more reliable if the teacher involves more data points in the checklist.

13. Factors that Affect Reliability  B. Pressure of time limit -extent to which the time limit of the test puts pressure on students in their effort to complete the test. (speediness)  1. Tests that are highly speeded tend to produce higher reliability coefficients than the same tests under generous timing conditions.  2. Speed is a serious consideration only when the time is so short that only a few rapid workers get the chance to attempt all of the items.

14. Factors that Affect Reliability  C. Heterogeneity of the group - the more heterogeneous in skill in the group of students who take an examination, the wider will be the range of scores.  1. The larger the spread of scores, the higher the reliability is expected to be.

15. Factors that Affect Reliability  D. Level of difficulty -difficult tests tend to produce skewed distributions with scores piling up in the lower ranges. Easy test scores pile up at upper end of the distribution range. 1. Narrow spread of scores has lower reliability

16. Factors that Affect Reliability  Generally, if you: avoid short tests, give student time to try all or nearly all of the problems, and include a range of difficulty that will spread out the scores, you can expect better reliability for your tests.

17. The Relationship Between Validity and Reliability  1. Validity is the more important feature, for if the results of an assessment are valid, it logically follows that they will also be reliable, and reliability is a prerequisite to having validity.  2. Reliability is an important characteristic and a necessary condition for validity. However, reliability is NOT a sufficient condition for the presence of validity. Simply because the results of an assessment are consistent does not guarantee that those results are also valid.

18. Teacher Responsibilities Related to Validity and Reliability  Teachers have several responsibilities that are directly related to assessment validity and reliability. However, some teachers do not see these types of quality assurance measures as being their responsibility. On the contrary, if classroom teachers do not ensure that their own assessments are both valid and reliable, who will? Claiming ignorance or a lack or responsibility is certainly not justification for simply assuming that your classroom assessments are valid and reliable. Teachers and only teachers are responsible for determining the extent to which their classroom assessments meet certain standards of quality. Teachers have a professional responsibility to students, parents and other teachers, and to themselves to ensure the quality of their self-developed or selected classroom assessments. Classroom teachers must take the time to gather the pertinent evidence and to ensure that their assessments result in legitimate and accurate conclusions about student behaviors and academic performance.