1. Chapter 7 Copyright © Allyn & Bacon 2008 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: Any public performance or display, including transmission of any image over a network; Preparation of any derivative work, including the extraction, in whole or in part, of any images; Any rental, lease, or lending of the program.

2.  Classifying educational measures  Four types of measures ◦ Tests ◦ Questionnaires ◦ Observations ◦ Interviews  Locating educational measures  Criteria for evaluating instruments Copyright © Allyn & Bacon 2008

3.  Three classifications ◦ Cognitive or non-cognitive ◦ Commercially or locally prepared ◦ Self-report or observations by others Copyright © Allyn & Bacon 2008

4.  Cognitive or non-cognitive ◦ Cognitive measures focus on what a person knows or is able to do mentally ◦ Non-cognitive measures focus on affective traits or characteristics (e.g, personality traits, attitudes, values, interests, preferences, etc.) Copyright © Allyn & Bacon 2008

5.  Commercially prepared or locally developed ◦ Commercially prepared measures are developed for widespread use with a focus on technical merit ◦ Locally prepared measures are developed by a researcher for specific situations with some, but not extensive, concern for technical characteristics Copyright © Allyn & Bacon 2008

6.  Self-report or observations by others ◦ Self-report measures require the participants to supply the response (e.g., tests, questionnaires, interviews, etc.) ◦ Observations by others require participants to be observed by others who record the data (e.g., observation, unobtrusive measures, etc.) Copyright © Allyn & Bacon 2008

7.  Four types ◦ Tests ◦ Questionnaires ◦ Observations ◦ Interviews Copyright © Allyn & Bacon 2008

8.  A test is an instrument that requires participants to complete a cognitive task by responding to a standard set of questions  Score interpretation ◦ Norm-referenced - individual scores are interpreted relative to the scores of others in a well defined norming group  John’s scores places him in the 95 th percentile  Sally’s score is in the bottom quartile Copyright © Allyn & Bacon 2008

9.  Score interpretation ◦ Criterion-referenced - individual scores are interpreted in terms of the student’s performance relative to some standard or criterion  Jeanne passed the Louisiana High School Graduate Exit Exam  Sammy did not make the cutoff for being promoted to the 7 th grade Copyright © Allyn & Bacon 2008

10.  Standard scores - transformations of raw scores into easily interpreted standard metrics ◦ Z-score - the difference between a raw score and the mean in standard deviation units  z = (raw score - mean) / standard deviation ◦ Z-scores are algebraically transformed to standard scales such as percentiles, grade equivalents, SAT, ACT, GRE, etc. Copyright © Allyn & Bacon 2008

11.  Standard scores ◦ All standard scores are interpreted relative to the scores of others in the norming group  Barbara’s SAT score of 700 is very, very good relative to the scores of the norm group because it is two (2) standard deviations above the mean (i.e., in the 99 th percentile)  Beau’s grade equivalent score of 3.0 is poor given that he is in the 6 th grade and has scored at a level equal to that of third graders taking the test Copyright © Allyn & Bacon 2008

12.  Standardization ◦ Standardized tests have uniform procedures for administration, scoring, and interpreting test scores ◦ Types of standardized tests  Achievement - tests of content knowledge or skills  Aptitude - tests which are used to predict future cognitive performance  Standards-based - criterion-referenced tests based on established standards  Domain-referenced  High-stakes tests High-stakes Copyright © Allyn & Bacon 2008

13.  A questionnaire is an instrument containing statements designed to obtain a participant’s perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits  Personality inventories ◦ Psychological orientation (i.e., general psychological adjustment) ◦ Educational orientation (i.e., traits such as self-concept or self-esteem that are related to learning and motivation) Copyright © Allyn & Bacon 2008

14.  Attitudes, values, or interests - affective traits that indicate some degree of preference toward something ◦ Scales - a continuum that describes participant’s responses to a statement  Likert  Semantic Differential  Checklists  Ranked items Copyright © Allyn & Bacon 2008

15.  Likert scales ◦ Response options require the participant to determine the extent to which they agree with a statement  An odd number of options provides for a middle or neutral response  Strongly agree, agree, neutral, disagree, or strongly disagree  An even number of options eliminates a response of neutral  Strongly agree, agree, disagree, or strongly disagree) Copyright © Allyn & Bacon 2008

16.  Likert scales ◦ Statements must reflect extreme positive or extreme negative positions  I hate my teacher.  The textbook has been a valuable resource. ◦ A subject’s response positions them on a continuum  Strongly agreeing with the statement “I hate my teacher” indicates a very negative attitude.  Strongly agreeing with the statement “The textbook has been a valuable resource” indicates a very positive attitude. Copyright © Allyn & Bacon 2008

17.  Semantic differential scales ◦ Response options reflect a continuum of bipolar adjectives related to some aspect of the trait being measured  Fair: __ __ __ __ __ :Unfair  Interesting: __ __ __ __ __ :Boring ◦ Aspects of the traits being measured are usually stated in a few words  My teacher is ….  The textbook is …. Copyright © Allyn & Bacon 2008

18.  Semantic Differential scales ◦ A participant’s response positions them on a continuum  Responses of “fair” to the statement “My teacher is ….” indicate a positive attitude  Responses of “boring” to the statement “My teacher is ….” indicate a negative attitude Copyright © Allyn & Bacon 2008

19.  Checklists - responses require participants to identify specific options from which they choose those options that appeal to them  Ranked items - responses require participants to place a limited number of items into sequential order Copyright © Allyn & Bacon 2008

20.  Problems with measuring non-cognitive traits ◦ Difficulty clearly defining what is being measured  Self-concept or self-esteem ◦ Response set - a tendency to respond the same way to all items  Strongly agreeing with each statement ◦ Social desirability - a tendency to respond to items in a way that is socially desired or accepted ◦ Faking - a tendency to respond inaccurately  Agreeing with statements because of the negative consequences associated with disagreeing Copyright © Allyn & Bacon 2008

21.  Controlling problems ◦ Equal numbers of positively and negatively worded statements ◦ Alternating positive and negative statements and/or bipolar adjectives ◦ Providing confidentiality or anonymity to respondents Copyright © Allyn & Bacon 2008

22.  Observations - direct observations of behaviors ◦ Natural or controlled settings ◦ Structured or unstructured observations ◦ Detached or involved observers Copyright © Allyn & Bacon 2008

23.  Inference ◦ Low inference - involves little if any inference on the observers part  Children are in their seats  A teacher uses math manipulatives ◦ High inference - involves high levels of inference on the observers part  Children are happy  A teacher lectures effectively Copyright © Allyn & Bacon 2008

24.  Laboratory observations ◦ Specified environment ◦ Use of structured forms and procedures ◦ Concern with demand characteristics  Structured field observations ◦ Natural setting ◦ Use of structured forms and procedures  Frequency counts  Duration  Interval  Continuous  Time sampling Copyright © Allyn & Bacon 2008

25.  Controlling observer effects ◦ Observer bias  Training  Inter-rater reliability  Multiple observers  Eliminating prejudice ◦ Contamination - knowledge of the study influences the observation  Training  Targeting specific behaviors  Observers do not know of the expected outcomes  Observers are “blind” to which group is which Copyright © Allyn & Bacon 2008

26.  Controlling observer effects ◦ Halo effect - initial ratings influence subsequent ratings  Training  Targeting specific behaviors  Multiple observers Copyright © Allyn & Bacon 2008

27.  Interviews involve orally questioning participants and recording their responses  Advantages ◦ Establish rapport ◦ Enhance motivation ◦ Clarify responses through additional questioning ◦ Capture the depth and richness of responses ◦ Allow for flexibility ◦ Reduce “no response” and/or “neutral” responses Copyright © Allyn & Bacon 2008

28.  Disadvantages ◦ Time consuming ◦ Expensive ◦ Small samples ◦ Subjective  Types of questions ◦ Structured ◦ Semi-structured ◦ Unstructured ◦ Leading Copyright © Allyn & Bacon 2008

29.  Sources of concern ◦ Bias ◦ Contamination ◦ Interviewer characteristics  Age, race, gender, etc. ◦ Conduct of the interview ◦ Response recordings Copyright © Allyn & Bacon 2008

30.  Test reviews ◦ Buros Institute Buros Institute ◦ ETS Test Collection ETS Test Collection ◦ ERIC/AE Test Locator ERIC/AE Test Locator  Test authors and publishers  Computerized ERIC searches for relevant articles  Reference sections of articles already read  See Table 7.4 Copyright © Allyn & Bacon 2008

31.  Validity evidence  Reliability evidence  Descriptions of the instruments  Administration procedures  Norming information for norm-referenced tests  Standards for criterion-referenced tests  Meaningful scores and score interpretations Copyright © Allyn & Bacon 2008

32.  Avoidance of response problems in non- cognitive measures  Training observers and interviewers  High standards for observers using high inference observations  Minimum interviewer effects Copyright © Allyn & Bacon 2008