1. Introduction to Indirect Student-Learning Assessment (Part II)
Dr. Wayne W. Wilkinson
September 28, 2016
ITTC Faculty Center
Arkansas State University

2. Refresher: Indirect Assessments
Require the inference of student learning:
No direct evidence or demonstration
Common topics of indirect assessments:
Perceptions of successfully meeting program outcomes
Satisfaction/attitudes/feelings toward program
Utility of program

3. Criteria: The Survey Must Have a Purpose
Standards used to help make evaluative judgments
Program outcomes, program quality, program utility . . .
Common issues
Choices of proper criteria to use (e.g., program outcomes)
Disagreements over definition (what does “communicate effectively” mean?)

4. Conceptual vs. Actual Criteria
Theoretical construct:
Ideal set of quality factors
Measures of conceptual criteria:
Operational definitions
Conceptual Criteria (English B.A. Program Outcomes)
Actual Indirect Criteria
Students will be able to discuss the conventions of literary genres (poetry, fiction, non‐fiction, and drama) in close reading and analysis of diverse texts. ?
Students will demonstrate breadth of knowledge regarding literary traditions of British, American, and world texts and also regarding different theoretical approaches to the study of literature, language, and writing.
Students will explain how texts are written and received within diverse cultural and sociohistorical contexts, including global and multicultural perspectives.
Students will develop effective oral and written communication skills, including using primary and secondary sources.

5. Conceptual and Actual Criterion Relations
Conceptual criteria are theoretical . . .
actual and conceptual should overlap
Criterion deficiency
Criterion contamination
Criterion relevance

6. Content and Face Validity
Content Validity: The degree that an assessment includes relevant aspects of the criteria
Determined by Subject Matter Experts
Face Validity: do the items seem legitimate for a measure of the criteria?
Reactions and attitudes of test takers

7. Types of Items Open-ended Restricted (closed-ended)
Partially open-ended
Rating scales

8. Rating Scales Likert-type scale: Sematic differential:
Aggregated ratings of favorable or unfavorable statements
Sematic differential:
Bipolar adjectives
Measure several variables on common scale

9. Rating Scale Issues Number of points on scale: Labeling of scale:
Odd or even (neutral point)
Labeling of scale:
Number of anchors
Numerical values
Item-rating scale congruence

10. Writing Good Items Avoid very long items
Avoid negative statements (no, none, not, etc.)
5th – 6th grade reading level
Avoid “check all that apply” items

11. Writing Good Items
Avoid items that express more than one thought (double barreled items)
Avoid evaluative assumptions
The program aided my understanding of selecting and conducting the proper analysis for specific research questions.
The program provided an excellent preparation for my career.

12. Assembling Your Survey
Survey space and justification
Question organization:
Keep related items together
Question order can have unintended effects
Sensitive topic items after less sensitive
Graphic navigation paths

13. Survey Evaluation Session
Survey goal: English BA Program Indirect Assessment
Justification for each item
Criteria Content Validity
Contamination or deficiency
Item issues
Conceptual Criteria (English B.A. Program Outcomes)
Students will be able to discuss the conventions of literary genres (poetry, fiction, non‐fiction, and drama) in close reading and analysis of diverse texts.
Students will demonstrate breadth of knowledge regarding literary traditions of British, American, and world texts and also regarding different theoretical approaches to the study of literature, language, and writing.
Students will explain how texts are written and received within diverse cultural and sociohistorical contexts, including global and multicultural perspectives.
Students will develop effective oral and written communication skills, including using primary and secondary sources.
Common indirect assessment topics:
Perceptions of successfully meeting program outcomes
Satisfaction/attitudes/feelings toward program
Utility of program