1. 1 Core Assessment: Attributes of a Successful Program at James Madison University TAIR Professional Development Workshop September 2004 Dr. Dena Pastor Center for Assessment and Research Studies

2. 2 James Madison University (JMU) Location: Harrisonburg, VA Location: Harrisonburg, VA Students: Students: ~15,000 undergraduate ~15,000 undergraduate ~700 graduate ~700 graduate State-funded university operated by its own board of visitors State-funded university operated by its own board of visitors

3. 3 JMU Infrastructure for Assessment Center for Assessment and Research Studies (CARS) Center for Assessment and Research Studies (CARS) Supports JMU assessment endeavors Supports JMU assessment endeavors General Education General Education Programs/Majors Programs/Majors Student Affairs Student Affairs Alumni Alumni 9 full-time faculty members & 3 administrative staff 9 full-time faculty members & 3 administrative staff Faculty split their time evenly between assessment duties & teaching/research Faculty split their time evenly between assessment duties & teaching/research Expertise in IRT, SEM, Validity, Computer-Based Testing, HLM, Examinee Motivation, Performance Assessment, Student Development Expertise in IRT, SEM, Validity, Computer-Based Testing, HLM, Examinee Motivation, Performance Assessment, Student Development Offers both M.A. and Ph.D. programs in Assessment & Measurement Offers both M.A. and Ph.D. programs in Assessment & Measurement 10-15 students 10-15 students

4. 4 JMU Infrastructure for General Education Dean of General Education Dean of General Education Cluster Coordinators Cluster Coordinators Faculty given release time for this task Faculty given release time for this task Steering Committee Steering Committee Broad faculty representation Broad faculty representation Positions allocated to departments to teach General Education and major courses Positions allocated to departments to teach General Education and major courses Each cluster undergoes a review process every 5 years Each cluster undergoes a review process every 5 years

5. 5 General Education at JMU 41 credit hours of courses organized into 5 clusters 41 credit hours of courses organized into 5 clusters Cluster 1: 9 credits Skills for the 21 st Century Cluster 2: 9 credits Arts & Humanities Cluster 3: 10 credits The Natural World Cluster 4: 7 credits Social & Cultural Processes Cluster 5: 6 credits Individuals in the Human Community Critical Thinking, Written & Oral Communication, Information Literacy Culture, Philosophy, Fine Arts & Literature. Quantitative & Scientific Reasoning Global & American: History, Govt., Economics, Anthropology Wellness, Psychology & Sociology General Education

6. 6 General Education Course Offerings GHTH 100: Personal Wellness GKIN 100: Lifetime Fitness and Wellness GEIC 101: The Wellness Dimension: Individual Perspectives GPSYC 101: General Psychology GPSYC 160: Life Span Human Development GSOCI 240: Individual in Society GEIC 102: The Sociocultural Dimension: Community Perspectives Wellness: Choose one Sociocultural: Choose one Example from Cluster 5’s Wellness & Sociocultural Domains

7. 7 Assessment Steps Establish Objectives Use Information Select/Design Instruments Collect Information Analyze Information Maintain Information

8. 8 Cluster Learning Objectives Created by cluster faculty with assistance from CARS Created by cluster faculty with assistance from CARS Start by asking: Start by asking: “What should a student know and be able to do as a result of completing their coursework in this cluster?” “What should a student know and be able to do as a result of completing their coursework in this cluster?” What the student knows & is able to do should NOT be tied to specific courses taken within the cluster. What the student knows & is able to do should NOT be tied to specific courses taken within the cluster.

9. 9 Example of Cluster Learning Objectives After completing Cluster Five students should be able to: Understand the dimensions of wellness, the various factors affecting each dimension, and how dimensions are interrelated. Understand the dimensions of wellness, the various factors affecting each dimension, and how dimensions are interrelated. Recognize an individual's level of health and wellness and understand how these levels impact quality of life. Recognize an individual's level of health and wellness and understand how these levels impact quality of life. Discriminate between ethical and non-ethical practices in the social/behavioral sciences. Discriminate between ethical and non-ethical practices in the social/behavioral sciences. Example from Cluster 5’s Wellness & Sociocultural Domains

10. 10 Match Learning Goals With Coursework Have instructors rate the extent to which the material necessary to achieve each of the goals is emphasized in their course Have instructors rate the extent to which the material necessary to achieve each of the goals is emphasized in their course This will help ensure that students are able to achieve goals regardless of which combination of courses they take This will help ensure that students are able to achieve goals regardless of which combination of courses they take GHTH 100: Personal Wellness GKIN 100: Lifetime Fitness and Wellness GEIC 101: The Wellness Dimension: Individual Perspectives GPSYC 101: General Psychology GPSYC 160: Life Span Human Development GSOCI 240: Individual in Society GEIC 102: The Sociocultural Dimension: Community Perspectives A student taking GHTH100 & GPSYC101 should have the same chance of achieving goals as student who takes GKIN100 & GSOCI240

11. 11 Assessment of Learning Objectives Faculty within each cluster decide how to assess the extent to which students are mastering the cluster’s learning objectives Faculty within each cluster decide how to assess the extent to which students are mastering the cluster’s learning objectives Most clusters create a multiple-choice test from scratch, with continual improvements made on the instrument Most clusters create a multiple-choice test from scratch, with continual improvements made on the instrument

12. 12 Assessment Tools Almost all assessment tools are multiple-choice tests created by JMU faculty.

13. 13 Matching Assessment Tools to Learning Objectives Example from Cluster 5’s Sociocultural Domain Assessment

14. 14 Administration of Assessment Tools Two institution-wide Assessment Days Two institution-wide Assessment Days Fall (August): Incoming freshmen tested at orientation Fall (August): Incoming freshmen tested at orientation Spring (February): Students with 45-70 credits (sophomore/junior) tested Spring (February): Students with 45-70 credits (sophomore/junior) tested All undergraduate classes are cancelled on this day All undergraduate classes are cancelled on this day All students are required to participate All students are required to participate course registration blocked for those who do not participate course registration blocked for those who do not participate Students are assigned to testing rooms by last 4-digits of their id #s where a particular series of instruments are administered Students are assigned to testing rooms by last 4-digits of their id #s where a particular series of instruments are administered

15. 15 Assessment Day Data Collection Scheme: Repeated Measures COHORT 1 COHORT 2 COHORT 3 Students in each cohort are tested twice on the same instrument – once as incoming freshmen and again in the second semester of the sophomore year.

16. 16 With this data, we can… 1. Compare pre-test scores to post-test scores (same students) of Cluster “Completers”. 2. Compare Incoming Freshmen with Sophomore Students (different students). 3. Examine the Effect of Number of Courses. 4. Correlate Scores and Course Grades. Sophomores should outperform incoming freshmen. Students who have completed a larger # of courses within a cluster should outperform students who have completed a smaller # of courses. There should be a positive correlation between test scores and the grades received by students in the cluster courses. Students completing their cluster courses should score higher on the post-test as sophomores than they did on the pre-test as incoming freshmen.

17. 17 1. Compare pre-test scores to post-test scores (same students) of Cluster “Completers”. Example from Cluster 4’s American Experience Test

18. 18 2. Compare Incoming Freshmen with Sophomore Students (different students). Example from Cluster 3’s Natural World Assessment

19. 19 3. Examine the Effect of # of Courses. Example from Cluster 3’s Natural World Assessment

20. 20 4. Correlate Scores and Course Grades. Example from Cluster 4’s Global Experience Test

21. 21 The GCOM test is a Computer-Based Test embedded within a course: This test serves as the final exam in all sections of the Gen. Ed. communication course. These are both Computer-Based Tests administered throughout the school-year at a computer lab on campus. All students must exceed the passing score set by the faculty within their first year. Assessment Information Not Collected on Assessment Day A random sample of ~400 students taking the Gen. Ed. writing course are asked to submit a portfolio of their best work which is rated according to a 4 point rubric by the faculty.

22. 22 Other Assessment Information Previously Collected on Assessment Day

23. 23 How does General Education Assessment Benefit JMU Faculty? Collaboration among faculty in different disciplines Collaboration among faculty in different disciplines Opportunity to communicate with one another Opportunity to communicate with one another Feel ownership over assessment tools, more invested in the results Feel ownership over assessment tools, more invested in the results Data may help support resources they would like to acquire Data may help support resources they would like to acquire Gain skills that they can take back into the classroom Gain skills that they can take back into the classroom Opportunities for scholarship Opportunities for scholarship

24. 24 How does General Education Assessment Benefit JMU Students? Receive instruction that is more tailored to their needs Receive instruction that is more tailored to their needs Receive instruction that is more consistent across sections Receive instruction that is more consistent across sections Held accountable for knowledge/skills (ISST, Tech Level tests, GCOM final exam) Held accountable for knowledge/skills (ISST, Tech Level tests, GCOM final exam)

25. 25 How does General Education Assessment Benefit the University? Enhanced public relations and visibility Enhanced public relations and visibility Can communicate information to parents, possible applicants, stakeholders and say “this is what a students will know and be able to do as a result of completing the Gen Ed courses at JMU” Can communicate information to parents, possible applicants, stakeholders and say “this is what a students will know and be able to do as a result of completing the Gen Ed courses at JMU” Enhanced grant funding opportunities Enhanced grant funding opportunities Marketable assessment tools can be a source of income Marketable assessment tools can be a source of income

26. 26 JMU’s Marketable Assessment Tools Computer-Based Multiple-Choice Tests Computer-Based Multiple-Choice Tests ILT: Information Literacy Test (60 items) ILT: Information Literacy Test (60 items) QR: Quantitative Reasoning Test (45 items) QR: Quantitative Reasoning Test (45 items) SR: Scientific Reasoning Test (80 items) SR: Scientific Reasoning Test (80 items) Marketable Tests Being Developed Marketable Tests Being Developed Oral Communication Oral Communication Critical Thinking Critical Thinking

27. 27 What are Other Institutions Doing? Approaches to Gen Ed assessment (Palomba & Banta, 1999) : Approaches to Gen Ed assessment (Palomba & Banta, 1999) : Individual Course-Based Approach Individual Course-Based Approach Multi-Course Approach Multi-Course Approach Noncourse-Based Approach Noncourse-Based Approach

28. 28 Individual Course-Based Approach Instructor submits a document showing how goals are being addressed in their particular course Instructor submits a document showing how goals are being addressed in their particular course May include syllabus, quizzes, tests, assignments, activities, examples of students’ work May include syllabus, quizzes, tests, assignments, activities, examples of students’ work May be judged by other faculty for the extent to which course is covering goals May be judged by other faculty for the extent to which course is covering goals

29. 29 Individual Course-Based Approach College of William and Mary is taking this approach College of William and Mary is taking this approach Also using faculty and student surveys Also using faculty and student surveys Different Gen Ed “clusters” assessed in different years Different Gen Ed “clusters” assessed in different years Office of Assessment oversees process Office of Assessment oversees process http://www.wm.edu/wmoa/index.htm

30. 30 Individual Course-Based Approach Advantages Advantages Maximizes faculty involvement Maximizes faculty involvement Assessment results may help identify changes that need to be made in a particular course Assessment results may help identify changes that need to be made in a particular course May ensure that goals are being addressed in the classroom May ensure that goals are being addressed in the classroom Disadvantages Disadvantages Can be labor intensive for faculty Can be labor intensive for faculty Doesn’t necessarily involve faculty collaboration Doesn’t necessarily involve faculty collaboration Faculty may only choose information that makes them “look good” Faculty may only choose information that makes them “look good” Not necessarily measuring whether or not student has mastered the goal – just measuring whether or not student was exposed to material that would allow them to master the goal Not necessarily measuring whether or not student has mastered the goal – just measuring whether or not student was exposed to material that would allow them to master the goal Some information from Palomba & Banta, 1999

31. 31 Multi-Course Approach Faculty from different disciplines join together to decide upon assessment for their “cluster” Faculty from different disciplines join together to decide upon assessment for their “cluster” Winthrop University takes this approach Winthrop University takes this approach Collect assessment data via course-embedded assessments: common essays, tests, assignments across courses that are graded according to a common rubric Collect assessment data via course-embedded assessments: common essays, tests, assignments across courses that are graded according to a common rubric Also administers the National Survey of Student Engagement (NSSE) and the College BASE to freshmen and seniors (repeated measures) Also administers the National Survey of Student Engagement (NSSE) and the College BASE to freshmen and seniors (repeated measures) Office of Assessment oversees process Office of Assessment oversees process Some information from Palomba & Banta, 1999 http://www.winthrop.edu/assessment/gened.htm

32. 32 Multi-Course Approach Advantages Advantages Encourages faculty collaboration Encourages faculty collaboration Focuses on student experience in a “cluster”, not in a particular course Focuses on student experience in a “cluster”, not in a particular course Shows evidence of student learning Shows evidence of student learning Disadvantages Disadvantages When course-embedded, dictates what assignments/tests, etc. must be administered in a particular course, which may be a turn-off to some instructors When course-embedded, dictates what assignments/tests, etc. must be administered in a particular course, which may be a turn-off to some instructors

33. 33 Noncourse-Based Approach Campus-wide assessment activities focused on individual student or groups of students, not a particular course Campus-wide assessment activities focused on individual student or groups of students, not a particular course Assessments administered outside classroom to all students or a sample or students Assessments administered outside classroom to all students or a sample or students Assessment tools may be locally- developed (like JMU) or standardized; may include surveys Assessment tools may be locally- developed (like JMU) or standardized; may include surveys

34. 34 Noncourse-Based Approach Advantages Advantages Assessment removed from classroom, instructors are given more freedom Assessment removed from classroom, instructors are given more freedom Faculty collaboration encouraged Faculty collaboration encouraged Focuses on students’ experience in a “cluster” Focuses on students’ experience in a “cluster” Disadvantages Disadvantages Logistics of collecting data Logistics of collecting data Student motivation to take assessment seriously Student motivation to take assessment seriously Mechanisms need to be in place:1) to ensure that students are exposed to materials within courses that will enable them to meet objectives; 2) to match instrument to objectives Mechanisms need to be in place:1) to ensure that students are exposed to materials within courses that will enable them to meet objectives; 2) to match instrument to objectives

35. 35 Assessment Information on Internet James Madison University James Madison University http://www.jmu.edu/assessment/ http://www.jmu.edu/assessment/ University of North Texas University of North Texas http://www.unt.edu/vpaa/Assessment/Assessment_Home.html http://www.unt.edu/vpaa/Assessment/Assessment_Home.html http://www.unt.edu/vpaa/Assessment/Assessment_Home.html American Association of Higher Education American Association of Higher Education http://www.aahe.org/ http://www.aahe.org/ http://www.aahe.org/ National Center for Postsecondary Improvement National Center for Postsecondary Improvement http://www.stanford.edu/group/ncpi/index.html http://www.stanford.edu/group/ncpi/index.html http://www.stanford.edu/group/ncpi/index.html NC State: NC State: Internet Resources for Higher Education Outcomes Assessment http://www2.acs.ncsu.edu/UPA/assmt/resource.htm http://www2.acs.ncsu.edu/UPA/assmt/resource.htm