1. Liberal Arts and Sciences Outcomes Assessment: Closing the loop or spiraling in the right direction? Dr. Jenn Berg Assistant Professor of Mathematics Dr. Christopher Cratsley Interim Director of Assessment Dr. Elizabeth Gordon Assistant Professor of Geo/Physical Sciences Dr. Joe Moser Assistant Professor of English

2. Closing the loop or spiraling in the right direction? Assess Student LearningUse data to improve instruction Assess Student Learning Use data to improve instruction & assessment

3. LA&S Assessment at FSU The Liberal Arts and Sciences (LA&S) assessment process carried out at Fitchburg State University from 2008-2010. The actions taken by the Fitchburg State University LA&S council in 2011 to reframe the objectives, reinterpret the role of LA&S courses, and revise the assessment system. Proposed Fitchburg State University LA&S course approval process that has emerged from the work of the LA&S council in 2011 including the rubrics utilized to assess the Fitchburg State University LA&S objectives.

4. LA&S Curriculum Objectives Aesthetic Appreciation Communication Problem Solving and Synthesizing Ethical Reasoning Citizenship

5. The Process of LA&S Assessment In the old days of 2008-10, we solicited student work as needed to match learning objectives. Faculty on the LA&S committee (and whomever else we could corral) performed assessment of a small sample of student work that matched a particular objective. Two to three faculty members scored each piece of student work on a 1-3 (insufficient, sufficient, or proficient) scale. Final assessment data stayed with the LA&S committee.

6. Example: Communication CriterionPerformance Rating InsufficientSufficientProficient Controlling idea Student work demonstrates little or no evidence of a theme or controlling idea. Student work demonstrates some evidence of a theme or controlling idea. Student work demonstrates a original and clear theme or controlling idea, skillfully limited to the dimensions of the assignment. Organization and Development Student work does not demonstrate empathy with the audience, which must work to extract meaning and organize the ideas in the student's communication effort. Students work evidences one or more supporting points related to the controlling idea and developed in a manner which anticipates and answers most audience questions concerning the theme or controlling idea. Student work evidences multiple points supporting the theme or controlling idea and developed in a manner which not only anticipates likely audience questions but poses and provides satisfactory answers to new questions. Standards of evidence "Evidence," in this student work, is limited primarily to personal assertions and/or challenges and responses which are emotional rather than rational in character. Student work is developed using typical reference sources which may be reliable and which seem to be represented fairly and in context. Documentation of sources appears to be substantially correct. Student work is developed using scholarly and unimpeachably reliable sources which are scrupulously documented and presented fairly and in context. Mechanics / Presentation Student work contains major and distracting errors in language and/or lapses in presentation. Student work has few distracting errors and/or lapses in presentation. Student work skillfully uses language which is free from distracting errors and other communication skills in order to enhance communication.

7. Communication results Spring 2010 – Nine student artifacts evaluated Fall 2010 – Eleven student artifacts evaluated; Honors course

8. Example: Problem solving Work is correct Work is organized Work is complete Uses formulas properly, where and when appropriate Creates graphs, tables and/or statistics to summarize data Gives clear, precise and relevant explanations, identifies causes and proposes solutions Uses appropriate methodology to collect data Integrates information from outside sources Uses numerical data to defend argument(s) Explains patterns or trends in observations, data, graphs and/or tables Identifies pros and cons of argument(s), including biases and/or limitations Analyzes outcomes from multiple perspectives Applies content knowledge, methods and/or results to new situations

9. Problem solving results Separation of math and science rubrics

10. Problem solving results Separation of math and science rubrics Many unused criteria

11. The Problems of LA&S Assessment Our sample sizes tended to be small. In some cases, due to a dearth of available student work and time constraints, the faculty performing the assessment did not have the assignments to which the pieces of student work were adhering. Some of our rubrics were either too vague or too detailed and thus hard to apply. Faculty buy-in was difficult to achieve, which undermined the assessment process, which led to further difficulty with faculty buy-in, which further undermined the process, which led to...

12. Questions? Assess Student Learning Use data to improve instruction & assessment

13. January 2011

14. The Vision Project Key Outcomes COLLEGE PARTICIPATION College-going rates of high school graduates COLLEGE COMPLETION Graduation and success rates of the students we enroll STUDENT LEARNING Academic achievements by our students on campus- level and national assessments of learning WORKFORCE ALIGNMENT Alignment of our degree programs with key areas of workforce need in the state's economy ELIMINATION OF DISPARITIES Achievement of comparable outcomes among different ethnic/racial, economic and gender groups

15. Implementation Plan Seek to become a LEAP State and establish a Massachusetts LEAP Team. Develop a Resource Plan that outlines direct and indirect resources needed to achieve the goals of the Vision Project. Engage campus provosts to provide leadership for faculty engagement and campus assessment activity including LEAP-related analysis. Establish a network of faculty and professional staff assessment leaders to engage in collaborative activities including determining system-wide learning outcome measures within LEAP framework.

16. Assessment Afternoons 4/6 LEAP Learning Outcomes and the Fitchburg State University LA&S Curriculum 4/13 Assessing Art Appreciation 4/20 Assessing Communication Skills 4/27 Assessing Problem Solving 5/4 Assessing Ethical Reasoning 5/11 Assessing Citizenship 5/23 LA&S AND YOU: Linking Liberal Arts and Sciences Assessment to what is happening in the departments and courses.

17. LEAP Essential Outcomes Knowledge of Human Cultures and the Physical and Natural World Through study in the sciences and mathematics, social sciences, humanities, histories, languages, and the arts Intellectual and Practical Skills, including Inquiry and analysis Critical and creative thinking Written and oral communication Quantitative literacy Information literacy Teamwork and problem solving Personal and Social Responsibility, including Civic knowledge and engagement—local and global Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative and Applied Learning, including Synthesis and advanced accomplishment across general and specialized studies

18. Assessment afternoons Who came? What were our goals? What actually happened?

19. LA&S Curriculum Objectives Aesthetic Appreciation ◦ Aesthetic Appreciation and Aesthetic Expression Communication ◦ Oral Communication ◦ Written Communication Problem Solving and Synthesizing ◦ Problem Solving through Quantitative Literacy ◦ Problem Solving through Inquiry and Data Analysis Ethical Reasoning ◦ Ethical Reasoning when Making Choices Citizenship ◦ Citizenship through Critical Analysis of Events

20. The Arts rubric DeficientSufficientProficient Artistic analysisStudent's work demonstrates a superficial, impressionistic analysis of artistic expression. Student's work demonstrates a basic conceptual analysis of artistic expression. Student's work demonstrates a sophisticated, multi- dimensional analysis of artistic expression. Art in contextStudent's work fails to engage art's historical, cultural, psychological, conventional, or other contexts. Student's work demonstrates some awareness of art's historical, cultural, psychological, conventional, or other contexts. Student's work engages and synthesizes art's historical, cultural, psychological, conventional, or other contexts.

21. The Arts rubric revision DeficientSufficientProficient Artistic analysisStudent's work demonstrates a superficial, impressionistic analysis of artistic expression. Student's work demonstrates a basic conceptual analysis of artistic expression. Student's work demonstrates a sophisticated, multi- dimensional analysis of artistic expression. Art in contextStudent's work fails to engage art's historical, cultural, psychological, conventional, or other contexts. Student's work demonstrates some awareness of art's historical, cultural, psychological, conventional, or other contexts. Student's work engages and synthesizes art's historical, cultural, psychological, conventional, or other contexts. Composition/ Performance Student’s work displays no coherence or elegance of composition or performance. Student’s work displays basic coherence and elegance of composition or performance. Student’s work displays sophisticated coherence and elegance of composition or performance.

22. Communication rubric revisions Students will speak, read, write and listen to create and understand meanings using a variety of media. They will recognize how to participate in or lead groups to accomplish goals. Written communication Controlling idea Organization and Development Sources and Evidence Academic Discourse Mechanics/Presentation Oral communication Controlling idea Organization and Development Supporting Material Oral and Physical Delivery Mechanics/Presentation

23. Problem-Solving rubric revisions Among these strategies, students will analyze and interpret data as a means to evaluate arguments and make informed choices. Topics Selection Information from sources Appropriate methodology Creates graphs, tables, etc. Explains patterns or trends Numerical data to defend Identifies pros and cons Applies content knowledge Form problem-solving strategies and evaluate their effectiveness. Work correct and complete Uses formulas properly Creates graphs, tables, etc. Explains patterns or trends Gives clear explanations Applies content knowledge

24. CriterionPROBLEM SOLVING RUBRIC: Performance Rating DeficientSufficientProficient Work is correct. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Work is organized. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Work is complete. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Uses formulas properly, where and when appropriate. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Creates graphs, tables and/or statistics to summarize data. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Gives clear, precise and relevant explanations, identifies causes and proposes solutions. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Uses appropriate methodology to collect data. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Integrates information from outside sources. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Uses numerical data to defend argument(s). Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Explains patterns or trends in observations, data, graphs and/or tables. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion. Identifies pros and cons of argument(s), including biases and/or limitations. Does not meet minimum expectation for the said criterion. Meets minimum expectations for the said criterion. Exceeds minimum expectations for the said criterion.

25. Proficient 3 Sufficient 2 Deficient 1 Work is correct and complete. Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. Calculations are also presented elegantly (clearly, concisely, etc.) Calculations attempted are either unsuccessful or represent only a portion of the calculations required to comprehensively solve the problem. Calculations are attempted but are both unsuccessful and are not comprehensive. Uses formulas properly, where and when appropriate. Uses formulas correctly and appropriately and with analysis/explanation. Uses formulas correctly and appropriately but without analysis/ explanation. Uses formulas incorrectly OR in inappropriate places. Creates graphs, tables and/or statistics to summarize data. Skillfully converts relevant information into an insightful mathematical portrayal in a way that contributes to a further or deeper understanding. Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate. Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate. Explains patterns or trends in observations, data, graphs and/or tables. Provides accurate explanations of information presented in mathematical forms. Makes appropriate inferences based on that information. For example, accurately explains the trend data shown in a graph and makes reasonable predictions regarding what the data suggest about future events. Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line. Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means. For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends. Gives clear, precise and relevant explanations Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work. Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work. Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work. Applies content knowledge, methods and/or results to new situations. Insightfully discusses in detail relevant and supported limitations and implications of content knowledge, methods and results. Presents basic, relevant and supported limitations or implications of content knowledge, methods and/or results. Any presented limitations and implications are possibly irrelevant and unsupported.

26. Ethical Reasoning rubric revision Students will recognize the ethical issues involved in human actions and be able to formulate a set of principles and virtues which can be brought to bear in personal and public decision making. Moral Reasoning Statement of Position Ethical Issue Recognition Application of Ethical Perspectives/Concepts Development of Rationale Ethical Self-Awareness Evaluation of Different Ethical Perspectives/Concepts

27. Citizenship rubric revision Students will articulate the relationships among local, national and global concerns, interests and needs. They will recognize possibilities and opportunities to enact positive change on an individual or group level. Explanation of Event Evidence Student’s Position Diversity of Communities and Cultures Analysis of Knowledge

28. Questions? Assess Student Learning Use data to improve instruction & assessment

29. NEEAN Summer Institute Through facilitated dialogue, participants will: 1. Explore options for institutionalizing assessment to improve student learning; 2. Learn how to improve the reporting process; 3. Consider ways to enhance faculty development and target resources to continuous improvement; 4. Become familiar with helpful tools and methods to assess and improve student learning; and 5. Discuss how to best analyze data.

30. Affecting change through governance What are our goals: ◦ Increase faculty awareness/involvement  Increase the variety of data we collect  Increase faculty input on the process ◦ Make LA&S-ness primary in course design

31. Means of effecting change Alter the course approval process to ask for: ◦ Which objective(s) the course addresses ◦ Description of an assignment (aligned with an LA&S rubric) whose student work could be submitted to the LA&S council ◦ Option to indicate need for new rubric

32. Pros and cons of this idea Pros: ◦ Achieves many of our goals  Increase faculty awareness/involvement  Increase the variety of data we collect  Increase faculty input on the process  Make LA&S-ness primary in course design Cons ◦ Expect faculty resistance

33. LA&S Assessment Cover Sheets Please indicate yes or no for each of these boxes and then make a final yes/no judgment on whether scorers should assess this criterion. Faculty member or department name _________________ Course #__________Course Name _________________ Assignment name __________# students in course ______ Date ___ % of grade □ 20% Aesthetic Appreciation through Aesthetic Expression Criteria: See rubric on back for details This criterion is... Taught as part of course instruction Addressed in assignment prompt Appropriate to be assessed Aesthetic analysis Work in context Composition/Performance

34. Final Summary Slide Assessing student work has value even when the process is imperfect because the data allows us to improve our assessment methods. A meaningful assessment system has to take into account what faculty are actually doing with their students in the classroom. Assessment may be more meaningful and effective if it is integrated into the process of course design and development.

35. Questions? Assess Student Learning Use data to improve instruction & assessment

36. Thanks to: The Fitchburg State University LA&S Council The Davis Foundation New England Educational Assessment Network Dr. Ben Railton and Dr. Eric Budd past chairs of the LA&S Council Fitchburg State University Office of Academic Affairs