1. Course Embedded Assessment of Graduate Learning: An Innovative Approach Based on The Degree Qualifications Profile Presented by William M. White, Ed. D. and Indre Cuckler, M. A. School of Graduate Studies

2. Who We Are

6. Outline  Levels of Assessment  Definition of Assessment  Theoretical Framework  Program Goals and Outcomes  Competency-Based Assessment  Course Embedded Assessment Model  Assessment Instruments  Curriculum Mapping  Resources  Questions/Discussion

7. Levels of Assessment Levels of Assessment: Level 1. Assessing individual student learning within courses Level 2. Assessing individual student learning across courses Level 3. Assessing courses Level 4. Assessing programs Level 5. Assessing the institution (Miller & Leskes, 2005)

8. Definition of Assessment Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development. (Palomba & Banta,1999)

9. Assessment Cycle Identify learning goals and outcomes Select assessment instruments & sources of evidence Gather and analyze data Implement action (s) based on the results Establish criteria for success Report assessment results

10. Theoretical Framework Theoretical Framework for MSU’s School of Graduate Studies Assessment based on: Course Embedded Assessment model The Lumina Foundation’s Degree Qualifications Profile (DQP) The Degree Qualifications Profile model allows our graduate programs to frame clear expectations of what students should be expected to know and be able to do at the associate, bachelor’s, and master’s level regardless of their academic discipline.

11. The Degree Qualifications Profile (2011) Five Areas of Learning: 1.Specialized Knowledge 2.Broad, Integrative Knowledge 3.Intellectual Skills  Analytic Inquiry  Use of Information Resources  Engaging Diverse Perspectives  Quantitative Fluency  Communication Fluency 4. Applied Learning 5. Civic Learning

12. Graduate Level Learning How do I know graduate level learning when I see it? What are its characteristics? The Degree Profile provides guidance to the above questions. Example: Analytic inquiry At the associate level, the student Identifies, categorizes and distinguishes among elements of ideas, concepts, theories and/or practical approaches to standard problems. At the bachelor’s level, the student Differentiates and evaluates theories and approaches to complex standard and non-standard problems within his or her major field and at least one other academic field. At the master’s level, the student Disaggregates, adapts, reformulates and employs principal ideas, techniques or methods at the forefront of his or her field of study in the context of an essay or project.

13. Programs at MSU’s School of Graduate Studies Mission The School of Graduate Studies believes in the transformative power of lifelong inquiry to enhance lives. We seek to develop and maintain innovative graduate programs that foster opportunities for personal and professional success in an increasingly multidisciplinary world. Master of Health Science (GHSC), Master of Arts or Science in Interdisciplinary Studies (GIDS), Master of Criminal Justice Administration (GCJA), and Master of Arts in Psychology (GPSY)

14. Program Goals Student Learning Outcomes (program level) Student Learning Outcomes (program level) G1: Provide an affordable and high quality graduate health science program. G2: Foster student achievement of core competencies appropriate for health science professionals. G3: Prepare students to serve the community and become leaders of change. G4: Prepare graduates for careers in health administration and health services related areas. 1. Communication Skills: Students will be able to demonstrate the ability to create logical oral and written arguments, explanations and reflections based on their individual or collaborative efforts to both general and specialized audiences. 2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ core disciplinary concepts, techniques or methods at the forefront of the health science field in the context of an essay, a thesis or a project. 3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity: Students will demonstrate knowledge, sensitivity and awareness of individual and cultural diversity demonstrated through a project, paper or performance. 4. Values and Ethics: Students will assess and justify their engagement with respect to civic, social and global responsibilities. 5. Application of Knowledge and Skills in Health Science Professional Settings: Students will be able to create a project, paper, exhibit, performance or other appropriate artifacts reflecting the integration of knowledge acquired in a practicum, work community and /or research activities with knowledge and /or skills gained from at least two academic disciplines in different segments of the curriculum. 6. Qualitative and Mixed Methods Fluency: Students will demonstrate knowledge and ability to interpret, evaluate design and implement qualitative and mix-methods research appropriate to health science studies in a formal research document or by performance a project. Note: student learning outcomes (1,2,&5) are adopted from the Lumina Foundation’s Degree Qualifications Profile Master of Health Science (GHSC)

15. Program Goals Student Learning Outcomes (program level) Student Learning Outcomes (program level) G1: Provide an affordable and high quality interdisciplinary studies program. G2: Foster student achievement of core competencies appropriate for graduate level learners and professionals. G3: Prepare students to serve the community and become leaders of change. G4: Prepare graduates for careers in interdisciplinary fields according to students’ academic concentrations. 1. Communication Skills: Students will be able to demonstrate the ability to create logical oral and written arguments, explanations and reflections based on their individual or collaborative efforts to both general and specialized audiences. 2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ core disciplinary concepts, techniques or methods at the forefront of their field of study in the context of an essay, a thesis or a project. 3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity: Students will demonstrate knowledge, sensitivity and awareness of individual and cultural diversity demonstrated through a project, paper or performance. 4. Broad, Integrative Knowledge: Students will articulate how their field (academic concentration) has developed in relation to other major domains of inquiry and/or practice. 5. Specialized Knowledge in an Academic Concentration Area: Students will be able to elucidate major theories, research methods, and approaches to inquiry and/or schools of practice in their field (academic concentration). 6. Research and Evaluation in Academic Concentration Area: Graduates will be able to utilize the scientific approach to knowledge generation and apply the appropriate research methods to practice. Note: student learning outcomes (1,2,&5) are adopted from the Lumina Foundation’s Degree Qualifications Profile Master of Arts or Science in Interdisciplinary Studies (GIDS)

16. Program Goals Student Learning Outcomes (program level) Student Learning Outcomes (program level) G1: Provide an affordable and high quality graduate criminal justice administration program. G2: Foster student achievement of core competencies appropriate for criminal justice administration professionals. G3: Prepare students to serve the community and become leaders of change. G4: Prepare graduates for careers in criminal justice administration field and related areas. 1. Communication Skills: Students will be able to demonstrate the ability to create logical oral and written arguments, explanations and reflections based on their individual or collaborative efforts to both general and specialized audiences. 2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ core disciplinary concepts, techniques or methods at the forefront of their field of study in the context of an essay, a thesis or a project. 3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity: Students will demonstrate knowledge, sensitivity and awareness of individual and cultural diversity demonstrated through a project, paper or performance. 4. Broad, Integrative Knowledge: Students will articulate how criminal justice administration field has developed in relation to other major domains of inquiry and/or practice. 5. Engaging Diverse Perspectives: Students will address core issues in criminal justice administration from the perspective of either a different point in time, culture, language, political order, or technological context. 6. Application of Knowledge and Skills in Criminal Justice Administration Professional Settings: Students will be able to create a project, paper, exhibit, performance or other appropriate means reflecting the integration of knowledge acquired in a practicum, work community and /or research activities with knowledge and /or skills gained from at least two academic disciplines in different segments of the curriculum. Note: student learning outcomes (1,2, 4,&6)are adopted from the Lumina Foundation’s Degree Qualifications Profile Master of Criminal Justice Administration (GCJA)

17. Program Goals Student Learning Outcomes (program level) Student Learning Outcomes (program level) G1: Provide an affordable and high quality graduate psychology program. G2: Foster student achievement of core competencies appropriate for mental health and human services professionals. G3: Prepare students to serve the community and become leaders of change. G4: Prepare graduates for careers in mental health, human services, and related areas. G5: Prepare students for doctoral and post graduate studies in psychology. 1. Communication Skills: Students will be able to demonstrate the ability to create logical oral and written arguments, explanations and reflections based on their individual or collaborative efforts to both general and specialized audiences. 2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ core disciplinary concepts, techniques or methods at the forefront of their field of study in the context of an essay, a thesis or a project. 3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity: Students will demonstrate knowledge, sensitivity and awareness of individual and cultural diversity demonstrated through a project, paper or performance. 4. Information Literacy: Students will be able to provide evidence (through papers, projects, notebooks, computer files or catalogues) contributing to, expanding, assessing and/or refining either a broadly recognized information resource or an information base within the field of psychology. 5. Research and Evaluation in Psychology: Graduates will be able to utilize the scientific approach to knowledge generation in psychology and apply the appropriate research methods to practice. 6. Quantitative Fluency: Students will apply mathematical, formal logic and/or statistical tools to problems appropriate to the field of psychology in a project, paper or performance. Note: student learning outcomes (1,2,4, & 6) are adopted from the Lumina Foundation’s Degree Qualifications Profile Master of Arts in Psychology (GPSY)

18. Competency-Based Assessment Competency - based assessment measures an individual’s performance against a predetermined criterion. There are three levels of competency achievement throughout a program of study: competency INTRODUCED (I) competency PRACTICED (P) competency REFINED (R) Note: the competency achievement expectations are established by faculty in each course.

19. Competency-Based Assessment Chart Yes NoYes No Review courses and revise curriculum Evaluate student work at the beginning of a program in the pre-selected courses Review courses and revise curriculum Write summary report Are competencies PRACTICED (P) in a course? Review courses and revise curriculum No Are competencies INTRODUCED (I) in a course? Yes Evaluate student work in the middle of a program in the pre-selected courses Are competencies REFINED (R) in a course? Evaluate student work toward the end of a program in the pre-selected courses Are competencies achieved? (Does the program do what is intended?)

20. Curricular Alignment Figure 1. Relationship among standards/objectives, instructional activities and materials, and assessments/tests. Adopted from Anderson (2002) Curricular Alignment: A Re-Examination. A signature assignment to measure achievement of student learning is required in courses where assessment data are being collected for program level assessment. Course Embedded Assessment Standards/Objectives (MSU: Core Competencies) Assessments/Tests (MSU: Rubrics) Instructional Activities/ Materials (MSU: Assignments)

21. Learning Outcome Assessment (2011-2012) Three student learning outcomes (SLOs) are assessed in 2011-2012 cycle in all four graduate programs: 1. Communication Skills 2. Critical Thinking 3. Cultural & Social Awareness (Note: other student learning outcomes will be assessed in 2012-2013) How are these outcomes assessed? By utilizing direct measures: Written Assignments, and Theses or Projects and indirect measures: Student Reflection Journals Course Evaluations

22. Assessment Instruments Rubrics are utilized to assess learning achievement.  Written Communication Skills Rubric  Critical Thinking Rubric  Cultural and Social Awareness Rubric *Note: rubrics must meet graduate level learning expectations.

23. Student Learning Outcomes Program level student learning outcomes must be reflected in individual courses. Review your program’s outcomes and align them with your course outcomes. All syllabi must include the following statement— “Upon successful completion of this course, students will have the following competencies…” [list competencies]

24. Student Learning Outcomes (example) IMPORTANT: Student learning outcomes have to be measurable! Upon successful completion of this course, students will have the following competencies: Students will be able to critically evaluate cultural and socioeconomic differences and the significance of these differences for personality development as evident in a formal research paper. Cultural and Social Awareness (program level) Upon successful completion of this course, students will have the following competencies…. (Give example) Critical Thinking (program level) course level

25. Curriculum Mapping Curriculum maps  Shows when, what, and on what level learning is delivered throughout the program. Why curriculum mapping is important?  Provides clarity and transparency of what is going on in the program and in specific courses.  Curriculum mapping is essential to program and course level assessment.  Program level student learning outcomes should be closely aligned with course level student learning outcomes and this alignment should be depicted in a curriculum map.

26. Core Courses GPSY 523 Profession al Ethics GPSY 545 History & Systems GPSY 563 Research Methods GPSY 505 Statistics GPSY 517 Adv. Develop. Psychology GPSY 527 Social Psychology GPSY 520 Physiological Psychology GPSY 525 Cognitive Psychology GPSY 610 Adv. Principles of Learning GPSY 620 Psychopathol ogy GPSY 630 Psychologic al Testing GPSY 670 Portfolio GPSY 680 Thesis or Project Program Outcomes 1.Communication Skills IAIA PP PAPA PP P P R RARA 2.Critical Thinking IAIA P PP PAPA PP P P R RARA 3.Information Literacy IAIA PPPP PAPA PP P P R RARA 4..Quantitative Fluency IAIA PAPA R RARA 5.Knowledge sensitivity, awareness of individual and cultural diversity IAIA P PAPA P R RARA 6.Research and evaluation in psychology IAIA PPPAPA PPPRARA Master of Arts in Psychology Program (GPSY) Curriculum Map Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome; A- data are collected and analyzed for program level assessment. Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.

27. Core Courses GHSC 501 Focus & Design GHSC 563 Research Methods GHSC 616 Health Care Systems GHCS 621 Health Care Financial Mngm GHSC 622 The Health Care Executive GHSC 503 Behavior in Organizations GHSC 504 Transforming Organizational Cultures GHSC 580 Concentrat. area and field experience GHSC 581 Concentration area and field experience GHSC 582 Concentrat. area and field experience GHSC 583 Concentratio area and field experience GCJA 670 Portfolio GCJA680 Thesis or Project Program Outcomes 1.Communication Skills IAIA PPPP PAPA PPPPR RARA 2.Critical Thinking IAIA PPPPP PAPA R RARA 3.Application of HSC IAIA P A PPR RARA 4.Values & Ethics IAIA PAPA PRR RARA 5.Knowledge sensitivity, and awareness of individual and cultural diversity IAIA PAPA P PPPR RARA 6.Qualitative and Mixed Methods Fluency IAIA PAPA RRARA Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome; A- data are collected and analyzed for program level assessment. Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment. Master of Health Science Program (GHSC) Curriculum Map (Concentration: Health Care Executive)

28. Core Courses GHSC 501 Focus & Design GHSC 563 Research Methods GHSC 511 Multidsp. Aspects of Aging GHCS 515 Psyc. Aspects of Aging GHSC 516 Soc. Aspects of Aging GHSC 572 Working Across Disabilities GHSC 575 Public Policy GHSC 580 Concentrat. area and field experience GHSC 581 Concentrat. area and field experience GHSC 582 Concentrat area and field experience GHSC 583 Concentrat area and field experience GCJA 670 Portfolio GCJA680 Thesis or Project Program Outcomes 1.Communica tion Skills IAIA PPP PAPA PR RARA 2.Critical Thinking IAIA PPPP PAPA PR RARA 3.Application of HSC IAIA I PAPA PPPR RARA 4.Values & Ethics IAIA PAPA PPPR RARA 5.Knowledge sensitivity, and awareness of individual and cultural diversity IAIA PAPA PPPR RARA 6.Qualitative and Mixed Methods Research Fluency IAIA PAPA RRARA Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome; A- data are collected and analyzed for program level assessment. Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment. Master of Health Science Program (GHSC) Curriculum Map (Concentration: Aging Studies)

29. Core Courses GIDS 501 Focus & Design GIDS 550 Professional Writing GIDS 557 Interdisciplinary Context and Theory GIDS 563 Research Methods GIDS 567 Statistical Methods GIDS 580 Concentration area GIDS 581 Concentration area GIDS 582 Concentration area GIDS 583 Concentration area GIDS 670 Portfolio GIDS 680 Thesis or Project Program Outcomes 1.Communication Skills IAIA P P A PPPR RARA 2.Critical Thinking IAIA PPP PAPA PPPR R A Specialized 3.knowledge in an academic concentration IAIA PAPA PP RR RARA 4.Broad Integrative Knowledge (interdisciplinary) IAIA PAPA R RARA 5. Knowledge sensitivity, and awareness of individual and cultural diversity IAIA PAPA PPPR RARA 6.Research and evaluation in an academic concentration IAIA PAPA PPPRRARA Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome; A- data are collected and analyzed for program level assessment. Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment. MA/MS Interdisciplinary Studies Program (GIDS) Curriculum Map

30. Core Courses GCJA 501 Focus & Design GCJA 531 History of Criminal Justice GCJA 563 Research Methods GCJA 580 Concentration area GCJA 581 Concentration area GCJA 582 Concentration area GCJA 583 Concentration area GCJA 584 Concentration area and field research GCJA 585 Concentration area and field research GCJA 670 Portfolio GCJA680 Thesis or Project Program Outcomes 1.Communication Skills IAIA PP PAPA PPPR RARA 2.Critical Thinking IAIA P PAPA PPPR RARA 3. Engaging Diverse Perspectives IAIA PAPA PPPR RARA 4.Broad Integrative Knowledge IAIA PAPA PPPR RARA 5.Application of CJA IPIP PAPA P PR RARA 6. Knowledge sensitivity, and awareness of individual and cultural diversity IAIA PAPA PPP RRARA Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome; A- data are collected and analyzed for program level assessment. Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment. Master of Criminal Justice Administration Program (GCJA) Curriculum Map

31. Summary “ By acting on the calls for reform in assessment, graduate education can truly lead the way in fulfilling its mission of excellence in teaching, research, and service to its students and to the broader community.” (Lydell, 2008)

32. References Association of American Colleges and Universities (AAC&U). Value Rubrics. Retrieved from http://www.aacu.org/value/rubrics/index.cfm Anderson, L.W. (2002). Curricular alignment: A re-examination. Theory into Practice, 41, (4). Retrieved from Academic Search Premier. Harden, R. M. (2001). AMEE Guide No 21: Curriculum mapping: a tool for transparent and authentic teaching and learning. Med Tech, 23, (2). Lydell, L. (2008). Assessing outcomes in graduate education. On the Horizon, 16, (2). Retrieved from Proquest data base. Lumina Foundation. (2011). The Degree Qualifications Profile. Retrieved from http://www.luminafoundation.org/publications/The_Degree_Qualifications_Profile.pdf Miller, R., & Leskes, A. (2005). Levels of assessment: From the student to the institution. Washington, DC: AAC&U. Retrieved from http://www.aacu.org/pdf/LevelsOfAssessment.pdf

33. Questions/Discussion For more information please contact: Dr. William M. White, Dean Office: (304) 929-1438; Email: wwhite@mountainstate.edu School of Graduate Studies or Indre Cuckler, M. A. Psychology Program Specialist Office: (304)-929-1401; Email: icuckler@mountainstate.edu School of Graduate Studies Thank you !