Learning Outcome and Key Performance Indicator 8/28/2018 Learning Outcome and Key Performance Indicator Dr. Sherwan Rahman Sulaiman MD MSc PhD College of Medicine Hawler Medical University 2016-2017

Workshop’s Learning Outcomes (LOs) Attendees at the end of this workshop will be able to: 1. Comprehend the nature and role of Program and Course learning outcomes in instruction. 2. Align an understanding relationships between Teaching Methods, Assessments Methods and LOs, 3. Write learning outcomes using the correct format. Summarize the role of learning outcomes in instruction and assessment.

Key Performance Indicators: Why? Performance of a higher education institution & its Programs is complex (teaching, research, community…) KPIs summarize performance in key areas  scientifically, rationally, and meaningfully for different stakeholders: Faculty & Staff Students External agencies (NCAAA, employers,…)

(maybe qualitative via rubrics) What are KPIs??? … KPIs = Key Performance Indicators …a measure of performance or achievement …a Key Success Indicator (KSI) …a measure of results and efficiency Quantifiable performance measures used to define success and measure progress toward the achievement of goals. (maybe qualitative via rubrics)

KPI KEY is fundamentally important to gain advantage; a make-or-break component for success. Performance when outcomes can be clearly measured, quantified, and easily influenced by the institution or Program. Indicator provides leading information on future performance. (ie., when the gas gauge is on “E” then the leading information tells us that the car’s future performance will be to stop; so the action plan is to get gas immediately)

NCAAA Learning & Teaching KPIs 5. Ratio of students to teaching staff (Based on full time equivalents) 6. Students overall rating on the quality of their courses. (Average rating of students on a five point scale on overall evaluation of courses.) 7. Proportion of teaching staff with verified doctoral qualifications. 8. Percentage of students entering Programs who successfully complete first year. 9. Proportion of students entering undergraduate Programs who complete those Programs in minimum time. 10. Proportion of students entering post graduate Programs who complete those Programs in specified time. 11. Proportion of graduates from undergraduate Programs who within six months of graduation are: employed, enrolled in further study, not seeking employment or further study

Measurable Indicators Goals and Objectives Major Strategies Measurable Indicators Analysis Strategic Goal 1: To enhance and expand Program facilities and infrastructure for research activities. Objective 1: (70 %) of the Program faculty conducting research or are involve in related by the end of 2014. Objective 2: ??? S1. Establish infrastructure for research. S2. Encourage collaborative research among faculty. S3. Provide appropriate funding for research and creative activities S4. Expand facilities for research KPI 1: Percentage of established research quality standards KPI 2: Number of publications (#/yr) in international reviewed journals KPI 3: Number of organized scientific/Research events (workshops / seminars & conferences) What do the findings mean? How are they applied? HOW? Strategic Goal 2: Objective 1: Objective 2: Strategic Goal 3: Quality Goal 1: Objective:

Introduction The Design phase of a typical curriculum development process (Analysis, Design, Development, Implementation, Evaluation) is largely concerned with developing clear learning objectives and learning outcomes. It is important to recognize during this phase that there is a direct relationship between objectives, learning outcomes, teaching strategy / methods & students assessments (Rubrics – KPIs with benchmarking and analysis).

5 Questions for Instructional Design Theory Into Practice 5 Questions for Instructional Design What do you want the student to be able to do? (Outcome) What does the student need to know in order to do this well? (Curriculum) What activity will facilitate the learning? (Pedagogy—learning & teaching) How will the student demonstrate the learning? (Assessment) How will the teacher know the student has done this well? (Criteria)

Learning Outcome are: Learning outcomes: describe what learners are supposed to know, understand, or are able to do at the end of the Program or course. LOs are based upon: • the needs of the learner (individual & personal). • the needs of society (mission statements). what the learner should know about a particular subject in order to perform successfully (career and personal lifestyle).

LOs are: What a STUDENT should learn as a result of a period of specified and supported study. The ACHIEVEMENTS of the learner rather then the intentions of the teacher. Focus is directly on Student Performance.

LOs are: Formal statements that articulate: What students know and are able to do after instruction Why students need to do this -- relevancy

LOs are: Must be measurable or observable Are concerned with the learning of the student: STUDENT… PERFORMANCE and ACHIEVEMENT What the student CAN DO What the student KNOWS AND CAN DO What the student UNDERSTANDS OR COMPREHENDS and CAN DO Must be measurable or observable

Objectives vs. LOs What is the difference? The distinction between learning outcomes and learning objectives is not universally recognized. Many instructors may find that the term “learning outcomes” describes what they have already understood by the term “learning objectives.” What is the difference?

For NCAAA the difference between course LOs and objectives… Learning objectives are statements of what the teacher intends for the students to learn and are generally part of a teacher-centered approach [are Mission, traditional, teacher or content driven]. Learning outcomes are statements of what the student will KNOW and be able to DO or demonstrate as a result of their learning and are part of a student-centered approach.

Objectives vs. LOs Learning objectives, for example, may outline the material the INSTRUCTOR intends to cover in the course / Program or the disciplinary questions the class will address. Known as IN-PUTS. By contrast, learning outcomes focus on what the STUDENTS know, comprehend and realistically are able to do… [skill performance] by the end of an assignment, activity, class, or course [achievement]. Known as OUT-PUTS.

Objectives vs. LOs learning outcomes, mean focusing on the application and integration of the course content from the perspective of the student. learning outcomes can more explicitly and directly address expectations for student learning.

Objectives ------- Outcomes Teacher-Centered Inputs Content-Centered Traditional Student-Centered Outputs  Results Performance & Achievement Assessments Measurable Observable What else overlaps?

Writing  Objectives & LOs Learning objectives can be written as teacher or curriculum centered content or they can be re-written as student-centered learning outcomes. The teacher will... OR the student will…. Both learning objectives and outcomes must be measurable or observable. One assessment is for teaching and another assessment is for a student’s learning.

Examples Example of a Learning Objective: Students will be taught the basic principles of database searching. [teacher will teach basic principles … ] Example of a Learning Outcome: Students will be able to apply the principles of database searching in a review of literature. [student will KNOW and APPLY…]

Objectives or Outcomes? Which Dentist do you want working on your teeth? Student A: The teacher will instruct the student to know how to successfully drill cavities and repair teeth… Student B: The student earns 100% on the exam for drilling cavities and repairing teeth…. Student C: The student knows how and successfully drills out cavities and repairs teeth… Student D: The teacher successfully taught the student to drill out cavities and repair teeth.

Objectives or Outcomes? Which Pharmacist do you want filling your meds? Student A: The teacher will instruct the students to know how to successfully fill medical prescriptions… Student B: The student earns 100% on the exam for filling medical prescriptions…. Student C: The student knows how and successfully fills medical prescriptions… Student D: The teacher successfully taught the student to fill medical prescriptions….

Exercise Please work as group in writing three learning objectives for your Program. Now, re-write these objectives as LOs Be prepared to share them and analyze the difference – they will be collected in order to be used latter.

The Importance of LOs 1. LOs build evidence for accountability, accreditation, and for continuous improvement. Show evidence of how well students learn. Use evidence for continuous improvement and strategic plans.

The Importance of LOs Know what you are doing… Know why you are doing it… Know what students are learning as a result; (key for assessment). Make improvement changes based on results (research based improvements)

The Importance of LOs Shifting from: Teachers teaching…. to students learning Teaching effectiveness…. to learning results

Course learning outcomes serve the following purposes… To inform students of what is expected of them. To guide the teacher in his/her approach to delivery of content and assessment that focuses on what the student will be able to do as a result of the learning. To influence the domain and level of learning required of the delivery and assessment. To fulfill the requirements of one or more Program outcomes.

Learning Outcomes help… 1. Select learning content objectives and skills (What to teach? Teaching content priorities?) 2. Development of instructional strategies that align with specific learning outcomes. 3. Develop and select instructional and Program materials that align with specific learning outcomes. Construct evaluation instruments for assessing student performance based on the learning. outcomes. Improve overall Program and as a faculty.

Benefits of Learning Outcomes Learning outcomes measure & characterize the values that an institution, Program, or course have articulated for student development & performance. A set of student learning outcomes define what students will know and be able to do when they have completed any degree, regardless of his/her major.

Benefits for Learning Outcomes Student learning outcomes will help guide faculty across the university to develop curricula, plan courses, determine financial needs, design syllabi, construct learning activities, and assess student learning. LOs provide a framework for learners and advisers in order to discuss the goals of the curriculum and the personal career goals for individual students.

Student Learning Outcomes Institutional Mission Framework for L.O. Employment Needs Student Needs Student Learning Outcomes Institutional Mission Assessment Methods Teaching Strategy Program Outcomes Teacher Objectives Course Outcomes

Benefit: OBE  “Outcome-Based Education” Outcome-based education is a method of teaching that focuses on what students know and can actually do after they are taught. All curriculum and teaching decisions are made based on how best to facilitate the desired outcome. The desired outcome is selected first and the curriculum is created to support the intended outcome. This leads to a planning process in reverse of traditional educational planning.

Learning Outcomes flows out from…… …the Mission Statement Where do L.O. come from? Learning Outcomes flows out from…… …the Mission Statement What must students do to demonstrate that the Institution and Program Mission Statements are accomplished?

Effects Program and Course Cause Mission Needs Learning Outcomes What will the student know and do in order to demonstrate the Mission is successfully achieved?

Learning Outcomes flows out from…… Where do L.O. come from? In addition to Knowledge & Cognitive Skills Domains, Learning Outcomes flows out from…… …student needs assessment … and employers needs (cause and effect chart)

Effects Learning Outcomes Cause Student Needs What will the student know and do in order to demonstrate student needs are successfully achieved?

Good LOs are… Usually written in the future tense Identify important learning requirements Are achievable Use clear language easily understandable to student When writing outcomes, it may be useful to use the following expression: “At the end of this Program or course the student should be able to…….” Then follow with a verb. Useful verbs are: ?????

Suggested Verbs Helpful? Need much more!! Establish Draft Provide Prepare Tabulate Write Schedule Update Audit Articulate Align Collect Construct Generate List Produce Compile Document Demonstrate Develop Helpful? Need much more!!

Verbs Not To Use Better? Yes… BUT!! Consider Maximize Maintain Reflect Continue Review Ensure Enlarge Understand Examine Strengthen Explore Encourage Deepen Some of these verbs can be used if tied to specific actions or quantification Better? Yes… BUT!!

NQF Learning Outcome Verbs NQF Learning Domains Suggested Verbs Knowledge list, name, record, define, label, outline, state, describe, recall, memorize, reproduce, recognize, record, tell, write Cognitive Skills estimate, explain, summarize, write, compare, contrast, diagram, subdivide, differentiate, criticize, calculate, analyze, compose, develop, create, prepare, reconstruct, reorganize, summarize, explain, predict, justify, rate, evaluate, plan, design, measure, judge, justify, interpret, appraise Interpersonal Skills & Responsibility demonstrate, judge, choose, illustrate, modify, show, use, appraise, evaluate, justify, analyze, question, and write Communication, Information Technology, Numerical demonstrate, calculate, illustrate, interpret, research, question, operate, appraise, evaluate, assess, and criticize Psychomotor demonstrate, show, illustrate, perform, dramatize, employ, manipulate, operate, prepare, produce, draw, diagram, examine, construct, assemble, experiment, and reconstruct

What is the major problem with this list?? Analyze this Example of Program LOs At the time of receiving a BSN Degree, students: Can identify, define, and solve problems; Can locate and critically evaluate information; Have mastered a body of knowledge and a mode of inquiry; Can understand diverse philosophies and cultures within and across societies; Can communicate effectively; Can understand the role of creativity, innovation, discovery, and expression across disciplines; and Have acquired skills for effective citizenship and life-long learning. What is the major problem with this list??

Specific Program LOs (Dentistry Examples) Graduates should demonstrate sound knowledge of the following areas as they relate to the practice of dentistry (includes a list of over 20 specific dentistry content areas). Graduates must have the ability to apply their knowledge and understanding of relevant principles and theories in carrying out the following types of responsibilities (e.g. problem recognition for disease identification and diagnosis, problem solving in dental care, critical thinking of dental research, patient investigation—biopsy techniques, radiography, treatment plans). Write 3 specific Program LOs for your Program by using the generic Program LOs shown on the previous slide. (present to group using flip chart + Analyze)

Question