Second day sessions First session Second session Third Session How to write LOs at Program and Course Levels Relationships between Teaching Methods and Assessments Methods and LOs KPIs in General and Application and Itegration with LOs 74

NCAAA How to write learning outcomes for program levels and courses? Session 1 75

(cause and effect chart) Where do L.O. come from? Learning Outcomes flows out from…… …students’ needs assessment … and employers needs (cause and effect chart) 76

Effects Learning Outcomes Cause Student Needs 77

(cause and effect chart) Where do L.O. come from? Learning Outcomes flows out from…… …the Mission Statement What must students do to demonstrate that the Institution/Program Mission is accomplished? (cause and effect chart) 78

Effects Program and Course Learning Outcomes Cause Mission Needs 79

Theory Into Practice 5 Questions for Instructional Design What do you want the student to be able to do? (Outcome) What does the student need to know in order to do this well? (Curriculum) What activity will facilitate the learning? (Pedagogy) How will the student demonstrate the learning? (Assessment) How will I know the student has done this well? (Criteria) 80

Let’s Write a Learning Outcome We’re taking a friend desert camping for the first time (not roughing it too much). What do they need to know? 81

1. What do you want the student to be able to do? This question asks you to develop the outcome. For Example: Student identifies, consults and evaluates reference books appropriate to the topic in order to locate background information and statistics. 82

Learning Outcomes Formula Verb or Action Phrase Great Learning Outcomes + = Why? “In Order To” Or What students need to know? “Student identifies, consults and evaluates reference books appropriate to the topic” Why do they need to know this? “locate background information and statistics.” In Order to 83

Suggested Verbs Establish Draft Provide Prepare Tabulate Write Schedule Update Audit Articulate Align Collect Construct Generate List Produce Compile Document Demonstrate Develop 84

Verbs Not To Use Consider Maximize Maintain Reflect Continue Review Ensure Enlarge Understand Examine Strengthen Explore Encourage Deepen Some of these verbs can be used if tied to specific actions or quantification 85

Example 1 Poor Learning Outcome: Students will name the three types of rock in order to differentiate among the three. Good Learning Outcome: Students will compare and contrast the characteristics of the three types of rocks in order to differentiate among the three. 86

Example 2 Bad Learning Outcomes: Good Learning Outcome: Discover that Al Fawo University offers a welcoming and helpful environment which can fulfill their educational, cultural and social needs in order to recognize the university’s role in lifelong learning. Use OVID in order to access materials not available at Al Fawo university Library. Good Learning Outcome: Utilize retrieval services in order to obtain materials not owned by Al Fawo University Library. 87

Let’s Write a Learning Outcome We’ll concentrate on how to build a tent Why do we want our friend to be able to properly build a tent? 88

Let’s Write a Learning Outcome Now let’s write the learning outcome What is our verb (use the right domain)? Why? 89

Let’s Write an Assignment-Specific Learning Outcome What does the student need to know? Why do they need to know this? See example and activity  90

Student needs Learning outcomes Student needs example: “Student needs to learn how to fish in the ocean to survive on the island.” What are some learning outcomes that fit this student’s needs? Create a student need for your specialized course. 3. Create five learning outcomes that are directly based on this need. Bubble Map 91

Student Need 92

NCAAA Relationship between teaching methods and assessment methods with learning outcomes. Session 2 93

Teaching Methods and LOs Focuses on learning outcomes in debates on teaching methods in higher education “Teaching methods” are not an end in themselves, they are a means to an end. They are the vehicle(s) we use to lead our students towards particular learning outcomes. We evaluate our teaching methods against the learning outcomes that we are seeking for our students. 94

Teaching Methods and LOs The first step in operationalizing it is to clarify the learning outcomes at which we are aiming. The second step involves developing a contingency approach to the choice of teaching methods whereby “fitness for purpose” 95

Teaching Methods and LOs When selecting any teaching and learning method it is important to ensure that the method will enable the students to achieve what are intended as learning outcomes. There are different kinds of methods available: effective in building up subject knowledge contribution to developing generic skills 96

Quality of Teaching NCAAA Standard 4, paragraph 4.6: Teaching must be of high quality with appropriate strategies used for different categories of learning outcomes and student learning styles. Differentiated Instruction 97

Teaching Methods Mapping Courses Codes 101 102 103 104 105 106 107 108 Teaching Methods Lecture Small Groups Discussion Project Activity Debate Research Lab Guest Expert Demonstration 98

Key for Learning Outcomes Key is ASSESSMENT Learning outcomes must be…. measurable and meaningful to be assessed accurately. Who will know?  Student? Faculty? How will I know?  Evidence What evidence is needed? Demonstrate 99

NCAAA: L.O. + Assessment NCAAA Standard 4, paragraph 4.4: Student assessment processes must be appropriate for the intended learning outcomes and effectively and fairly administered with independent verification of standards achieved. 100

Quality Assurance of Assessment of Learning Outcomes…. May be achieved: By direct observation – inspection of assessments before they are set, and of samples of marked work Indirectly – by examining the specification of assessment processes Via feedback – from students, from employers, from external examiner, from professional bodies. 101

Assessment Strategies… NCAAA Domains of Learning Strategies Knowledge Facts Concepts, theories Procedures Cognitive Skills Apply skills when asked Creative thinking and problem solving Interpersonal Skills and Responsibility Responsibility for own learning Group participation and leadership Act responsibly-personal and professional situations Ethical standards of behavior Communication IT and Numerical Skills Oral and written communication Use of IT Basic math and statistics Psychomotor Skills 102

Quality Assessment of L.O.s Requires That a range of assessment methods is used, so as to cover different LOs and different learning styles That specifications for assessment, and criteria for grading, are transparent and appropriate to the level of the course within the NQF That second-marking or moderation is routinely used That there is clarity about the grounds on which students may appeal a mark, and that students are aware of this 103

Quality Assessment of L.O.s Requires…. That deadlines for the return of marked work are published and adhered to That full and constructive feedback on assessments is provided to students, linked to the LOs and to the assessment criteria That guidelines for dealing with plagiarism, and with requests for time extensions and mitigating circumstances, are clear and are adhered to 104

Start at the End Teaching strategy and methods depend on the assessment methods utilized. The assessments utilized depends on the learning outcome VERB that guides the assessment process to validate student learning and direct teaching methods. Teaching Assessment Learning Objective Student Methods Methods Outcomes Content Needs 105

Assessments 1st Learning Outcome Methods Learning Content Objective 2nd LO 1st Learning Outcome Assessments Teaching- Methods Learning Content Objective 3rd LO 5th LO 4th LO 106

Introductory Assessment Map 107

Intermediate Assessment Map 108

Assessment Map 109

NCAAA Key Performance Indicators, Application and Integration with Learning Outcomes Session 3 110

What are KPIs??? …a type of measure of performance …a Key Success Indicator (KSI) …a measure of results and efficiency Quantifiable performance measures used to define success and measure progress toward the achievement of goals. 111

Success Indicators Bubble Map Group Activity …for a car …for a job (think Quality Indicators) …for a car …for a job …for dinner …for a business trip …for a house …for a vacation 112

Good KPIs are SMART Specific – clear and well-defined Measurable – show distance from attainment Agreed – shared by stakeholders Realistic – fit available time, money, etc Time-based – set a specific time-scale Sometimes ‘achievable’ is used, not ‘agreed’, but ‘achievable’ implies ‘realistic’, so you might use ‘achievable’ and ‘relevant’ 113

FIRST…… An intended learning outcome must be selected and then a KPI may be chosen which best reflects the intended student performance ... or intended student achievement … or the student outcome 114

They create a context for action KPIs are like…. …like yeast when it permeates bread and makes it rise… …like salt seasoning food, it changes the entire flavor.. …like light in the darkness.. They create a context for action 115

KPIs are not independent KPIs are “Married” to benchmarks KPIs do not stand alone or operate in a vacuum; a clear understanding of what is possible is required “upper and lower limits.” What are “limits”??? (ie., speed limits for cars) 116

“competitive analysis” Limits are… Limits refer to “BENCHMARKS” which are essential to make KPIs useful; both internal and external… They provide a basis for ANALYSIS…or as we see in the sporting business… “competitive analysis” 117

Sebring Racetrack Benchmarks Example: Daytona 500 Sebring 118

Benchmarks prevent race car crashes Benchmark Cones Control Speed Cars are forced to slow down in order to safely make the hairpin curve. How does this apply to teaching and learning? = 119

Benchmarking for quality: Why? Benchmarking fixes reference points relative to other measurements In the context of quality, benchmarking specifies reference values for KPIs Benchmarking is used to ensure that the KPIs are realistic Ensure KPIs are sufficiently demanding 120

Benchmarking for quality: What? Judgments about quality are based on comparisons with similar good institutions The levels of performance identified are benchmarks that a used for comparisons. Aggregate national figures for KPIs can also serve as benchmarks Should establish other benchmarks for matters of importance to them Remember: benchmarks should be both internal and external 121

KPI & Benchmarks (for Dr. Bothyna’s nursing program) Learning Outcome: The nursing student will use new Disposable Gloves at all times when nursing patients and whenever body fluids are present. KPI  The student uses new disposable gloves 100% of the time while nursing patients and whenever body fluids are present. Target Benchmark  100% Goal KPI Finding Benchmark  95% Assessment finding Internal Benchmark  98% Medical College External Benchmark  New Target Benchmark  98% Cairo University 100% New Goal Analysis: How is this data interpreted? What is the improvement plan to reach the new goal? 122

KPIs for learning outcomes Using a Bubble Map, write a course learning outcome for one of your courses in the center circle. Next write teaching methods for learning outcome in the connected circles. On the extended lines write an assessment method with a KPI and benchmarks. 123

Extra information that may Help 124

Program Learning Outcomes What is essential to this degree?? Why are students enrolling?? What does the public think?? What the STUDENT expects to know and/or do via the Program… 125

What the STUDENT will achieve and perform at the end of the program. Program Outcomes What the STUDENT will achieve and perform at the end of the program. These are general, specialized learning outcomes for all students in all departments of the college program. Institutional learning outcomes are general learning outcomes for all students. 126

Course Outcomes What the STUDENT will achieve and perform at the end of a specific course. These are highly specialized learning outcomes for all students in a given course. 127

Purposes of Program and Course Learning Outcomes Measurable Improvement is #1 Evaluate programs/courses, not personnel Assist with QA and accreditation… Initiate “best practices”… Set learning to KPIs & benchmarks… 128

Curriculum Alignment? Consistency and Intentionality Learning Objectives Curriculum Alignment is an iterative process involving systematic study (curriculum mapping, analysis, and interpretation) of curricular components to determine the degree of consistency between what faculty expect students to learn, what faculty think they teach, and what students learn as a result of their educational experiences. Curriculum Alignment ensures that Faculty teach what they claim they teach Student learn what they are supposed to learn English (1978) described the “fictional curriculum”. This is the declared curriculum – what it is assumed the student is learning. This may different from the ‘real” or taught curriculum – that is, the curriculum as it is delivered to the student. It may also be different from the “learned” or “tested” curriculum – what students actually learn. In a perfect, ideal world, these three circles will be perfectly aligned, that is there will be one circle. Curriculum Intentionality is a deliberate and systematic alignment of intended program learning outcomes with course-level pedagogies and instructional and learning activities. Three Intentionality questions: “Are we teaching what is being tested”? (opportunities to learn) “Is what we are teaching being tested?” (Content coverage) “Does assessment measure student progress in achieving intended learning outcomes?” (content validity) Harden, R.M. (2001). AMEE Guide No. 21. Curriculum mapping: a tool for transparent and authentic teaching and learning. Medical Teacher, 23 (2), 123-137. Hobson, E.H. (2005). Changing pedagogy. Presentation at SACS-COC Institute on Quality Enhancement and Accreditation, Orlando, FL, July 24-27, 2005. Learning Outcome 129

Examples Student Outcome Student will be Teaching Objective  teacher will teach & demonstrate 10 science lab safety rules Student Outcome Student will be able to identify 10 safety rules for the science lab in a paragraph. Student Outcome Student will demonstrate ability to use 10 conventions of safe science research in a science lab. What are the differences??? What are the different teaching methods? What are the different assessments? 130

Thank you for your time and reflections Dr Bothyna Zakaria Murshid Conclusion Thank you for your time and reflections Dr Bothyna Zakaria Murshid Dr Greg Maffet 131