Training for PEC Program Evaluators

Training for PEC Program Evaluators
Islamabad, Pakistan 5-7 November 2014 Prof. Megat Johari Megat Mohd Noor Director of Quality & Risk Management, Universiti Teknologi Malaysia (UTM) Kuala Lumpur Head of Ecolgical Engineering Research Laboratory Malaysia Japan International Institute of Technology Universiti Teknologi Malaysia, Kuala Lumpur

Program Day 1 Time Item Facilitator 09.00 10.00
Introduction and Accreditation Plan Megat Johari 10.30 Tea break 11.00 Opening Session and Program Educational Objectives (1) 13.00 Lunch break 14.00 Program Outcomes (2) 15.30 16.00 Curriculum & Learning Process (3) 17.00 End

Program Day 2 Time Item Facilitator 09.00 Students (4) and Faculty (5)
Megat Johari 10.30 Tea break 11.00 Infrastructure & Facilities (6) and Institutional Support & Financial Resources (7) 13.00 Lunch break 14.00 Continual Quality Improvement (8), and Industrial Linkages (9) 15.30 16.00 Judgment and Exit Meeting 17.00 End

Objectives Plan for accreditation
Conduct the Opening and Exit meetings Probe in-depth on Objectives and Outcomes Conduct the accreditation sessions with aplomb and decorum Aware of the PEC Manual 2014’s requirements Make sound judgment

Program Day 1 Time Item Facilitator 10.00
Introduction and Accreditation Plan Megat Johari

COMPETENCY OF EVALUATORS
Organizing skills Knowledge of the manual Questioning skills Comprehensiveness of the evaluation Listening to persons Overall appearances Reporting Overall judgment Overall rapport with persons Aplomb (self-confidence) and decorum (etiquette)

Expectations of Accreditation
Education content and level are maintained Programme Continual Quality Improvement (CQI) Outcome-based Education (OBE) Programme Systematic (QMS)

Broadly Defined Problems
Depth of Knowledge Required Complex Problems Broadly Defined Problems Well defined Problems Requires in-depth knowledge that allows a fundamentals-based first principles analytical approach Requires knowledge of principles and applied procedures or methodologies Can be solved using limited theoretical knowledge, but normally requires extensive practical knowledge

Continual Quality Improvement (8)
Program Objectives (1) & Outcomes (2) Students (4) Faculty (5) Institutional Support & Financial Resources (7) Infrastructures & Facilities (6) Industry Linkages (9) Curriculum & Learning Process (3) Continual Quality Improvement (8)

Plan for Accreditation
Manual SAR Previous accreditation report Checklist Schedule

Discuss findings (interim report) Strategy (schedule & assignment)
Pre‐Visit Meeting Meet at least once (in addition to the meeting on Day ‐1) before the Accreditation Visit, to study and discuss documents, and systematically identify shortcomings. Strategically plan and/or request supplementary input from the University to fill the gaps. (Update interim report, checklist, schedule and assignment) Further information required, communicate through EAD, PEC. Discuss findings (interim report) Strategy (schedule & assignment) Update checklist

PEC Manual 2014 Accreditation Team’s Activities (p11)
Pre-visit meeting On-site visit starts with a meeting with Head of the Institution (Opening Meeting) Presentation by the program head Visit to Laboratories, Libraries, Workshops, and other Infrastructure/Facilities Meeting with administrative staff, faculty, students and other stakeholders such as alumni, employers, parents. Exit meeting with the Management, Principal/Deans to briefly share the strengths and weaknesses of the program. Discussion among team members followed by compilation of Visit Report and submission to EAB. On-site Visit

On-site Visit (p13) Meeting with senior administration of the institution; Discussion with program as well as shared faculty from supporting departments to assess the program strength and its conduct; Interaction meetings with students, alumni and other stakeholders including employers for obtaining their feedback; Meeting with services and administrative officials of the institute in connection with provision of support regarding finance, infrastructure, examination, admission & registration etc.; Review and analysis of all the documents furnished by the department / institution; Visits to laboratories, library, computing facilities, auditorium, sports facilities, hostels, faculty offices, classrooms, career placement office, medical and such other facilities. A concluding meeting with senior management of the program and institution to share observations of the visiting team.

Analyse inputs and outputs Table, matrices, flowcharts and checklists
EVALUATION TOOLS Sensible questioning Check records Observing processes Analyse inputs and outputs Table, matrices, flowcharts and checklists

HEI’s Evidence (p14) Why are you looking for these evidences ?
Samples of minutes of meetings; policy documents; faculty profile; syllabi; research publications; project reports. Details pertaining to faculty members to verify their requisite qualifications and to ensure their continuity and effectiveness for teaching, learning& research pursuits. Program curriculum, evidence of regular review and consistency with PEC / HEC guidelines and adoption of Outcome Based Education (OBE) system. Course files for the subjects offered in the program. Evidence for continuous assessment and improvement of the program and implementation plan. Random check of students’ work, question papers and answer sheets and student attendance record. Annual budgets for the period under review. Details of laboratories with equipment, its supporting staff and lab manuals. Measures taken for provision of general safety, health and environment. Availability of training aids for imparting quality education. Mapping of Program Educational Objectives /Course Learning Outcomes (PEOs/CLOs) with Program Outcomes. Other additional document(s) required in support of the program. Why are you looking for these evidences ?

VISIT DAY 1 (p14) Intensive & Lack of Time
Opening meeting with senior administration of the institution; Presentation by the Head of the Department of program being evaluated and ensuing discussion; Assessment and analysis of documents displayed in the exhibit room; Visit of program laboratories and allied facilities; Interaction with students; Visit to supporting and interdisciplinary departments and discussion with supporting staff; Visit to allied facilities such as library, computing, internet, medical, sports, hostels etc.; Discussion with alumni, employers and other stakeholders; Meeting with the faculty members; Second review meeting of team members regarding assessment of the program. Intensive & Lack of Time

VISIT DAY 2 (p15) Rather Relaxed
The evaluation team may request for any additional information / data or facts for clarifications to resolve issues or queries; Third review meeting of team members on overall assessment of the program; Sharing observations (strong and weak areas of the program) with the higher management of HEI; Final meeting (post-visit) of the team members for compilation of draft visit report; Submission of final visit report to EAB.

List down the parties involved in an accreditation visit
Exercise 1a List down the parties involved in an accreditation visit PEV, EAD, EAB Management Head of Department (HoD) Faculty Support staff Stakeholders: Industry; Alumni; Parents Students

List down the involved parties Accreditation Plan Opening Session
Exercise 1b Item/Criteria List down the involved parties Accreditation Plan Opening Session Program Educational Objectives (1) Program Outcomes (2) Curriculum & Learning Process (3) PEV, EAD Management, HoD Management, HoD, Faculty, Stakeholders Management, HoD, Faculty, Students HoD, Faculty, Students, Support Staff Students (4) Faculty (5) Infrastructure & Facilities (6) Continual Quality Improvement (7) Institutional Support & Financial Resources (8) Industrial Linkages (9) Exit Meeting HoD, Students, Faculty, Stakeholders Management, HoD, Faculty, Students Management, HoD, Students, Support Staff Management, HoD, Faculty Management, HoD, Faculty, Students, Support Staff Management, HoD

What are the typical words to begin with when questioning?
Quiz What are the typical words to begin with when questioning? What, Why, Where, Who, When and How

Questioning 6 friends – What, When, Why, Who, Where, How
Best friend – Show Me Additional skills of LISTENING and OBSERVING

OBE in a nut shell What do you want the students to have or able to do? Knowledge, Skill, Affective How can you best help students achieve it? Student Centred Delivery How will you know what they have achieved it? Assessment How do you close the loop PDCA

Self Assessment Report
Exercise 2 Self Assessment Report The SAR from XYZ University has been received by you. Prepare the list of questions that you plan to probe, based on the following criteria: Programme Objectives Programme Outcomes Curriculum and Learning Process Students Faculty Support staff Industry stakeholders Alumni Facilities and Infrastructure Institutional and Financial Support Continual Quality Improvement Industry Linkage

Thank You

Opening Session and Program Educational Objectives (1) Megat Johari

Quiz What are the methods or techniques to be employed when conducting an accreditation exercise? Check documents, Interview and Observe

Exercise 3 Opening Meeting You are chair the delegation to accredit three programmes; Mechanical Engineering, Civil Engineering and Electrical Engineering, for the third cycle. Prepare the list of talk points sequentially for addressing at the Opening Meeting?

OPENING MEETING Introduce evaluation team members
Mention the objective of the visit (programmes) Mention that it is not fault finding exercise but to identify the programme conformance to the Accreditation criteria Explain the methods of conducting the evaluation Review the plan and schedule Confirm the time of the closing meeting Invite the Programme owner to fill up the latest (within a specified timeframe) if any

PEC Manual 2014 1. Program Educational Objectives
Well-defined and published Program Mission Program’s educational objectives defined and consistent with the mission Program’s educational objectives based on the stakeholder’s needs A process in place to evaluate the attainment of educational objectives Evaluation results used for continual improvement of the program CQI

PEC Manual 2014 Program Educational Objectives
Broad statements: What graduates are expected to achieve (BE) a few years after graduation. Linked to student outcomes and learning assessment process. Include feedback from faculty, employers, alumni and other stakeholders CQI

Write down your evaluation on the following PEO statements
Exercise 4a Write down your evaluation on the following PEO statements To provide graduates with sufficient knowledge in engineering. To produce graduates who are sensitive and responsible towards the environment. To prepare graduates for design and innovation. Megat Johari Megat Mohd Noor

Write down your evaluation on the following PEO statements
Exercise 4b Write down your evaluation on the following PEO statements To provide graduates with sufficient knowledge in engineering and possess the necessary skills for work in the industry. To produce graduates who are sensitive and responsible towards the society, culture and environment. To prepare graduates for work in advanced design and innovation at international level. Megat Johari Megat Mohd Noor

PROGRAM EDUCATIONAL OBJECTIVE (PEO)
Limit number of statements (manageable) No restatement of outcomes Forward looking and challenging Distinctive/unique features/having own niche Specific, Measurable, Achievable, Result oriented, and having a Time frame (SMART) EAC Training Modules (OBE for Panel Evaluator)

PEO What are the typical questions to ask? Why are you asking them?
Exercise 5 PEO What are the typical questions to ask? Why are you asking them? What are you comparing against?

Thank You

Program Day 1 Time Item Facilitator 14.00 Program Outcomes (2)
Megat Johari

Program Outcomes What the graduates are expected to know and able to perform or attain by the time of graduation (skills, knowledge and behaviour/attitude) There must be a clear linkage between Objectives and Outcomes Need to distribute the outcomes throughout the program, and not one/two courses only addressing a particular outcome

PEC Manual 2014 2. Program Learning Outcomes
Well-defined and published (12) Program Outcomes Program Outcomes linked to the Program Objectives Mapping of Program Outcomes to Course Learning Outcomes (CLOs) Teaching-learning and assessment methods appropriate and supportive to the attainment of Course Learning Outcomes Quality of assessment mechanism to evaluate achievement levels for all the Program Outcomes by each student Process in place by which assessment results are applied to further refine the assessment mechanism and/or redefine the program / course outcomes, thus leading to continuous improvement of the program

PEC Manual 2014 Program Learning Outcomes
KSA Range of complex problem solving and complex engineering activities (Tables (1 & 2) in Annex A). Specific details relating to the processes adopted for assessing, evaluating and reviewing the program outcomes should be provided HEI can also present the internal quality assessment cycle adopted by its Quality Enhancement Cell (QEC).

Washington Accord Graduate Attributes PROGRAMME OUTCOMES
Engineering Knowledge Breadth & depth of knowledge WA2 Problem Analysis Complexity of analysis WA3 Design/Development of Solutions Breadth & uniqueness of engineering problems i.e. the extent to which problems are original and to which solutions have previously been identified and coded WA4 Investigation Breadth & depth of investigation and experimentation WA5 Modern Tool Usage Level of understanding of the appropriateness of the tool WA6 The Engineer and Society Level of knowledge and responsibility WA7 Environment and Sustainability Type of solutions WA8 Ethics Understanding and level of practice WA9 Individual and Team Work Role in and diversity of team WA10 Communication Level of communication according to type of activities performed WA11 Project Management and Finance Level of management required for differing types of activity WA12 Life-long Learning Preparation for and depth of continuing learning

PROGRAMME OUTCOME (i) Engineering Knowledge
Apply knowledge of mathematics, science, engineering fundamentals and an engineering specialisation to the solution of complex engineering problems;

PROGRAMME OUTCOME (ii) Problem Analysis
Identify, formulate, research literature and analyse complex engineering problems reaching substantiated conclusions using first principles of mathematics, natural sciences and engineering sciences

PROGRAMME OUTCOME (iii) Design/Development of Solutions
Design solutions for complex engineering problems and design systems, components or processes that meet specified needs with appropriate consideration for public health and safety, cultural, societal, and environmental considerations

PROGRAMME OUTCOME (iv) Investigation
Conduct investigation into complex problems using research based knowledge and research methods including design of experiments, analysis and interpretation of data, and synthesis of information to provide valid conclusions

PROGRAMME OUTCOME (v) Modern Tool Usage
Create, select and apply appropriate techniques, resources, and modern engineering and IT tools, including prediction and modelling, to complex engineering activities, with an understanding of the limitations

PROGRAMME OUTCOME (vi) The Engineer and Society
Apply reasoning informed by contextual knowledge to assess societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to professional engineering practice

PROGRAMME OUTCOME (vii) Environment and Sustainability
Understand the impact of professional engineering solutions in societal and environmental contexts and demonstrate knowledge of and need for sustainable development

PROGRAMME OUTCOME (viii) Ethics
Apply ethical principles and commit to professional ethics and responsibilities and norms of engineering practice

PROGRAMME OUTCOME (ix) Communication
Communicate effectively on complex engineering activities with the engineering community and with society at large, such as being able to comprehend and write effective reports and design documentation, make effective presentations, and give and receive clear instructions

PROGRAMME OUTCOME (x) Individual and Team Work
Function effectively as an individual, and as a member or leader in diverse teams and in multi-disciplinary settings

PROGRAMME OUTCOME (xi) Life-long Learning
Recognise the need for, and have the preparation and ability to engage in independent and life-long learning in the broadest context of technological change

PROGRAMME OUTCOME (xii) Project Management & Finance
Demonstrate knowledge and understanding of engineering and management principles and apply these to one’s own work, as a member and leader in a team, to manage projects and in multidisciplinary environments

Engineering Programme
Attributes Complex Problems Preamble Engineering problems which cannot be resolved without in-depth engineering knowledge and having some or all of the following characteristics: Range of conflicting requirements Involve wide-ranging or conflicting technical, engineering and other issues Depth of analysis required Have no obvious solution and require abstract thinking, originality in analysis to formulate suitable models Depth of knowledge required Requires in-depth knowledge that allows a fundamentals-based first principles analytical approach Familiarity of issues Involve infrequently encountered issues Level of problem Are outside problems encompassed by standards and codes of practice for professional engineering Extent of stakeholder involvement and level of conflicting requirements Involve diverse groups of stakeholders with widely varying needs Consequences Have significant consequences in a range of contexts Interdependence Are high level problems possibly including many component parts or sub-problems Engineering Programme

Exerise 6 Program Outcomes Discuss on a HEI’s possible approaches or methods to demonstrate implementation of the 12 program outcomes Discuss on the possible models to show attainment of the 12 program outcomes

Megat Johari Megat Mohd Noor
Problem Organised Project Work or POPBL (Project Oriented Problem Based Learning) Group Studies Lectures Literature Problem Analysis Problem Solving Report Tutorials Field Work Experiment Megat Johari Megat Mohd Noor

Requirements High degree of supervision Office space Lectures to be constantly changing or renewed Flexibility in the distribution of resources Megat Johari Megat Mohd Noor

Graduates’ Strength AALBORG UNIV Strong in problem solving Communication Cooperation General technical knowledge TECHNICAL UNIV Specialist knowledge Technical methodology Megat Johari Megat Mohd Noor

Instructors/Supervisors
Pedagogical skills Scientific skills Time management Project based on staff research Megat Johari Megat Mohd Noor

PO Attainment Knowledge Skills Affective Knowledge Skills Affective
Final Year Project Final Year Project Final Year Design Project Final Year Design Project Final Year Courses Final Year Courses Knowledge Skills Affective Third Year Courses Third Year Courses Second Year Courses Second Year Courses Knowledge Skills Affective First Year Courses First Year Courses

Curricula Models Distribution of Knowledge, Skills & Attitude
elements throughout the 4 years Yr. 4 S&A 30% S&A 30% Yr. 3 K 70% K 70% K 70% K 70% Yr. 2 S&A 30% Yr. 1 S&A 30% A B C D

Thank You

Curriculum & Learning Process (3) Megat Johari

PEC Manual 2014 3. Curriculum & Learning Process
Balanced (technical and non-technical) contents with appropriate assessment and evaluation methods. Well-defined core of essential subjects Supported by requisite compulsory as well as elective courses. Invoke awareness and comprehension of societal problems amongst the students and should motivate them to seek solutions for improving the quality of life. The theory content of the curriculum has to be supplemented with appropriate experimentation in laboratories.

PEC Manual 2014 Curriculum & Learning Process
Inputs from all stakeholders, especially from the industry, in developing curriculum contents Program structure should cover fundamental principles, leading to integrated studies in the final year (levels defined in Bloom’s taxonomy) The hallmark of a curriculum is to infuse original thinking, resourcefulness and entrepreneurial spirits among students

Number of instructional days, quality of faculty, contact hours per week, design and delivery of syllabi, student evaluation and feedback are the important aspects in reviewing the effectiveness of teaching-learning processes. In addition to regular teaching / learning activities such as classroom interaction, lab experimentation and faculty consultation, other aspects of student learning such as tutorial system, research / design projects, seminar / workshops and exposure to industrial practice should form an integral part of curriculum. Internal reviews of quality assurance procedures should be carried out periodically.

Internship 4-6 wks: report on training details, interest shown by the student; his/her work habits and punctuality. Lab Work: encouraged to develop practical skills; motivated to come up with their own design ideas; demonstrate the ability to investigate, analyze and solve complex engineering problems. Design Project: encouraged to undertake design projects as an integral part of every core subject; should inculcate intuitiveness, resourcefulness and the spirit to compete; motivated to participate in competitions to use their ingenuity, creativity and innovation

Final Year Project: comprise literature search, individual analysis, design and putting together various hardware, software and firmware modules to demonstrate a functional concept; Design projects shall include complex engineering problems and design systems, components or processes integrating core areas and meeting specified needs with appropriate consideration for public health and safety, cultural, societal, and environmental considerations.

The program must ensure that each student has achieved all PLOs to acceptable level through assessment of CLOs. The appropriateness of the assessment methods along with the level of achievement against the targeted outcomes must be evaluated. Mapping of program outcomes to individual courses, nature of assessment tools (direct/ indirect/rubrics) Process of evaluation to determine the attainment of PLOs should be demonstrated through reasonably convincing evidences.

New Bloom’s Taxonomy Bloom’s Taxonomy Knowledge (list)
Comprehension (explain) Application (calculate, solve, determine) Analysis (classify, predict, model,derived) Synthesis (design, improve) Evaluation (judge, select, critique)

lower order Intermediate Higher order
EAC Training Modules (OBE for Panel Evaluator)

Why are course outcomes important?
They are essential because they: define the type and depth of learning students are expected to achieve provide an objective benchmark for formative, summative, and prior learning assessment clearly communicate expectations to learners clearly communicate graduates’ skills to the stakeholders define coherent units of learning that can be further subdivided or modularized for classroom or for other delivery modes. guide and organize the instructor and the learner.

3 components of a learning outcome
3) Standard (criteria of acceptable level of performance) describe the principles used in designing X.(V) orally describe the principles used in designing X. (V&C) orally describe the five principles used in designing X. (V&C&S) design a beam. (V) design a beam using Microsoft Excel design template . (V&C) design a beam using Microsoft Excel design template based on BS 5950:Part 1. (V&C&S)

Learning outcomes by adding a condition and standard
Poor Students should be able to design research. Better Students should be able to independently design and carry out experimental and correlational research. Best Students should be able to independently design and carry out experimental and correlational research that yields valid results. Source: Bergen, R A Program Guideline for Outcomes Assessment at Geneva College

Course Outcomes (CO) Contribution to Programme Outcomes (PO)
Ability to function in multidisciplinary team Assign multidisciplinary design projects in engineering courses. Implement design projects with multidisciplinary teams Exercise: Identify a course and discuss how it can be implemented

Course Outcomes (CO)Contribution to Programme Outcomes (PO)
Broad education necessary to understand the impact of engineering solutions in a global, environment and societal context + knowledge of contemporary issues Include structured controversies in engineering course Conduct class exercise or homework problems that involve global/societal issues Exercise: Identify a course and discuss how it can be implemented

Course Outcomes (CO) Contribution to Programme Outcomes (PO)
Life Long Learning Teach students about learning styles and help them identify the strength and weakness of their styles and give them strategies to improve Use active learning methods to accustom them to relying on themselves Give assignments that requires library and www searches Anything done to fulfil criteria on: (a) understanding ethical and professional responsibility and (b) understanding societal and global context of engineering solutions, will automatically satisfy this criteria

ASSESSMENT: Processes that identify, collect, use and prepare data for evaluation of achievement of programme outcomes or educational objectives. EVALUATION: Processes for interpretation of data and evidence from assessment practices that determine the program outcomes are achieved or result in actions to improve programme.

Rubric Adopted from G.Rogers 4 - Exceeds Criteria 3 - Meets
2 - Progressing to Criteria 1 - Below Expectations Content Provides ample supporting detail to support solution/ argument Provides adequate argument. Some details but may include extraneous or loosely related material. Inconsistent or few details that may interfere with the meaning of the text. Organization Organizational pattern is logical & conveys completeness & wholeness. & wholeness with few lapses. Little completeness & wholeness, though organization attempted. Little evidence of organization or any sense of wholeness & completeness. Style Uses effective language; makes engaging, appropriate word choices for audience & purpose. language & appropriate word choices for intended audience Limited & predictable vocabulary, perhaps not appropriate for intended audience Limited or inappropriate vocabulary for the Consistently follows the rules of standard English. Generally follows the rules for standard English. Generally does not follow the rules of Does not follow the rules of standard

Performance Criteria/ Indicators - Good Teamwork
Students are able to demonstrate 1. Positive contribution to the team project (minutes of meeting) 2. Well prepared and participate in discussion (observation) 3. Volunteer to take responsibility 4. Prompt and sufficient attendance 5. Aplomb and decorum

Quiz PO1 PO2 PO9 PO10 CO1 + CO2 CO3 CO4
How would you design the assessment for the above matrix?

Quiz Table 1 Q1 CO1 + Q2 CO2 - Q3 CO3 Q4 CO4
Discuss on the attainment of COs and POs (using Exercise 5)for both Tables, 1&2 Table 2 Q1 CO1 + CO2 Q2 CO3 - Q3 CO4 Q4

Quiz PO1 PO2 PO3 C1 3 2 1 C2 C3 C4 Discuss on the potential problems, if any, where 3, 2, 1, and 0 refer to High, Moderate, Low, and No emphasis, respectively. C1..4 refer to the courses, whereas PO1..3 refer to Programme Outcomes. How would cohort POs attainment be obtained?

Quiz Delivery Assesment Lecture Laboratory PBL Case Method
Project Based Identify suitable assessment techniques for the different delivery modes.

Programme Outcome Assessment Matrix
Outcome indicators & core courses Outcome 1 Outcome 2 Project Report A B Course 1 Course 2 C A: slightly, B: moderately, C:substantively - base on a review of course materials (syllabus, learning objectives, tests, other assessment…..) Outcome 1: ability to ….. Outcome 2: ability to …..

Course Assessment Matrix
Outcome-related learning objectives Outcome 1 Outcome 2 Explain A C Perform calculation B Identify Solve A: slightly, B: moderately, C:substantively Outcome 1: ability to ….. Outcome 2: ability to …..

Probing on PEO/PLO What are the PEOs/PLOs?
Who were involved in the development of the PEOs/PLOs? How were they developed/improved? To what extent the stakeholders were involved? How their attainment were determined? What were the improvements introduced?

Random Questions How was the curriculum developed? What is your niche?
How are the PLOs demonstrated? How are the FYP titles being identified? What are your expectations on the FYP?

Curriculum & Learning Process
Exerise 7 Curriculum & Learning Process Discuss on the possible relationship between taxonomy levels and the different knowledge profile with consideration to the 12 program outcomes What are typical probing questions in ascertaining that students PLO have been attained? What are typical probing questions in ascertaining that students CLO have been attained?

Student-Centered Learning

Learning Style Model Perception Sensing Intuitive
Input Modality Visual Verbal Processing Active Reflective Understanding Sequential Global

Visual (Vs) Learners Verbal (Vb) Learners “Show me” “Explain it to me” - pictures - spoken words - diagrams - written words, symbols (seen, but translated by brain into their Oral equivalents) - sketches - schematics - flow charts - plots

Active (A) Learners Reflective (R) Learners Tend to process actively (doing something physical with presented material, then reflecting on it) Tend to process reflectively (thinking about presented material, then doing something with it) Think out loud Work introspectively “let’s try it out and see how it goes” “Let’s think it through and then try it” Tend to jump in prematurely Tend to delay starting Like group work Like solo or pair work

How will you know what they have achieved it?
Formative Assessment Sumative Assessment Course Assessment Program Assessment Assessment Tools Direct and Indirect Assessment

How do you close the loop ?
Assessment Plan Who is doing what and when Stakeholder participation CQI in place

Thank You

Program Day 2 Time Item Facilitator 09.00 Students (4) and Faculty (5)
Megat Johari

Direct / Indirect Questions

PEC Manual Students frame and enforce policies for admitting fresh (ENTRY) as well as transfer students guide students regarding academic and career matters policies to maintain a manageable teaching load in all semesters provide conducive teaching-learning environment, and also monitor / evaluate students’ progression towards achieving program outcomes and objectives inculcate ethical practices and inter-personal skills through opportunities / facilities for extra- and co-curricular activities provision of in-door and out-door sports facilities for physical fitness and mental endurance provide administrative and financial support for establishing student clubs, societies, and chapters for various co-curricular activities (meant to transform the students / graduates into proficient engineers)

PEC Manual 2014 Students Annual Intake: considering the capacity of the program and its allied facilities Admission response: applicants: accepted: joined Transfer of students: Academic counseling: Career and Student Wellness Counseling: Class Size (Theory): Class Size (Practical): Semester Academic Load: Completion of Courses and Student Feedback: gathered from the official record, e.g. course-file as well as through feedback and interaction with students Participation in Competitions: Student Performance Evaluation:

PEC Manual Faculty The faculty must demonstrate complete familiarity with Outcome-Based Educational (OBE) approach Their familiarity with the program objectives and outcomes and understanding of the outcome-based assessment cycle are the key elements to ensure attainment of program objectives. Performance appraisal mechanism …. for their adherence to program’s objective and outcomes.

PEC Manual 2014 Faculty Faculty strength
Full-Time Dedicated Faculty: 4 engineer faculty members holding post-graduate Engineering Qualifications (i.e. MS or PhD) and actively engaged in teaching core subjects Shared Faculty Visiting Faculty Faculty qualifications Teacher student ratio Faculty training and mentoring: Outcome-Based Education (OBE) system Faculty Retention, Development and Career Planning: management commitment, plan Pyramid of Academic Structure: Prog head? Faculty workload Faculty Research & Publications

Thank You

Infrastructure & Facilities (6) and Institutional Support & Financial Resources (7) Megat Johari

PEC Manual 2014 6. Facilities & Infrastructure
The adequacy of teaching and learning facilities such as classrooms, learning-support facilities, study areas, information resources (library), computing and information-technology systems, laboratories, workshops, and associated equipment to cater for multi-delivery modes. Describe the adequacy of support facilities such as hostels, sports and recreational centers, health care centers, student centers, and transport in facilitating students’ life on campus and enhancing character building.

PEC Manual 2014 7. Institutional Support & Financial Resources

Support Staff

Thank You

Continual Quality Improvement (8), and Industrial Linkages (9) Megat Johari

PEC Manual 2014 8. Continuous Quality Improvement

Industry Stakeholders

Alumni

PEC Manual 2014 9. Industry Linkages

Thank You

Program Day 2 Time Item Facilitator 16.00 Judgment and Exit Meeting
Megat Johari

Triangulation & Judgement

Total performance appraisal (p12) (Triangulation & Judgment)
Physical verification of infrastructure/ facilities, records Interviews with administrators, faculty, alumni, students / stakeholders and other activities To mention strengths and weaknesses against each criterion (p13)

Scenario 1 An evaluator asked the XYZ University management as to why the research and development activities at the university were below par as compared to other universities. He suggested that the management to institute a center for R&D and undertake rigorous collaborative R&D efforts with top three universities in the country. He also commented that his own university has an excellent set-up to drive the R&D agenda that the XYZ university can learn from. Discuss and give you opinion on the transpired interview.

TRIANGULATION APPROACH
Curriculum development (specification/input) Curriculum implementation (process) Demonstrated outcomes (output)

Objective Evidence Evidence is the facts or information used to prove or disprove a proposition. It should be collected through:- Interviewing Observation of environment Observation of implementation Checking of records of document

Objective Evidence Evidence that exists
Not influenced by emotion or prejudice Can be documented Is about quality Can be quantitative or qualitative Can be verified

Objective Evidence cont…
The facts or information used: to come to a conclusion on the objective evidence of whether programmes have or have not undertaken appropriate activities effectively to the attainment of the necessary outcomes

Obtaining Objective Evidence
Among the methods used will be:- a) Document Review (against the Manual) b) Questioning c) Check Records

Deficiency/Major/Minor Shortcomings
DECISION FLOW CHART Objective Evidence Acceptable Deficiency/Major/Minor Shortcomings OFI yes Does it breach the Manual ? yes yes no Trends? Can it be improved? no yes Does it Affect quality? no no

POINTS TO CONSIDER IN DERIVING FINDINGS/CONCLUSION
Establish requirement Probe process Whom do you speaks to? What to look for? Sampling How long to persist? Is there any shortcomings? Is it significant? Consult team members

EFFECTIVE COMMUNICATION
Occurs when the right person, says the right things, to the right people, at the right place at the right time and in the right way to be heard and understood and to produce the right response. Important Person is at ease in communicating with the Evaluator. Evaluator should do all he/she can to make person feel at ease.

EFFECTIVE COMMUNICATION (Cont..)
Tips Gain attention from the person before starting. Explain clearly the purpose of the session/visit. Include friendly remarks or express your interest in what he/she is doing. Politeness all the way never antagonise or belittle the person. Establish eye contact all the times. Communicate in the language he/she is comfortable. Use of body language to promote the dialogue. (Spoken message is 7%, verbal and vocal 38% and 55% facial). Listen, listen, listen, an Evaluator need to train himself to be an active listener.

TIPS ON GOOD LISTENING Tip # 1
Be open! Switch off all negative thoughts about the person. Respective to what is being said. Drop those emotional barriers that filter out what is being said or cause you to hear only what you want to hear. Tip # 2 Start listening to the first sentence! Self-centered people can’t actively listen. They tend to be preoccupied with their own daydreams. Put aside what you are doing and concentrate on what the person is saying. Tip # 3 Concentrate on what is being said! Actively try to hear every words as if it were the most important thing you could hear at that moment. Avoid the temptation to think faster than the person is talking.

Tip # 4 Look for the meaning of what is being said. Don’t try to read your own meanings into what the person is saying. Rather, help the person convey his or her own meanings by showing genuine interest. Tip # 5 Avoid the temptation to interrupt! Dr. David Schwart, in his book The Magic of Thinking Big, says “Big people monopolize the listening. Small people monopolize the talking’. Tip # 6 Ask questions that stimulate the person to talk and clarify your understanding of what is being said. Use trail questions, like “Do I understand correctly that….,” to test your understanding.

Tip # 7 Record important point being made. If appropriate, take notes. Tip # 8 Screen out interruptions and ignore distractions. Tip # 9 Use facial expressions and body language to express interest and comprehension. Tip # 10 Don’t over-react to highly charged or emotional words; look for the meanings behind those words. Avoid jumping to conclusions. Hear the person out.

EVALUATOR’S CHARACTERISTICS
YES Punctual Objective Analytical Open minded Inquisitive Polite Good communicator Honest Impartial Industrious/good judgment Patient NO Cynical A nit picker A quitter Argumentative Anxious to please opinionated

PRACTICAL TIPS ON EVALUATION
Evaluators are adequately trained. The blind can only led the blind. Tip 2 Evaluators know the area they are entering. Do some research prior to the evaluation. Read manual, procedures , previous evaluation reports and other records. Take time to prepare your checklist. Tip 3 Department Head, supervisor and other personnel should be clear of the purpose of the evaluation. Tip 4 Record findings immediately

Reporting Qualitative Strength Shortcomings (weaknesses) Concerns
Opportunities for Improvement

Chair of Team(s) Presentation
University Programmes evaluated Strength, weakness, concern, OFI Recommendation(s)

Accreditation Decision
Five years. Less than five years, for Minor shortcoming(s). A further visit will be scheduled to verify the results of the remedial action(s), unless deemed unnecessary Decline, a further application will normally not be considered within the next one year. Defer, to allow the IHL to fulfil condition(s) that may be imposed by the EAC

At an accreditation visit the HEI was found to:
have 11 (7 full time and 4 part time) engineering faculty and 2 full time non-engineering faculty conduct a programme with 127 credits over 8 semesters (within 3.5 years) maintain a student-faculty ratio of 25.1:1 based on full-time faculty have implemented the final year project with 5 credits

Expectations on Evaluators
Commitment Not Auditors Reference Material: EAC Manual Pre-Visit planning & discussion Day-1 meeting (seen doing) Visit Day Aplomb & Decorum Reporting Response to factual inaccuracies

Random Observations Cut & Paste Bullet points & Aggregation Ambiguous
Poor time management Guidelines supersede Manual Keywords as sole determination Interrogative Cut & Paste

Washington Accord Evaluate the Process
EAC Processes are adhered Manual based Time management Aplomb & Decorum Probing & ability to make valid judgment Safety

Washington Accord Evaluate the Standard
Threshold or exception standard Demonstration of PEO; PO; Complex Problem; Knowledge Profile Delivery & Assessment Minimum mastery of engineering knowledge

Thank You

Training for PEC Program Evaluators

Similar presentations

Presentation on theme: "Training for PEC Program Evaluators"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Training for PEC Program Evaluators

Similar presentations

Presentation on theme: "Training for PEC Program Evaluators"— Presentation transcript:

Similar presentations

About project

Feedback