1. Assessment and Reporting
Graduate Attributes
Assessment and Reporting
December 2016

2. Accreditation Requirements
Assessment of “Graduate Attributes” (GAs)
Use of the above results as part of a “Continual Improvement” (CI) process

3. Graduate Attributes - CEAB
“Generic characteristics, specified by the Accreditation Board, expected to be exhibited by graduates of accredited Canadian engineering programs at the time of graduation.” 1.
KB:
Knowledge base for engineering 2.
PA:
Problem analysis 3.
IN:
Investigation 4.
DE:
Design 5.
ET:
Use of Engineering tools 6.
IT:
Individual and teamwork 7.
CS:
Communication skills 8.
PR:
Professionalism 9.
IE:
Impact of engineering on society and the environment
10.
EE:
Ethics and equity
11.
EP:
Economics and project management
12.
LL:
Life-long learning
The graduate attributes broadly define the required outcomes, but they are difficult to measure directly.

4. Indicators Note: We are required to “measure” indicators, not GAs
To measure graduate attributes, all universities must define their own measurable descriptors called “indicators”:
“Descriptors of what students must do to be considered competent in the attribute; the measurable and pre-determined standards used to evaluate learning (i.e. measureable characteristics of attributes or components of attributes).”
Note: We are required to “measure” indicators, not GAs

5. Indicators (cont.)
The Faculty has developed a “universal” set of indicators
See:
or just Google “McGill
Engineering Accreditation”

6. Our CI Process

7. Continual Improvement Process
Step 1- Program Evaluation
Step 2-Curriculum Mapping
Step 3- Data Collection of Student Learning
Step 4- Data Analysis and Interpretation
Step 5- Data-informed Curriculum Improvement
Step 5 – 1: Program Evaluation Feedback Loop
Step 5 – 2: Curriculum Map Feedback Loop

8. (Current) Procedure Quantitatively assess all GAs:
at all levels: “I” (Introduced), “D” (Developed) or “A” (Applied),
in all courses,
in all years,
for all students,
by correlating performance in “assessment tools” with that in GAs / indicators
However, to get satisfactory data for all GAs in a given program, departments must also qualitatively assess all GAs.
use other approaches to evaluate “soft skills” GAs (e.g., IT, CS, PR, IE, EE, EP, LL)
employ other approaches to evaluate GAs other than just correlating GAs with graded work
use rubrics to evaluate GAs (either the Faculty’s universal rubric, or a customized one)
A special emphasis on obtaining data for the “soft skills” GAs is undertaken in FACC 100 and FACC 400 (which are common to all programs).
A new on-line course (FACC 250: Responsibilities of the Professional Engineer) has been introduced by the Faculty to give students more exposure to PR, IE, and EE

9. (Current) Procedure (cont.)
The Faculty is providing general support to assist departments in meeting the necessary requirements to obtain, analyze and present their GA data.
However, every course, program and department is different, so the process may not satisfy all situations
Adaptation and customization will be required by each department
CEAB guidelines and expectations are very limited
Although the faculty can provide guidance, professors and departments will sometimes need to use their judgment, as many questions do not have clear answers!

10. GA/CI Organizational Structure
Execution & Operation Level
Management Level
Leadership Level
Associate Dean-Academic Programs
F-CPEI Committee
D-CPEI Committee
Instructors/faculty members
Administrative staff members
Accreditation Assessment Specialist
Pedagogical Coordinator

11. F-CPEI & D-CPEI
In 2013, the Faculty-level Continuous Program Evaluation and Improvement (F-CPEI) committee was created.
Chaired by the Associate Dean-Academic Programs
Manages/monitors the implementation and development of the processes and procedures for outcome-based assessment for all Engineering programs.
The management of outcome-based assessment for each program is carried out by the Department-level Continuous Program Evaluation and Improvement (D-CPEI) committee.
Responsible for curriculum assessment and improvement implementation for undergraduate courses.
All faculty members are engaged in the outcomes-based assessment process.

12. D-CPEI Membership Typical timetable for and purpose of D-CPEI meetings
Leadership: Chair of the department
Academic committee representative
Faculty CPEI representative
Faculty members
External member(s) (i.e. members from other departments, universities and/or industry)
Undergraduate student (VP (Academic) – Departmental student organization)
Administrative Assistant
Typical timetable for and purpose of D-CPEI meetings
Smaller meetings in the Fall and Winter semesters
Major annual meeting in the Summer to:
Interpret departmental GA analysis
Identify short-term actions
Monitor long-term actions
Approve Continual Improvement actions being forwarded to F-CPEI
Keep minutes of these meetings!

13. Evaluating GAs

14. Overview
Programs
(i.e. correlating graded assessments with individual GAs)
Infer GAs from assessment tools in courses
General approach
(most courses, Faculty does the analysis)
Using myCourses
(analysis is the responsibility of the prof. / dept.)
Using Excel
Alternate approach
To ensure good coverage of GAs is obtained
Used to evaluate GAs by other means

15. General Approach
Use myCourses for the majority of courses to correlate individual student performance in certain GAs with the various assessment tools used by the instructor
The gradebook setup is done by the instructors or TAs
One-on-one consultation with professors is available
Group TA training will be offered at the beginning of each semester
Professors (only) to associate graduate attribute indicators with each graded item
Professors (or TAs) are responsible for entering data (grades) in myCourses at a level that is sufficiently refined to distinguish between different graduate attribute indicators
Analysis of the data will be done centrally

16. Gradebook

17. Alternate Approaches
Departments will need to add to the general approach to:
Acquire sufficient quality data for the “soft skills” graduate attributes
Use alternate GA evaluation approaches to satisfy CEAB requirements regarding types and variety of measurement tools
Maria Orjuela-Laverde (Academic Associate, Pedagogical Coordinator) is available for pedagogical assistance to develop alternate GA evaluation methods

18. Additional approaches (cont.)

19. Additional approaches (cont.)

20. Recommendations Recommendations to departments:
Examine your curriculum maps to look for “holes” in the data
Start by focussing on “Intro to XXX Engineering” course (or its closest approximation) and capstone design courses to complement your general approach for GA data acquisition
List GA indicator codes (e.g. “KB.1,” “PA.2,” etc.) for each question/exam sheet for all graded items (assignments, labs, exams)
List learning outcomes on all course outlines (a requirement for the revised course information sheets!)
Remember to collect appropriate sample materials for your courses.

21. Faculty Assistance
Analyzing and processing the data acquired via myCourses
Providing support (Nasim and Maria) to both professors and TAs
Providing reports on the Faculty-wide survey results for:
Capstone design survey
Alumni survey
Obtaining GA data
in common FACC courses
Establish a faculty-wide “program improvement” procedure based on GA data

22. Misc. comments Some courses are not on myCourses
For a “standard” course, the process has been designed so that the additional work required by a professor is small (as low as 2 hours per course)
Subsequent years will require less work (e.g. myCourses gradebook template can be imported from previous years)

23. Course Information Sheet
Learning Outcomes
Course Information Sheet

24. http://www. engineerscanada

25. Course Learning Outcomes
“What the student is expected to have learned - These should be analogous to indicators and may have been used for GA assessment purposes. A small number of outcomes (general measurable skills acquired) are expected - not a list of content.”

27. Resources Website: Video clips Instructions TA Workshops(
Short/Detailed list of graduate attributes and indicators
Graduate Attribute Indicator Rubrics
Video clips
How to setup your gradebook
Instructions
Gradebook setup in myCourses
Copying gradebook from one course to another
Importing grades from Excel to myCourses
TA Workshops
One session is offered per semester

28. Thank you!