1. Program development process at Queen’s University to demonstrate graduate attributes Brian Frank Director (Program Development) Faculty of Engineering and Applied Science Queen's University 1

2. Focus “ The institution must demonstrate that the graduates of a program possess the attributes under the following headings... There must be processes in place that demonstrate that program outcomes are being assessed in the context of these attributes, and that the results are applied to the further development of the program.” 2

3. CEAB Instructions Describe the processes that are being or are planned to be used. This must include: a)a set of indicators that describe specific abilities expected of students to demonstrate each attribute b)where attributes are developed and assessed within the program… c)how the indicators were or will be assessed. This could be based on assessment tools that include, but are not limited to, reports, oral presentations, … d)evaluation of the data collected including analysis of student performance relative to program expectations e)discussion of how the results will be used to further develop the program f)a description of the ongoing process used by the program to assess and develop the program as described in (a)-(e) above 3 Engineering Graduate Attribute Development (EGAD) Project

4. Approach Short term objectives (2010-2011): Set up a comprehensive process limited to a small number of courses to help programs understand the process Use data to help faculty see value in outcomes assessment for program improvement Long term: Comprehensive assessment of all attributes throughout programs Evaluate validity of data Students take responsibility for demonstrating some attributes

5. Queen's University timeline  Summer 2009: Working groups of faculty, students, topical experts created specific program-wide indicators (next slide, and in Appendix 3.1A)  Summer 2009: Setup learning management system (Moodle) to manage assessments  Sept 2009-April 2010: Piloted assessment in first year  Sept 2010-April 2011: Piloted assessment in first year, faculty wide second year, and fourth year (common across programs)  April – July 2011: Student surveys and focus groups, curriculum mapping, data analysis Curriculum planning happening throughout

6. Why initial emphasis on first year? First year is faculty-delivered, core to all students Provides opportunity to pilot a process Help disseminate outcomes assessment procedures to other instructors Long term: assessment process continue in first year program to inform development

7. Aside: Idealistic course development process 7 Engineering Graduate Attribute Development (EGAD) Project Identify course objectives and content Identify course objectives and content Create specific outcomes for each class Create specific outcomes for each class Map to experiences (lectures, projects, labs, etc.) Map to experiences (lectures, projects, labs, etc.) Identify appropriate tools to assess (reports, simulation, tests,...) Identify appropriate tools to assess (reports, simulation, tests,...) Student input Deliver, grade, seek feedback Deliver, grade, seek feedback Analyze and evaluate data Analyze and evaluate data Overall Improvement Overall Improvement Create and Execute a Plan Create and Execute a Plan

8. Program-wide assessment process flow 8 Engineering Graduate Attribute Development (EGAD) Project Defining Purpose and Outcomes Defining Purpose and Outcomes Program Mapping Program Mapping Stakeholder input Identifying and Collecting Data Analysis and Interpretation Analysis and Interpretation Create a Program Improvement Plan Create a Program Improvement Plan Program & Course Improvement Program & Course Improvement

9. Human capital Director, Program Development to manage process Faculty member from each program Other experts as appropriate (economics, information management, etc.) Currently separate from faculty-wide curriculum development committee 9

10. Resources/time commitment  Creating assessment criteria: 7 committees of approximately 5 people who each met about 4 times  Mapping criteria to a course and creating rubrics for assessment: ~ 10 hours  Large scale curricular changes: ~10 person committee, most of whom had 1 course relief bought out by dean  Coordination (resource gathering, planning, curricular planning): ~30% of a position

11. Academic and curricular structure Dean Associate Dean (Academic) Director (Program Development) NSERC Design Chair DuPont Canada Chair in Engineering Education Faculty-wide curriculum committee Dean’s Retreat Curriculum Review Committee (DRCRC) Graduate attribute assessment committee

12. What are indicators? Engineering Graduate Attribute Development (EGAD) Project 12 Lifelong learning An ability to identify and address their own educational needs in a changing world in ways sufficient to maintain their competence and to allow them to contribute to the advancement of knowledge Lifelong learning An ability to identify and address their own educational needs in a changing world in ways sufficient to maintain their competence and to allow them to contribute to the advancement of knowledge Can this be directly measured? Can this be directly measured? Would multiple assessors be consistent? Would multiple assessors be consistent? How meaningful would the assessment be? How meaningful would the assessment be? Probably not, so more specific measurable indicators are needed. This allows the program to decide what is important Probably not, so more specific measurable indicators are needed. This allows the program to decide what is important

13. Indicators: examples Engineering Graduate Attribute Development (EGAD) Project 13 Lifelong learning An ability to identify and address their own educational needs in a changing world in ways sufficient to maintain their competence and to allow them to contribute to the advancement of knowledge Lifelong learning An ability to identify and address their own educational needs in a changing world in ways sufficient to maintain their competence and to allow them to contribute to the advancement of knowledge Critically evaluates information for authority, currency, and objectivity when referencing literature. Critically evaluates information for authority, currency, and objectivity when referencing literature. Uses information ethically and legally to accomplish a specific purpose Uses information ethically and legally to accomplish a specific purpose Identify gap in knowledge and develop a plan to address Identify gap in knowledge and develop a plan to address Gradua te attribut e The student: Describes the types of literature of their field and how it is produced Describes the types of literature of their field and how it is produced Indicators

14. Establishing Indicators A well-written indicator includes: what students will do the level of complexity at which they will do it the conditions under which the learning will be demonstrated Engineering Graduate Attribute Development (EGAD) Project 14 Critically evaluates information for authority, currency, and objectivity in reports. Content area Level of expectation (“describes”, “compares”, “applies”, “creates”, etc.) context

15. Graduate attribute categories levels Assessment criteria 15 Engineering Graduate Attribute Development (EGAD) Project Linkage to OCAV UDLEs

16. Rubric example  Creating defined levels (“scales”) of expectations reduces variability between graders, makes expectations clear to students thresholdtarget

17. Sample First year indicators for problem analysis and design 17 Engineering Graduate Attribute Development (EGAD) Project 3.02-FY1Identifies known and unknown information, uncertainties, and biases when presented a complex ill-structured problem 3.02-FY2Creates process for solving problem including justified approximations and assumptions 3.02-FY3Selects and applies appropriate quantitative model and analysis to solve problems 3.02-FY4Evaluates validity of results and model for error, uncertainty 3.03-FY1Generates ideas and working hypothesis 3.03-FY2Designs investigations involving information and data gathering, analysis, and/or experimentation 3.03-FY3Synthesizes data and information to reach conclusion 3.03-FY4Appraises the validity of conclusion relative to the degrees of error and limitations of theory and measurement 3.04-FY1Adapts general design process to design system, component, or process to solve open- ended complex problem. 3.04-FY2Accurately identifies significance and nature of a complex, open-ended problem 3.04-FY3Identifies customer and user needs 3.04-FY4Gathers and uses information from appropriate sources, including applicable standards, patents, regulations as appropriate. 3.04-FY5Produces a variety of potential design solutions suited to meet functional specifications 3.04-FY6Performs systematic evaluations of the degree to which several design concept options meet project criteria 3.04-FY7Compares the design solution against the problem objective

18. Sample fourth year indicators for Problem analysis and Design 18 Engineering Graduate Attribute Development (EGAD) Project 3.02-GY1Identifies problem, known and unknown information, uncertainties, and biases 3.02-GY2Creates process for solving problem including justified approximations and assumptions 3.02-GY3Selects and applies appropriate model and analysis to solve problems 3.02-GY4Evaluates validity of results and model for error, uncertainty 3.04-GY1Identify problem and constraints including health and safety risks, applicable standards, economic, environmental, cultural and societal considerations 3.04-GY2Applies appropriate knowledge, judgement, and design tools, in creating and analyzing conceptual design solutions to select best concept 3.04-GY3Creates and tests simulations, models, and/or prototypes at various points in design with complexity appropriate to design stage 3.04-GY4Assesses design performance based on requirements, yield, reliability, and/or safety as appropriate 3.04-GY5Identifies possibilities for further improvement and conducts design review to evaluate performance of the overall process.

19. Program-wide assessment process flow 19 Engineering Graduate Attribute Development (EGAD) Project Defining Purpose and Outcomes Defining Purpose and Outcomes Program Mapping Program Mapping Stakeholder input Identifying and Collecting Data Analysis and Interpretation Analysis and Interpretation Create a Program Improvement Plan Create a Program Improvement Plan Program & Course Improvement Program & Course Improvement

20. Curriculum mapping 20

21. 21

22. 22

24. Student surveys and focus groups Provides student input: implementing attribute assessment in program perceptions on where attributes are developed within the program as complement to curriculum mapping via faculty survey perception of importance within program

25. Questions What do you think are priorities within the program? What courses contribute to development of attribute {}? Which attributes are difficult to demonstrate? How would you recommend that attributes be developed?

26. Self reported demonstration at program entry Top five Grad Attributes where students reported a rating of 2 or 3 (yes or to a great degree) out of 3 Individual and Team Work88.73% Communication Skills78.17% Professionalism69.02% Problem Analysis61.26% Investigation60.56% Potential for students to perceive little value in learning activities directed toward developing these attributes

27. First year program supports: 27 Attributes in students’ top five responses Individual and Team Work*94.97% Knowledge Base in Engineering93.53% Problem Analysis*93.53% Professionalism*85.58% Investigation*82.48% Design80.58% Impact of Engineering on Society80.58% *Identified as a strength coming in to the program

28. First year program supports 28 Bottom three responses Ethics and Equity 64.03% Economics and Project Management69.56% Lifelong Learning73.19% These three are a significant focus in APSC-100, embedded in various activities.

29. Attributes perceived to be program priorities 29 Attribute# of students who incl it in top 5 # of students who selected it as #1 priority Problem Solving*12034 Individual & Team Work*9717 Knowledge Base9551 Communication*76 6 Professionalism*474

30. Graduating students: low priority attributes in program 30 Attribute# who included it in bottom 3 # who ranked it lowest priority Lifelong learning8038 Economics and Project Management 6825 Ethics and Equity451 Use of Eng Tools3815 Impact on Society 376

31. Focus group suggestions Communicate graduate attributes and draw attention back to them What is lifelong learning”? Professionalism and ethics and equity should be focused on in upper years

32. Program-wide assessment process flow 32 Engineering Graduate Attribute Development (EGAD) Project Defining Purpose and Outcomes Defining Purpose and Outcomes Program Mapping Program Mapping Stakeholder input Identifying and Collecting Data Analysis and Interpretation Analysis and Interpretation Create a Program Improvement Plan Create a Program Improvement Plan Program & Course Improvement Program & Course Improvement

33. Assessment in 2010-2011 AttributeFirst yearUpper years Knowledge baseFCI, Calculus Problem analysisComplex problem solving Capstones Investigation DesignDesign projectCapstones Engineering toolsComplex problem solving Capstones CommunicationsDesign projectCommunications, Capstones Individual and teamwork Design projectCapstones Impact of engineeringDesign projectCapstones ProfessionalismComplex problem solving Capstones Ethics and equityComplex problem solving Capstones Economics Lifelong learningDesign projectCapstones 33

34. Analyze and evaluate…  Histogram of results by level (did or did not meet expectations)  Histogram of results by student (how many indicators did each student fall below  Trend over time  Triangulation: examination of correlation between results on multiple assessments of the same indicator data with exam results)

35. First year: Second year of pilot

37. time

38. Knowledge base: Mathematics Calculus instructor asked questions on exam that specifically targeted 3 indicators for “Knowledge”: 1.“Create mathematical descriptions or expressions to model a real-world problem” 2.“Select and describe appropriate tools to solve mathematical problems that arise from modeling a real-world problem” 3.“Use solution to mathematical problems to inform the real-world problem that gave rise to it”

39. Indicator 1: The student can create and/or select mathematical descriptions or expressions for simple real-world problems involving rates of change and processes of accumulation (overlaps problem analysis) 39 Context: calculating Intersection of two trajectories

40. Indicator 2: Students can select and describe appropriate tools to solve the mathematical problems that arise from this analysis 40 Context: differentiation similar to high school curriculum

41. Indicator 2: Students can select and describe appropriate tools to solve the mathematical problems that arise from this analysis 41 Context: implicit differentiation, trig inverse

42. Program-wide assessment process flow 42 Engineering Graduate Attribute Development (EGAD) Project Defining Purpose and Outcomes Defining Purpose and Outcomes Program Mapping Program Mapping Stakeholder input Collecting Data Analysis and Interpretation Analysis and Interpretation Create a Program Improvement Plan Create a Program Improvement Plan Program & Course Improvement Program & Course Improvement

43. All first year indicators over time

44. # Students falling below expectations in first year

45. Graduating year

46. Starting point: histograms Very few students falling below threshold level in capstone courses for most indicators 46

47. Area for improvement in graduating year: technical literature 47 3.12-FY1Uses information effectively, ethically, and legally to accomplish a specific purpose, including clear attribution of Information sources. 3.12-FY2Identifies a specific learning need or knowledge gap. 3.12-FY5Identifies appropriate technical literature and other information sources to meet a need 3.12-FY6Critically evaluates the procured information for authority, currency, and objectivity.

48. Data evaluation Across multiple capstone courses, students scoring lower on indicators involving: Evaluating validity of results Evaluating techniques and tools Evaluating effectiveness of results Evaluating information Pattern: evaluation 48

49. Curriculum Mapping: CurriKit Curriculum mapping software developed by U Guelph Provides information to identify: the courses which develop each graduate attribute what assessment is done and when which instructional approaches are used

50. Program-wide assessment process flow 50 Engineering Graduate Attribute Development (EGAD) Project Defining Purpose and Outcomes Defining Purpose and Outcomes Program Mapping Program Mapping Stakeholder input Collecting Data Analysis and Interpretation Analysis and Interpretation Create a Program Improvement Plan Create a Program Improvement Plan Program & Course Improvement Program & Course Improvement

51. First year development More focus on grader training and calibration Problem analysis: greater focus on making an effective argument Design: the focus on safety and risk assessment Communications: Weak communicators flagged and supported in fall semester (EPT). Required resubmission/assistance for falling below threshold Lifelong learning: evaluating information Ethics and equity and professionalism get stronger emphasis

52. Development in upper years In 2012-2013 all programs will have an open-ended design experience in third year which will apply disciplinary tools and principles, and incorporate professional issues and communications In 2013-2014 capstone courses will be revised to strengthen professional skills (safety, role of engineers in protecting public safety, ethics, communications)

53. Program improvement Students take more responsibility for learning and demonstrating attributes Faculty-wide curriculum development Engineering design and practice sequence Assessment built into sequence Multi-disciplinary approach to developing professional skills in an integrative experience Developing leadership through peer mentoring program 53

54. Process development Improve common indicators Develop disciplinary indicators Triangulation: Indicators measured using multiple methods or events to assess validity Attributes measured at multiple times in students' program with leveled expectations Satisfy both CEAB and province (OCAV UDLEs for Queen’s Quality assurance process) 54

55. End

56. Moodle

57. Sample course Moodle page

58. Moodle online assignments 58 Engineering Graduate Attribute Development (EGAD) Project

59. Assignment upload 59 Engineering Graduate Attribute Development (EGAD) Project

60. Outcome grading for assignments 60 Engineering Graduate Attribute Development (EGAD) Project

61. Outcome grading - popup 61 Engineering Graduate Attribute Development (EGAD) Project

62. Moodle development  Customizing Moodle for our purposes  Group upload/grading of assignments  Peer evaluation  Class response system (“clickers”)  Future collaboration