Implementing Outcomes Assessment: An Approach Based on Competencies Steve Mickelson Agricultural and Biosystems Engineering

2 Agricultural & Biosystems Engineering (ABE)  Two undergraduate majors - Agricultural Engineering (~150 students) - Agricultural Systems Technology (~150 students) - Industrial Technology (~220 students)  Two graduate majors (~90)  In two colleges: Engineering and Agriculture  33 faculty - 22 involved in undergraduate teaching - Faculty are academic advisors - No teaching assistants Agricultural & Biosystems Engineering

3 Presentation Road Map  Our path to competencies  Our outcomes assessment plan  Implications and implementation  Now what? Agricultural & Biosystems Engineering

4 The Path to Competencies Agricultural & Biosystems Engineering

5 The Buzz: Outcomes Assessment!  ISU university-wide accreditation in Evidence of active outcomes assessment programs is required.  ISU College of Engineering - ABET in 2000 & 2006  ABE has been thinking and working a lot on this with mostly good faculty buy-in Agricultural & Biosystems Engineering

6 Our Outcomes via ABET a)an ability to apply knowledge of mathematics, science, engineering and technology b)an ability to design and conduct experiments, as well as to analyze and interpret data c)an ability to design a system, component, or process to meet desired needs d)an ability to function on multi-disciplinary teams e)an ability to identify, formulate, and solve engineering problems f)an understanding of professional and ethical responsibility g)an ability to communicate effectively h)the broad education necessary to understand the impact of engineering solutions in a global and societal context i)a recognition of the need for, and an ability to engage in life-long learning j)a knowledge of contemporary issues k)an ability to use the techniques, skills, and modern engineering tools necessary for engineering and technology practice We don’t know how to measure an ability. Agricultural & Biosystems Engineering * * * * * Alverno “ABET” Outcomes

7 Outcomes  Abilities  Competencies  A complex ability cannot be observed directly; it must be inferred from performance.  Abilities are complex combinations of competencies.  Competencies are the application of behavior and motivation to knowledge, understanding and skill.  Key actions that demonstrate workplace competencies can be measured. Agricultural & Biosystems Engineering

8 Identifying Workplace Competencies  Cooperative effort with Development Dimensions International, Inc. (DDI)  Stakeholder dialogue, faculty analysis - “Critical incident” stories of workplace success - Stories identified and validated 14 competencies mapped back to outcomes  Each competency has Key Actions that are definable and measurable Agricultural & Biosystems Engineering

9 ISU Engineering Competencies Engineering Knowledge InnovationTeamwork General Knowledge Cultural Adaptability Integrity Continuous Learning Analysis & Judgment Professional Impact Quality Orientation PlanningCustomer Focus InitiativeCommunication Leadership and Safety? Agricultural & Biosystems Engineering

10 ABET Outcomes – ISU Competency Matrix

11 Competencies Have Meaning  “Transparent” to all participants  A great match with: - employers’ goals - student and faculty understanding  Provides a map and tools for students to achieve their goals in a meaningful way Agricultural & Biosystems Engineering

12 Our Outcomes Assessment Plan Agricultural & Biosystems Engineering

13 How Should We Assess Outcomes?  Actual student work is the primary evidence.  Exit interviews, post-graduate surveys, etc. are opinions and are only supporting evidence  ABE will use: - ePortfolios - Internship evaluations - Employer evaluations of alumni - FE exam Agricultural & Biosystems Engineering

14 What are ePortfolios?  Collection of artifacts for certain purposes displayed on the web  Our purpose: developing and demonstrating student competencies  It’s all about learning! - Reflection - Demonstration Agricultural & Biosystems Engineering

15 Why ePortfolios? (Students)  Student learning - Building an ePortfolio requires reflection Reflection → Learning! - Faculty can have access to student work for comment, review  Adds depth to resume for career search success - Highlight skills in ways not previously possible - Potential employers are interested Agricultural & Biosystems Engineering

16 Why ePortfolios? (Faculty)  Outcomes assessment  A tool to better manage, review, reflect, and comment on student work (class & advising)  Faculty learning - Building an ePortfolio requires reflection Reflection → Learning! - Adding access to student work to document teaching excellence Agricultural & Biosystems Engineering

17 On-line Competency Assessment  OPAL  by DDI, Inc. - On-line Performance and Learning - Web-based assessment and learning tool built on competencies  Multiple uses for us - Internship evaluations by employers - Alumni evaluations by employers - Classroom (self and 360º) Agricultural & Biosystems Engineering

18 Mission, Goals & Objectives Stakeholders Ind Advisory Board Faculty ABET / ASAE Outcomes & Competencies Classes mapped to outcomes & competencies Key Assignments Co-op & Internships Extra-curricular activities ePortfolios ABE Outcomes Assessment Map Annually 6 years INDIRECT MEASURES DIRECT MEASURES Curriculum Committee Recommendations Ind Advisory Board Senior exit surveys SEI 5 year post grad survey CSREES review Advisor evaluations Placement statistics Co-op & internship evaluations ePortfolios Employer evaluation of 5- year grads FE exam Faculty (Individuals and as a Whole) Strengths & Weaknesses

19 TTYP – Turn to Your Partner  What are the barriers to implementing an outcomes assessment plan? Agricultural & Biosystems Engineering

20 Our Barriers  Faculty and administrative buy-in  Financial resources  Expertise  Clear vision  Time - Faculty development - Competing demands  Rewards  Champion(s) Agricultural & Biosystems Engineering

21 Implications and Implementation Agricultural & Biosystems Engineering

22 Stakeholders’ View of the Probability of Developing and Demonstrating the Communication Competency 50%

23 Competency-based Learning (CBL)  Learning outcomes in terms of competencies - curricula focus on competency development  “Transparent” to all participants  Great match with employers’ goals and student understanding  Provides a map and tools for students to achieve their goals in a meaningful way Agricultural & Biosystems Engineering

24  Faculty must think, teach and assess in terms of competencies.  Our curriculum must evolve. - It can no longer be a series of requirements (or hurdles). - Opportunity to integrate previously disparate parts of the undergraduate experience. CBL demands change Agricultural & Biosystems Engineering

25 A CBL Model Voorhees et al, U.S. Dept. of Education, 2001 Agricultural & Biosystems Engineering Demonstrations Competencies Skills, Abilities and Knowledge Traits and Characteristics Assessment of Performance Acquired Skills, Abilities, and Knowledge Developed in the Learning Process Foundation ASSESSMENT Integrative Learning Experiences Learning Experiences

26 Implementing CBL  Avoid “extra” work for faculty  Classes already mapped to outcomes - Map competencies to classes  Identify key assignments for student portfolios  Help faculty learn the “language” of competencies  Provide common rubrics Agricultural & Biosystems Engineering

27 Implementing CBL with Technology  ePortfolios to develop and demonstrate competencies  On-line case simulations to simulate the workplace  On-line assessment for internships, alumni and in the classroom Agricultural & Biosystems Engineering

28 ePortfolio Implementation  Freshman Learning Community - introduce competencies - start collecting and reflecting on artifacts  Sophomore and Juniors seminars - Start ePortfolios - Feedback from instructor and advisor  Senior seminar - Final ePortfolio required for graduation - Assessed by faculty and professionals Agricultural & Biosystems Engineering

29 ABE ePortfolio System  Database for holding multi-modal artifacts, reflections, and assessments  Student-designed, student-owned interface to display artifacts Agricultural & Biosystems Engineering

30 On-line Case Simulations  Case Studies - Promotes competencies and reflection  English 314 (Technical Writing) - Communication - the weakest competency Agricultural & Biosystems Engineering

31 One Instance: Omega Molecular  Students are employees of consulting firm  Client: Omega Molecular - Engagement with virtual biotech organization - Analyzing and producing recommendations and communications Feeding the world, better. Agricultural & Biosystems Engineering

32 On-line Case Simulation Agricultural & Biosystems Engineering

33 Now What?  Our status today: - Students are collecting artifacts. - Pilot group is creating ePortfolios. - Faculty are designating key assignments and learning about CBL. - We’re assessing internships & alumni.  Just Completed: - Implement seminar classes (S’06) - Do a complete cycle (S’06) - Conducted ABET visit (F’06) Agricultural & Biosystems Engineering

34 Agricultural & Biosystems Engineering