1. What to Expect When You’re Expecting
a Visitor from ABET
August 22, 2016

2. Today’s Learning Objectives
At the end of this discussion, you should be able to:
List your responsibilities and tentative schedule for the ABET visit
Describe our previous assessment process
Describe our future assessment process (and why we changed it)
List the primary responsibilities of each group in the assessment process
Develop a rubric for assessing the achievement of a learning outcome

3. The Big Question Do we have a good process that:
Measures student performance on established outcomes
Assesses the results of these measurements to identify real or potential issues
Responds by identifying and taking corrective action aimed at improving performance on outcomes
Closes the loop by returning to step 1 to measure and reassess the outcomes
There should be a similar process for program objectives.
Perfection is the goal of the process, but not the expectation of ABET.

4. ABET Visit Schedule & Responsibilities
ME Program Evaluator (PEV): Prof. Imin Kao, Stony Brook University
Sunday, October 2, 2016
Activity: Review of course materials, tours of facilities
Responsible: Chair and ABET coordinator to meet with PEV
Responsible: Faculty to prepare materials, clean up labs, give lab tours
Monday, October 3, 2016
Activity: Meetings with faculty, staff and students (to be arranged)
Responsible: faculty, staff and students
Tuesday, October 4, 2016 (first half of day)
Activity: More meetings with faculty, staff and students (to be arranged)

5. Purpose of Faculty Meetings with Visitor
Evaluate the overall quality and attitude of the faculty regarding undergraduate education
Passion and enthusiasm about the program and about our students is important, but be honest
Complaining / whining about known issues does not help, but these should be acknowledged
It’s expected to discuss our challenges, but also be prepared to discuss what we are doing to address those challenges
Evaluate the faculty’s understanding of the
Program Educational Objectives (PEOs) and how they are determined and assessed
Student Outcomes and how they are determined and assessed
Provide examples of improvements (closing the loop)!
A cheat sheet will be provided

6. Enrollment Growth & Teaching Capacity
Page 3 of our ABET self-study report:
“The above activities have been accomplished in an environment of extreme enrollment growth in the BSME program. Since 2010, total enrollment in the program has approximately doubled, while the number of faculty has stayed constant. The teaching capacity has been increased through the hiring of three full-time teaching specialists, each of whom teaches three sections per semester. Additional tenure-track faculty and teaching specialist hires are expected in the next few years.”
Update:
Two teaching specialists and one faculty member recently left the program
This summer we hired three teaching specialists and one part-time instructor as a result of additional college support

7. Problem Solving Skills
Page 49 of our ABET self-study report, referring to outcome assessment results:
“… the low scores are largely due to deficiencies in the process of solving for internal loads and stress states at a point. This issue needs further attention.
Informal discussions with other ME 471 instructors confirmed the weakness noted above, which is also consistent with the most common mistake in the assessment performed in ME 361. As a result, discussions are underway to modify the series of mechanics and design courses to improve these skills.”
Response:
A new approach to teaching ME 222 will be attempted during fall 2016, involving aspects of mastery learning
The mechanical design courses ME 371 and ME 471 are being restructured
Recitation sections in core courses are being introduced next year to emphasize problem solving skills and practice

8. Our Objectives
Program Educational Objectives (PEOs) describe the expected accomplishments of our students several years after graduation
Our graduates will:
Be competent and ethical engineers practicing in a diverse range of activities
Use their mechanical engineering education as a stimulus for personal and professional growth
Be recognized for their capability, creativity, and application of knowledge
Be independent and critical thinkers who identify problems and develop effective solutions

9. Assessment of Our Objectives
1. Groups are surveyed:
Faculty, Students, Alumni,
Board of Advisors,
Capstone Design Sponsors
2. Processed survey data is reviewed by the MEUCC
3. MEUCC reports to faculty with recommended changes to the PEOs or to the assessment process
4. Faculty discusses and approves changes to the PEOs or to the assessment process
Administered by the Associate Chair for Undergraduate Studies

10. Our Student Outcomes
Student outcomes describe the expected capabilities of our students at graduation
Our graduates will have:
(a) An ability to apply knowledge of mathematics, science, and engineering
(b) An ability to design and conduct experiments, as well as to analyze and interpret data
(c) An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability
(d) An ability to function on multidisciplinary teams
(e) An ability to identify, formulate, and solve engineering problems
(f) An understanding of professional and ethical responsibility
(g) An ability to communicate effectively
(h) The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context
(i) A recognition of the need for and the ability to engage in life-long learning
(j) A knowledge of contemporary issues
(k) An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice

11. Assessment of Student Outcomes
Currently, we use a sampling approach, which takes place every 2-3 years:
Courses are identified that emphasize one or more student outcomes
For each outcome emphasized, the instructor identifies an assignment to measure that outcome as well as a level of minimum competency (e.g., a score of 65% on this assignment)
Measurement results are provided to the MEUCC, which sets a metric goal (e.g., > 80% of students reach above minimum competency)
The Board of Advisors and the MEUCC each assign a grade based on achievement of the metric goals
Areas of weakness are identified by the MEUCC and discussed with instructors in that area. Instructors make any necessary modifications and then reassess the outcomes.
In addition, a graduating senior focus group survey is conducted in the spring semester of each year. Data and comments are evaluated by the MEUCC.

12. NEW Assessment of Student Outcomes
In the future, we will use rubrics to assess student outcomes:
Courses are identified that emphasize one or more student outcomes
For each outcome emphasized, the instructor identifies an assignment to measure that outcome and uses an established rubric to determine what percentage of students meet or exceed the minimum competency goal
Measurement results are provided to the MEUCC, which sets a metric goal (e.g., > 80% of students reach above minimum competency)
The Board of Advisors and the MEUCC identify areas of weakness and discuss possible ways to improve student performance on that outcome
Instructors make any necessary modifications and then reassess the outcomes
In addition, a graduating senior focus group survey is conducted in the spring semester of each year, and survey data from all graduating seniors is collected each year. Data and comments are evaluated by the MEUCC.

13. NEW Assessment of Student Outcomes
In the future, we will assess student outcomes on a continuous basis (instead of every 2-3 years)
Measure and assess approximately one-third of items of evidence each year (in Sets), with most outcomes represented in each set
More continuous improvement process, with more focus on each outcome
Reduced chance that a cohort of students will pass through the program without being assessed in each outcome
Proposal: MEUCC writes a yearly report summarizing assessment results
Measure
Assess
Improve
Fall 2017 / Spring 2018
Set 1
Previous changes
Fall 2018 / Spring 2019
Set 2
Fall 2019 / Spring 2020
Set 3 + PEOs
Fall 2020 / Spring 2021
Fall 2021 / Spring 2022
Fall 2022 / Spring 2023
continue …

14. Rubrics
A rubric defines the expectations for an assignment by listing the criteria and describing levels of quality
We have decided to use three (3) levels, or Degrees of Achievement:
Deficient – Does not meet minimum competency
Satisfactory – Meets expectations
Superior – Exceeds expectations
Levels 2 and 3 meet or exceed the expected minimum competency, while level 1 does not
Need to define explicitly the expectation for each level
For each outcome, there may be multiple Performance Indicators, each representing an individual knowledge or skill being measured, or a particular part of the outcome

15. Example – Part 1 (the problem)

16. Example – Part 2 (first indicator)
Rubric for Outcome (a): An ability to apply knowledge of mathematics, science, and engineering Course: ME 361 Semester: Spring 2016 Instructor: Geoff Recktenwald Tool: Final Exam, Problem 5 No. students: 72
TOO GENERIC
Degree of Achievement
Performance Indicator
Deficient
Does not meet Minimum Competency
Satisfactory
Meets Expectations
Superior
Exceeds Expectations
Application of fundamental principles of science and engineering
Identifies which fundamental principles govern the process or system. Unable to carry through from knowing principles or theory to generating solution.
Has good knowledge of governing principles/ theory. Generally is able to apply correct principles to problem solutions, but may make inappropriate assumptions or simplifications.
Combines scientific and engineering principles/theory to formulate correct solutions to engineering problems.
Enter grade range, if used.
e.g.: 0-60 / /
0-5 on part b
6-9 on part b
10 on part b
Enter number of students
in this range
2 (2.8%)
13 (18.1%)
57 (79.2%)

17. Example – Part 3 (second indicator)
Rubric for Outcome (a): An ability to apply knowledge of mathematics, science, and engineering Course: ME 361 Semester: Spring 2016 Instructor: Geoff Recktenwald Tool: Final Exam, Problem 5 No. students: 72
COULD BE BETTER
Appropriate mathematical techniques to achieve solutions to engineering problems
Recognizes what mathematical techniques are required to reach a solution, but can only apply correct mathematical techniques with guidance or applies them incorrectly.
Applies mathematical principles to obtain analytical or numerical solution to model equations. Generally chooses an appropriate method, but maybe not best method and/or without analysis of results.
Has strong understanding of mathematical techniques and applies “best” methods to achieve correct solutions.
Enter grade range, if used.
E.g.: 0-60 / /
0-16 on problem 5
17-21 on problem 5
22-30 on problem 5
Enter number of students in
this range
7 (5.6%)
16 (22.2%)
49 (68.1%)

18. Your Turn to Develop a Rubric
Divide into groups of 3-5 based on area of specialization
Select one of the courses listed below and develop a test problem and a rubric to assess achievement of either outcome (a) or (e) – your choice.
Dynamics, Systems and Control (DSC)
ME 361
ME 451
ME 461
Solid Mechanics, Design & Manufacturing (SMDM)
ME 391
ME 471
Fluid-Thermal Science and Engineering (FTSE)
ME 332
ME 410

19. Test Problem for Outcome (____): Course: ME ______

20. Rubric Template – fill the empty cells
Rubric for Outcome (____): Course: ME ______
Degree of Achievement
Performance Indicator
Deficient
Does not meet Minimum Competency
Satisfactory
Meets Expectations
Superior
Exceeds Expectations
Enter grade range, if used.
e.g.: 0-60 / /
Enter number of students
in this range X Y Z