1 In Search of Closing the Continuous Improvement Loop: Outcomes Assessment of Student Knowledge M. Suzanne Clinton Associate Dean, School of Business Lisa K. Wolfe MBA Graduate Coordinator

2 Assessment Identify what we should do Measure the quality and effectiveness of our work Document what we are doing Are we doing what we should be doing? (Advisory Board) Plan how to IMPROVE what we do A five-step continuous process IESIES

3 Step One Develop common syllabi for courses taught by various faculty Identify approximately ten core competencies for each course A core competency is the knowledge that a student must acquire to be able to complete the course successfully

4 Step Two Match core competencies to specific exam questions, writing assignments, projects, etc. Set up spreadsheets to simplify data entry Establish appropriate benchmarks

5 Step Three Collect Data –Perceived knowledge –Actual knowledge Compare Perceived and Actual measures to benchmark Document, document, document…

6 Step Four Evaluate comparisons to determine –If changes are needed –What specific changes are needed Document planned changes Implement changes Document actual changes Document results (this takes us back to Step Three)

7 Step Three REVISITED Collect Data AGAIN –Perceived knowledge –Actual knowledge Compare Perceived and Actual measures to benchmark AGAIN Document, document, document…

8 Step Four REVISITED AGAIN, evaluate comparisons to determine –If changes are needed –What specific changes are needed Document planned changes AGAIN Implement changes AGAIN Document actual changes AGAIN

9 Step Five Repeat Steps Three and Four each time a course is offered Develop trend lines Periodically review syllabi and core competencies Review syllabi and core competencies with Advisory Board As exams, written assignments and projects are revised, so too will be the core competencies and their measures The entire process is continuous

10 Examples of Core Competencies for Statistics Core Competency #1 The student will be able to calculate the mean, median, mode, standard deviation and variance Core Competency #2 The student will be able to calculate relative frequency, conditional probability and joint probability Core Competency #3 The student will be able to calculate permutations and combinations and be able to apply Bayes Theorem Core Competency #4 The student will be able to calculate probabilities associated with binomial distributions, Poisson distributions and the hypergeometric distribution

11 Examples of Perceived Survey 1.Your ability to calculate mean, median, mode, standard deviation and variance. No Understand Understanding /____/____/____/____/____/____/____/ Very well T he student will be able to calculate relative frequency, conditional probability and joint probability No Understand Understanding /____/____/____/____/____/____/____/ Very well The student will be able to calculate permutations and combinations and be able to apply Bayes Theorem. No Understand Understanding /____/____/____/____/____/____/____/ Very well The student will be able to calculate probabilities associated with binomial distributions, Poisson distributions and the hypergeometric distribution No Understand Understanding /____/____/____/____/____/____/____/ Very well

12 Sample Test Questions for Core Competency #1 XP(X) Calculate the mean of this distribution. 2.Calculate the standard deviation of this distribution.

13 Sample Test Questions for Core Competency #2 1.Find the entire area under the standard normal curve (the Z curve) which lies to the left of Z = Find the probability that a potential investor will purchase shares in at least one of the two funds. 3.What is the probability that a company employee believes that he/she would be financially secure if he/she lost his/her job?

14 Sample Test Questions for Core Competency #3 1.How many combinations of 3 can be chosen from 75? 2.A busy executive has to meet with five production managers during the day. The executive needs to decide in which order to see the managers. How many different orderings can the executive choose?

15 Sample Test Questions for Core Competency #4 1.In a shipment of 15 room air conditioners, there are 4 with defective thermostats. You are going to randomly inspect 5 of the air conditioners in this shipment. What is the probability that of the 5 air conditioners you choose to inspect 3 are good and 2 are defective? 2.Make a continuity correction and use the normal approximation to the binomial to determine P(X = 50). Suppose that X has a binomial distribution with n = 100 and p =.5.

16 Sample of Core Competency Calculations Core Competency #1Core Competency #2 Exam # & Question #E1 Q1E1 Q2E1 Q3E1 Q4 Max. Points5510 Student #15510 Student # Student #35510 Student #40510 Total points Total points/# of students Average Score by Question75%100% 75% Average Score by CC88%

17 Program Objectives Program Objectives – educational purposes developed from an analysis of consumer needs –Based on Camerons mission –Clearly stated –Relatively stable over time (2-3 years) –Operationalized (put in measurable terms) –Basis for program planning and strategy development –Faculty expectations about what is to be learned –Criteria for program evaluation –Basis for curriculum development and faculty selection and development

18 Program Strategies Describe how program requirements ensure student learning experiences Clearly state which program components serve which program objectives Ensures appropriate attention is given to each objective Assessment results are used annually to evaluate and revise program strategies

19 Assessment Strategies Measurement options to be used for each objective (exam, group assignment, etc.) Students who are targeted Time lines for data collection and analysis Methods of analysis to be used on the assessment data Decision criteria to be used to formulate conclusions from the results of the analyses

20 Assessment Model Essential part is the evaluative conclusions regarding student strengths and weaknesses related to program strengths and weaknesses Assessment results are used annually to evaluate each data collection and analysis method Appropriate modifications are included in the assessment plan for the next cycle

21 Display of Assessment Data Display the tables and narrative to describe assessment measures taken and number of students participating Measures must be reliable and valid Measures should tie directly to program objectives

22 Analysis of Achievement of Objectives Assessed Describe how the assessment data indicated program quality Use multiple measures of program outcomes Discussion of instructional changes that occurred and their impact on the quality of the program should be included

23 Action Plan List specific actions to be taken as a result of the assessment process Indicate –Modifications proposed –Time line for implementing changes –Resource implications of the changes Changes submitted to curriculum committee must be supported with assessment data Identify instructional changes that will improve program outcomes

24 Conclusion Discussion Questions Thank you for coming!