1. Pharmacy Curriculum Outcomes Assessment (PCOA)
Moderated by:
Abby A. Kahaleh, BPharm, MS, PhD, MPH
February 16, 2016

2. Contact Information
Abby A. Kahaleh, BPharm, MS, PhD, MPH Curriculum SIG Chair (Phone)

3. Presenters Elizabeth A. Coyle, Pharm.D., FCCM, BCPS
Assistant Dean of Assessment
University of Houston College of Pharmacy
2. Tara Jenkins, RPh, PhD
Associate Dean of Academic Affairs
3. Justine Gortney, Pharm.D., BCPS
Clinical Assistant Professor
Wayne State University College of Pharmacy

4. Program Description
According to NABP, the Pharmacy Curriculum Outcomes Assessment (PCOA) is an effective assessment tool for colleges and schools of pharmacy to use for evaluating student performance in the curricula. PCOA facilitates a review of a pharmacy student’s performance from year to year in relation to the entire curriculum and compares the results to national data. The purpose of this program is to provide participants with expertise from administrators and faculty who used PCOA in their institutions.  

5. Objectives
Discuss the benefits of using PCOA as a non-high stakes APPE-readiness exam
Discuss the benefits of using PCOA as a high-stakes APPE-readiness exam
Compare and contrast different techniques of setting minimal competency when using PCOA
Design a remediation plan for students who do not meet minimal competency on PCOA

6. Presenter I Justine S. Gortney, Pharm.D., BCPS Director of Assessment,
Division of Pharmacy
Clinical Assistant Professor
Wayne State University,
Eugene Applebaum College of Pharmacy and Health Sciences

7. Pharmacy Institution’s Stakes Attached to Student Performance
Note: What was surprising to us was the lack of impact on student given the college dollars spent on the PCOA. This would tell us that the college’s major value and use of the PCOA is for program/curricular assessment. Colleges may value the PCOA for these reasons even more than individual student assessment.
Question: Since the PA profession has a similar exam, does a survey like this exist regarding its use?
Gortney JS, Bray B, Salinitri F. Am J Pharm Ed 2015; 79(9) Article 137.
N=38

8. Where We House PCOA: Professionalism Curriculum
Event
P1 Year
P2 Year
P3 Year
P4 Year
Professional Community Service
Student
Curriculum
Professional Meeting
Professional Event
Summative Evaluation
----
PCOA

9. Wayne State Operational Timeline
Education about Process
PCOA
Thursday & Friday
Results Receipt and Distribution
Students- August, November, December
Mid January
Distributed to Students and Faculty in April
Remediation (Student Development)
Evaluated by SEC March-April
Advisor meetings May; Plans due in late May/Early June
Rollout: P2s 2013, P3s 2014
Education about Process
P2 PCOA
Friday
Results of Exam
Students & Faculty:
Aug., Nov., December
Typically mid January (2013 was exception)
Distributed to students in April
Remediation
Evaluated by SEC Mar-April
Advisor meetings May; Plans due in late May/Early June

10. With Low Stakes, How Do We Incentivize Students?
Social Element- Breakfast the day of exam
High Performance on PCOA - Rewards
Recognition at graduation for top two
Letter of recognition from the Dean’s Office for top 10%
Overall examination scores above the college mean will be rewarded with 1% bonus on your final PPS grade
Poor performance: Student development plan

11. WSU Results Distribution Plans
Student Class Meetings
Faculty Meeting Rollout
Review overall class results
Review how to interpret PCOA scorecard
Distribute individual student scorecards
Distribute individual student development plans
P3 only: Review performance change from P2 and reflection
Provide comparison of the following:
Overall school results
Comparison with national
Comparison within class and across classes
Review timelines of student development plans
Publish results on dashboard:

12. Example of WSU P2 to P3 scorecard
Name P2 P3
Change in Score P2 to P3
Assess the magnitude of the change in PCOA
Confidence Rank
Overall_SS
343
369 26  -
Area_1_SS
377
418 41
Area_2_SS
338
357 19
Area_3_SS
348
329
-19
Area_4_SS
330
380 50
Overall Percentile 58
Area 1 Percentile 66 85
Area 2 Percentile 54 52 -2
Area 3 Percentile 56 33
-23
Area 4 Percentile 53 -1
Physiology 70 7
Biochemistry 44 38 -6 6

13. Past: Student Development Plans / Thresholds
Year
2013 P2a
2014 P2a
2015 P2a
2014 P3b
(2013 P2)
2015 P3 b
(2014 P2)
Basic Bio A B - E F G L
Pharm Sci I P Q H N O
SBAS C D J K M R S
Clin Sci T U V
4 student crossover per year
aTwo SD below WSU mean
bSame as P except Clin Sci (current P3 2 SD below P3 mean)

14. WSU Student Development Plan 2013-2015 (Student Driven)
Self assessment of the content area
How have you studied this content in the past?
What are your current study habits?
What do you perceive as a limitation to studying or understanding the specific content at hand?
Plan for improvement
How will you proceed forward to improve your knowledge and skills in these areas?
Generate a personal action plan that is practical and implementable
Describe the timeframe you propose to do this over
What resources will you use to do this?
Self-reassessment of the content area after plan for improvement implemented
What specific methods will you use to assess your progress? (eg. online quizzes in Access Pharmacy)

15. WSU Student Development Plans Modified for 2016
Cut-points modified
Agreed 1 SD below 3yr NABP composite based on curriculum map for P2s & overall score P3s
P2s for Basic Biomed (282) and PharmSci (280)
P3s Overall Score (303)
Revised Student Development Plan
Development/rollout being finalized
Faculty and content-structured
Timed during 6 weeks of first APPE block

16. How Have We Used This Data in WSU Curricular Assessment

17. Limitations We Have to Date
Using as low stakes; student motivation
Most of our data analysis to this point has been in P2s
PCOA with P1 GPA: r=0.50, p<0.01 (N=143)
PCOA with P2 GPA: r=0.53, p<0.01 (N=142)
Early curriculum, science-based classes showed low-moderate but significant correlations with PCOA scores
Lacking NAPLEX correlation data due to date of exam
Does not appear on surface to relate to risk for APPE rotation failure; APPE grading scale lacking variability

18. At WSU, How Do We Feel this Has Benefited our Students and Program
Provided a starting block for summative evaluations
Provided positive feedback for students regarding their knowledge
Provided specific assessment data to track our students’ knowledge and benchmark against others
Used specifically during curricular renewal process
Part of a triangulation of data driving “renewal”
Plan on using as a consistency check after “renewal”
ACPE self-study data
Higher Learning Commission (HLC) outcomes measurement

19. Presenters II and III
Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of Assessment University of Houston College of Pharmacy Tara Jenkins, RPh, PhD Associate Dean of Academic Affairs University of Houston College of Pharmacy

20. Utilizing the PCOA as a High Stakes Exam

21. Objectives UHCOP High-Stakes Exam History PCOA Pilot
Setting Cut-off Scores
Remediation
Questions

22. PCOA Project Team Elizabeth A. Coyle, Pharm.D., FCCM, BCPS
Catherine L. Hatfield, Pharm.D.
Gerida Brown
Tara L. Jenkins, RPh, Ph.D.
UHCOP Faculty

23. Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
Milemarker
Homegrown cumulative didactic exam
Started as high stakes for P3’s in 2000
Included questions from all 3 years of didactic curriculum
Written by faculty
200 case-based questions
Questions Angoffed
Mean of Angoff score from 200 questions had to be met for minimal competency
Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.

24. Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
Milemarker
P3’s had to meet minimum competency on Milemarker III to advance to APPEs
Average pass rate from was 99%
P1 & P2’s also took Milemarkers I & II
Formative with remediation and points towards Milemarker III
New cases & validated questions were hard to accumulate
Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.

25. PCOA Pilot January of 2013 P3 students All required to take
Formative information only
P3 students still had to take Milemarker III in April 2013

26. Exam Comparisons

27. Results Comparison
Milemarker III
PCOA

28. PCOA Benefits
External measurement and benchmark of our students’ performance in pharmacy curricula
Psychometrically sound questions written by content experts from colleges of pharmacy around the country
Ability to trend results longitudinally
Does not require faculty time to analyze and write capstone questions

29. Identify Cut off Score
For P3 PCOA

30. X Tuan and R Michel. R & D Connections No 16. Sept. 2011
Which Scores to Use?
Percentile rank — the percentage of scores in a distribution that is equal to or lower than the measured score.
Raw score — Total number of raw points a test taker receives based on the number of questions answered correctly; typically, for example, 20 correct answers means a raw score of 20
Scaled score — Scores that have been mathematically transformed from one set of numbers (i.e., the raw score) to another set of numbers in order to make them comparable in some way — for example, across different editions, or “forms,” of the same test
X Tuan and R Michel. R & D Connections No 16. Sept. 2011

31. National Scaled Scores
700
Year 1 2 7 3 4 5 6 8
50%ile
National Scaled Score (normed reference sample)
361
50%ile
357 (scaled normed referenced score change)

32. UHCOP Cutoff Score * NSS vs. 10% below NSS vs. 1 or 2 STD
* PCOA Cutoff score for P3’s is 1 STD from the NSS scale score. The STD will be calculated from the PCOA student scores each year.

33. Number of Remediating after 1st PCOA
3 Year Comparisons
P3 Class (N)
Class Percentile
National Scaled Score
Class Ave Scaled Score
Standard Deviation
Passing Score
Number of Remediating after 1st PCOA
Number (%) < NSS
Number ≥ 90th percentile
Number ≤ 20th percentile
P3 Class 2013 (N=109)
75th
361
405 41
N/A (pilot)
N/A (pilot)
14 (12.8%) 28 1
P3 Class 2014 (N=113)
56th
388
325
14*
45 (39.8%) 10 11
P3 Class 2015 (N=109)
65th
357 47
310 2
23 (21.1%) 22 3
* If we would have used 1 STD cutoff, 11/113 would have remediated

34. High Stakes Exams & Academic Performance
Coyle EA, Hatfield CL, Cottreau JM, Brown G, Smesny A. Poster (July 2013) AACP Annual Meeting, Chicago, IL

35. Remediation What kind of remediation do you use? Retest PCOA?
Homegrown exam?
Tailored remediation?
How do remediated students perform on rotations?
How often do you remediate?

36. * Not in PCOA normal testing windows

37. Conclusions
Adapting PCOA as a high stakes assessment was an easy transition at UHCOP
Minimum competency should be set off of the scaled score and not percentile or raw score
A clear remediation process is imperative to close the loop for high stakes exams

38. Education about Process
Q&A
Education about Process
P2 PCOA
Friday
Results of Exam
Students & Faculty:
Aug., Nov., December
Typically mid January (2013 was exception)
Distributed to students in April
Remediation
Evaluated by SEC Mar-April
Advisor meetings May; Plans due in late May/Early June

39. Discussion Questions
1. Compare and contrast the PCOA exam with other assessment instruments 2. What are the advantages/disadvantages of various remediation plans? 3. Describe examples of successful student/faculty development plans 4. What are the steps for closing the assessment feedback loop based on the PCOA results?

40. Thank You!