1. Standard 4 Breakout Session Inspection Audit Program Russell Lilly Manager, Manufactured Food Program Missouri Department of Health and Senior Services Wednesday, March 12, 2014 8:00 – 10:00 Elm Fork I Patrick Kennelly Chief, Food Safety Section California Department of Public Health Peter Haase Director, Bureau of Food Safety and Inspection Wisconsin Department of Agriculture, Trade and Consumer Protection Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection Angela Kohls Standards Implementation Staff FDA ORA, Office of Partnerships

2. Agenda Strengthening Compliance and Enforcement Welcome and Introductions Current Status of Standard 4 Implementation Presentations on Implementing Standard 4 Q & A/Group Discussion Summary and Challenge

3. Housekeeping Honor time schedule Stay on topic Get input from everyone Be willing to share Respect different program cultures Do not reference specific audits Be creative, ask questions – parking lot Facilitators – responsible for process Participants – responsible for content

4. Session Objectives Generate a list of resources, best practices, lessons learned and challenges/recommendations for developing, implementing and monitoring an inspection protocol. (Group Exercise Instructions) 1. Listen to presentations 2.Discuss and list suggestions on flip charts 3.Share your group’s suggestions with room

5. Standard Four

7. Breakout Session Focus Elements 4.2b &c, 4.3b & c, and 4.4b & c Implementation of the 3 Types of Audits – Field Inspection – Inspection Report – Sampling

8. Standard 4: Field Inspection Audits Patrick Kennelly, Chief Food Safety Section California Department of Public Health March 12, 2014

9. Field Inspection Audits Quality assurance review to assess the effectiveness of its inspection program. – 2 audits of each inspector every 36 months – Conducted by a qualified auditor – Documented on Audit form that is equivalent to the FDA Contract Audit form – Need an 80% audit rating to be considered a successful audit

10. Field Inspection Audits Then and Now – Better Defined Process – Uniform Documentation – Ability to Compare Ratings Across the Program

11. Field Inspection Audits

12. II. INSPECTION OBSERVATIONS AND PERFORMANCE (Continued) 6.DID THE INSPECTOR DEMONSTRATE THE ABILITY TO DISTINGUISH BETWEEN SIGNIFICANT VERSUS INSIGNIFICANT OBSERVATIONS AND ISOLATED INCIDENTS VERSUS TRENDS? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement) 7.DID THE INSPECTOR REVIEW AND EVALUATE THE APPROPRIATE RECORDS AND PROCEDURES FOR THIS ESTABLISHMENT’S OPERATION AND EFFECTIVELY APPLY THE INFORMATION OBTAINED FROM THIS REVIEW? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement) 8.DID THE INSPECTOR COLLECT ADEQUATE EVIDENCE AND DOCUMENTATION IN ACCORDANCE WITH STATE PROCEDURES GIVEN THE NATURE OF THE INSPECTIONAL FINDINGS? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement)

13. Field Inspection Audits Things to Consider – How do you select facilities for staff audits? – What constitutes a “qualified auditor”? – Do you want to vary or alternate auditors for each staff? – How do you address individual “Needs Improvement” Ratings?

15. Field Inspection Audits Scoring across the spreadsheet identifies areas of “Needs Improvement” ratings within a rated category. – Need 80% “Acceptable” ratings within each rated category across the program to be in conformance. – How do you address Non-Conformance ratings?

16. Field Inspection Audits Benefits – More Knowledgeable Workforce – Improved Inspection Procedures – Identifying Training Gaps – Findings Drive Quality Improvement

17. Contact Information Patrick Kennelly, Chief Food Safety Section California Department of Public Health (916) 650-6598 pat.kennelly@cdph.ca.gov

18. Inspection Report Audits Russell Lilly Manufactured food program manager Missouri Department of Health March 12, 2014 8:00 AM

20. Because… I had three inspectors and reviewed every inspection report It is wildly prescriptive – At least half of the widgets don’t apply If the shoe doesn’t fit it is uncomfortable

22. Our situation changed 18% of 22 inspectors A quality assurance program becomes necessary and valuable

23. We took the money… We owe an honest effort to comply with the standard

24. We did it… My best advice – Hire someone smart and assign it to them

27. Was it worth it? I think so – We have sent a number of “corrective emails” Mark this here not there Be sure to put your “time out” on the form Turn in your reports on time

28. I would still like To have a quality system designed from the ground up to fit our program

29. Contact Information Russell Lilly Manufactured food program manager Missouri Department of Health Email: russell.lilly@health.mo.gov

30. Steps to Implementing Standard 4 Sample Collection Record Audit Peter Haase Director, Bureau of Food Safety and Inspection Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection

31. Sample Collection Record Audit Outline Introduction and Challenges Steps to customization Conclusion Documents on flash drive

32. Intro: Wisconsin Department of Ag Trade & Consumer Protection Enrolled in MFRPS in 2007 Challenges: Full Standard 4 Implementation Planned for 2015 – Phase III Contract Audit since 2011 Currently Piloting Non-Contract Field Inspection Audit Inspection Report Audit 2013 SCR Audit 2014

33. Intro: Wisconsin Department of Ag Trade & Consumer Protection Current Surveillance Sampling Program for Manufactured Food Establishments >Sampling at 2,400 establishments annually: – 2 product samples and – 5-10 environmental swabs at each routine inspection

34. Intro: Wisconsin Department of Ag Trade & Consumer Protection Standard 10 Full Conformance – ISO Certified since 2007 Sampling Policies and Procedures Two Sample Collection Record Forms, Food and Dairy

36. Challenges of Auditing our Sample Collection Records: Step 1 Test drive of Appendix 4.7.

37. Challenges of Auditing our Sample Collection Record (SCR): Step 1 Results of Test drive of Appendix 4.7. – Findings: 4.7 is more easily managed in table format FDA Criteria: – Some not applicable or redundant for our reports – Very general: guidance specific to our SCRs required

38. Audited against our Policy Added guidance specific to our Sample Collection Record form and procedures

39. Audited against our SCRs

40. Customized 4.7 Form: Added guidance specific to our Sample Collection Record form and procedures

41. Piloted new form and guidance by auditing 71 SCR’s from Calendar 2012 Results: – Opportunities for improvement: Use of hash marks and down arrows, form completeness Redundant criteria: – SCR and sample always together

42. Challenges of Auditing our Sample Collection Records (SCR): Problem detected but not an audit criterion: Use of Correct Form Substituted alternate criteria: Use of correct form

43. Challenges of Auditing our Sample Collection Records (SCR): For redundant criteria, combined and substituted alternate criteria

44. Sample Selection (included in SOP for Standard 4) 75 SCR’s chosen per calendar year from MF establishments for auditing: – at least 1 per sampler, – random with regard to date, establishment

45. SOP for Standard 4 Cont. Entering and tabulating results: Excel Spreadsheet 1 for acceptable, 0 for needs improvement

46. SOP for Standard 4 Cont. SCR Audit completed in March for previous calendar year Needs Improvement Factors (<80%) become training topics at staff meetings Other improvements can be made immediately upon discovery: – Already identified from SCR 2013 audit: “Foodborne Outbreak” reason for sampling on new Dairy SCR

47. Conclusion – Adapting 4.7 to our Sample Collection Record Meaningful results for quality improvement – Documents on AFDO-MFRPA website: Lab SOP #2060-Receipt and Integrity of Samples SCR Policy and Procedures SCR Forms (Food and Dairy) Our Audit Form 4.7 and Data Entry/Calculations Sheet Description of sample selection and procedures for Standard 4 including Sample Collection Record Audits

48. Contact Information Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection Email: skya.murphy@wi.govskya.murphy@wi.gov

51. Contact Information Patrick Kennelly, Chief Food Safety Section California Department of Public Health Email: pat.kennelly@cdph.ca.gov Russell Lilly Manufactured Food Program Manager Missouri Department of Health Email: russell.lilly@health.mo.gov Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection Email: skya.murphy@wi.gov Peter Haase, Director Bureau of Food Safety and Inspection Wisconsin Department of Agriculture, Trade and Consumer Protection Email: Peter.Haase@Wisconsin.gov