1. Certificate vs. Certification: What’s the difference, why does it matter? PETD SIAC Discussion Notes PRESENTED BY Michelle Flewell, CPLP VP, eLearning & Informatics BioSoteria, Inc. mflewell@biosoteria.com (919) 563 1667 mflewell@biosoteria.com

2. Certification vs. Certificate There are two paths to creating and maintaining a certification or a certificate program if you follow accreditation standards (set forth by ICE or NOCA): Certification Voluntary process by which a non-governmental body grants time-limited recognition and use of a credential to individuals who have demonstrated that they have met predetermined and standardized criteria for required knowledge, skills, or competencies To retain the credential, certificants must meet requirements for renewal Independent of a specific class, course, or other education/training program AND of any provider of classes, courses, or programs Assessments are NOT designed to evaluate accomplishment of the intended learning outcomes of a specific class, course, or other education/training program or event Certifier is NOT the sole provider of any education or training that may be required for certification Primary focus on assessment Links assessment to predetermined standards for knowledge, skills, or competencies Assessment-based Certificate program (ABC or ABCP) The provision of education/training, with validated assessment(s) to confirm that participants have achieved the intended learning outcomes (vs. an attendance-only “Certificate” that meets no accreditation standards and does not assess if participants achieved the defined learning outcomes) No renewal: certificate holders complete the training, pass the exam, and then hold the certificate Primary focus on providing education/training and a standards-based assessment to certify that a candidate passed a knowledge- or skills-based exam based on the key learnings from the content taught in the program Links assessment to learning outcomes of a particular education/training program Assessment process does not have to be separated from the education/training program or provider

3. The Path to Developing an Accredited Certification [GREEN TEXT: Often executed by/with help of certification services vendor/provider] STEP ONE: Establish need for certification and set-up infrastructure to develop and maintain it Organizations seeking to develop a certification typically form a separate “credentialing institute” or a credentialing board that is firewalled/separate from the main organization (particularly if the organization offers training presently); organizations are often membership-based professional organizations who want to offer certification to their membership, but the standards do not require the organization to be membership-based or non-profit (though many are), they just can’t “mingle” education/training with certification The credential institute/board or organization (if they haven’t formed a credential group yet) typically conducts a Financial Feasibility study and/or Market Research (Survey, Focus Groups) to document need for credential, what it would include, investment required to develop and sustain, and if there is the “market” to support it long-term (through exam fees, etc.) If the decision is made to proceed, an organization would need to secure funding to develop certification program, and determine how they will fund maintenance of the program (a private corporation can “donate” or even loan $$ to develop a certification program, but governance remains with the credentialing body); a financial plan of sustainability is required by the standards to accredit the certification program

4. STEP TWO: DEVELOP CERTIFICATION STANDARDS, EXAM, AND PROGRAM Perform Role Delineation Study (job analysis) – Identify tasks performed by practitioners and knowledge required to perform the tasks – Determine how often role delineation needs to be repeated to ensure test remains relevant to current practice Develop the test blueprint (similar to a content outline, and it should be based on the job analysis), which dictates content domains, numbers of questions in each domain, weighting, etc. Write items for the exam (and train item writers, if non-SMEs, they need training on the content to be able to write items, if non-ISDs/writers, they need training on how to write test items); if non-SMEs write items, SMEs conduct content review; once content is deemed accurate, the test items go through technical review by psychometric experts; items are further refined Construct draft exam according to the rubrics in the test blueprint (multiple “forms” or versions are developed at this stage) Hold Passing Standard Setting meeting(s) with subject matter experts to determine the cut score for the minimally competent candidate (MAC) Pilot the exam Psychometrically analyze test/item performance data, refine exam as necessary Develop all systems, infrastructure, and marketing needed for the program: website, marketing plan/collateral, certification documentation, testing systems, certificate generation process, records/document/candidate management system, eCommerce, etc. (this step can be done in parallel with test development)

5. Launch certification program (without accreditation), begin collecting test data, building needed “n” of test takers (typically 500, but “low volume” programs may run the program for one year and then seek accreditation) Test refinement does not stop at launch, it is ongoing, particularly during the first year or with the first 500 test takers; as you collect data on your assessment overtime, you examine the reliability and validity of the assessment and the individual items, and continuously improve it and build your pool of alternate questions and test forms; the standards for accreditation address this and recommend that psychometric principles are followed File intent to apply for certification accreditation with ICE or ANSI Apply to ICE or ANSI for certification program accreditation; the application materials document in great detail that your program meets accreditation standards STEP THREE: LAUNCH PROGRAM AND APPLY FOR ACCREDITATION

6. Process to Set-Up (accredited) ABCP OPTIONAL: Conduct job analysis and/or market research survey Develop Certificate Program Policies and Procedures Documentation Develop training/educ. course(s) and Validated Assessment (see next slide) OPTIONAL: Accredit Courses for continuing education Apply to become IACET authorized provider (AP), accredit courses for IACET CEUs AND/Or offer CEUs through other avenues (ACCME, ACPE, nurses orgs) Develop testing platform and infrastructure necessary to sustain the program Launch program Ongoing activities to sustain the program Continue to collect test data, update assessment exam at scheduled intervals for quality improvement Update assessment to reflect course updates (as needed) Apply for ABCP program accreditation through ICE or NOCA (when available in 2010)

7. Developing the Assessment for ABCP Determine passing standard (“cut-score”) Conduct passing studyAnalyze results/set cut-score Analyze pilot results Analyze the test overall and individual itemsRevise assessment Pilot test the assessment Revise/Finalize Assessment for Pilot Analyze all items Technical reviewSME Review Collect/Develop “items” (e.g., test questions) Collect existing final assessment questionsDevelop new questions if gaps exist Define purpose and specifications Create assessment development plan Develop exam blueprint (content, weight, cognitive requirements)

8. Which Credentialing Agency? ICE/NOCA vs. ANSI Both ANSI and ICE have established standards and paths to accreditation for certification programs Some say… ANSI has more international recognition, but ICE develops their own standards, which are more rigorous; ICE has strong legacy in healthcare-related certifications For ABCPs: ICE just released their ABCP standards, they are not yet accredited; that process will get underway in 2010 ANSI has an accredited program already (endorses other group’s standards, in this case, ASTM); their ABCP accreditation program is in pilot phase, pilot programs already selected (FDA has a program in the pilot phase with ANSI); they expect to have ABCP accreditation online and accepting applications in March 2010 (cost to apply not announced yet) – https://www.ansica.org/wwwversion2/outside/CAPgeneral.asp?menuID=212 https://www.ansica.org/wwwversion2/outside/CAPgeneral.asp?menuID=212 ANSI allows a quasi-credential in associate with ABCP: “certificate holder,” e.g. candidates who complete a PV ABCP could use a designation, such as CH-PV (certificate holder in pharmacovigilance); there are rules to this, including additional tracking and administration measures (standards are vague on this point)

9. Disadvantages of Non-accredited Certificate or Certification programs There are “non-accredited” certification and certificate programs in our field currently There is typically no validated, standards-based assessment of competency or knowledge in unaccredited programs (e.g., certificate programs where the only requirement to earn the certificate is attending conferences or completing courses, with no assessment of knowledge or competency) There are also certification programs (a few in drug development even) where no nationally/internationally recognized standards were followed to develop the assessment exam; therefore, individual questions or the entire exam may (or may not) accurately assess if the candidate possesses the knowledge and skills necessary to perform that job (the standards of accreditation are designed to ensure validity, relevance, reliability, etc.) A certificate or certification provider can legally follow their own rules or none at all, quality can vary, and they could change or drop the program at any time; the accreditation process in part validates that a provider follows good practices and possesses sufficient resources for ongoing maintenance and support of the program as condition for accreditation, giving certificate seekers more confidence in the program An unaccredited program may have no link to performance- or competency-based standards; programs that follow the accreditation process must follow rigorous standards for program and test development and validation that are tied to specific job-based knowledge, skills, and performance

10. Questions to consider RE Certification and Certificate Programs in Pharmacovigilance What value does certification or a certification program in pharmacovigilance (PV) have to stakeholders, such as: – Individuals (in the job currently, or job seekers)? – Industry (pharma/biotech, CROs, PV service providers)? – Regulatory authorities? – Professional societies (DIA, ISoP, ISPE)? – Educators (universities) and training providers? Which is needed: PV Certification or PV ABCP? If TRAINING is the real need, then an ABCP program or programs might meet that need better, but if STANDARDIZATION in competency in the practice of pharmacovigilance is needed across the board, than a certification might best meet the needs of various stakeholders (some argue that both are needed) Would industry hire candidates who have completed an ABCP but have little to no OJT or experience? Would a corporation offer ABCPs to their employees/contractors? Will regulators ever require certification for those who perform pharmacovigilance activities? Who would develop, support, and maintain a certification program in PV?

11. More Info For more info on the two major credentialing agencies: – Institute for Credentialing Excellence (ICE, formerly NOCA) website www.credentialingexcellence.org)www.credentialingexcellence.org – American National Standards Institute (ANSI) www.ansi.org. www.ansi.org – Both ICE and ANSI offer paths to accreditation of certification and assessment-based certificate programs (ABCP starting in 2010*) *ICE’s ABCP accreditation program development is advancing; they have formed a council (on which I was appointed) to develop policies and procedures defining the governance and operations model for the ICE accreditation services program for ABCPs