Welcome, Examiners! Washington State Quality Award Return Examiner Training 2009

WSQA Return Examiner First things first! THANK YOU! WSQA’s mission: Improving the way we live, learn, and work in Washington by helping organizations improve through the use of the Baldrige Criteria for Performance Excellence

WSQA Return Examiner Objectives For This Class Review Changes to the Baldrige Criteria Strengthen Skills in Areas of Opportunity Review Resources for Examiner Software

WSQA Return Examiner Agenda: 1.WSQA organization overview 2.Baldrige criteria changes 3.WSQA ethics review 4.WSQA process Full Vs Lite 5.Review process 1.Individual review worksheets 2.Consensus process 3.Improvements 6.Examiner Software 7.Summary

For a Successful Course… Be an active participantBe an active participant Give only constructive feedbackGive only constructive feedback Be courteous to your colleaguesBe courteous to your colleagues Honor diverse opinionsHonor diverse opinions Seek to understandSeek to understand Ask questionsAsk questions Have fun!!Have fun!!

WSQA Return Examiner Introductions Name, organization, what you do Questions and expectations? What do you like to do when you’re not working?

WSQA Return Examiner What is WSQA? Created by State law to: Promote excellence Recognize achievement Educate and train Patterned after the Baldrige National Quality Award Awards presented annually by the Governor Awarded to private, public, and not-for-profit organizations in manufacturing, service, education, and healthcare

WSQA Return Examiner What is WSQA? WSQA is a 501(c)(3) nonprofit organization supported by Corporate donations Individual memberships Workshop, collaborative and conference fees Many (many, many!) volunteer hours

WSQA Return Examiner Who is WSQA? Examiners Lite Assessment Full Assessment Certification follows assessment Senior Examiners Level 3 Team Leads Scorebook Editors Panel of Examiner & Process Development (PEPD) Mentors Panel of Judges Board of Directors Executive Director and support team

2. Baldrige criteria changes

WSQA Return Examiner Baldrige criteria booklet Beginning in 2009 revised every 2 years

WSQA Return Examiner Baldrige 2007 Criteria Changes Strategic advantages Workforce engagement Leadership- performance measures Strategic Planning-innovation, strategic advantages

WSQA Return Examiner Baldrige 2008 Criteria Changes Order of information in the booklet Scoring system description Diagrams added for core values and maturity levels Added strategic advantages as a term to the Glossary

WSQA Return Examiner Criteria Changes Customer Focus/ Engagement Organizational Core Competencies Sustainability and Social Responsibilities 14

WSQA Return Examiner 2009 Customer Focus/ Engagement Customer commitment to your brand and offerings Characterized by loyalty and willingness to make an effort to do business with you, willingness to do business with you 15

WSQA Return Examiner 2009 Core Competencies Areas of greatest expertise Strategically important Provide market place advantage May involve technologies, unique offerings, market place niche, business acumen 16

WSQA Return Examiner 2009 Sustainability Organization’s ability to address business needs and agility and strategy to address the future Considers external and internal factors Considers impact on society and well being of environment, social and economic systems 17

WSQA Return Examiner Conflict of interest issues WSQA seeks to avoid conflict of interest to protect process integrity Conflicts are frequent! First step in assignment process: determine any conflicts

WSQA Return Examiner Code of Ethical Conduct Why are ethics of utmost importance to WSQA and this process?

WSQA Return Examiner WSQA Full and Lite Process Comparison table 20

5. WSQA Process Objective: Review key steps in the process. Review key process opportunities.

Stages of the Assessment Process

WSQA Return Examiner Preparing for independent review Read the criteria Read the application Familiarize yourself with the online tool All information on the WSQA website: webinars and manuals Check in with Team Lead on Key Factors and after entering category 1 and associated results

WSQA Return Examiner Independent review 1.Select the most relevant Key Factors (from list) 2.Read the criteria for the category 3.Read (and annotate) the relevant section of the application 4.Identify and record the applicant’s processes 5.Record observations. Note missing responses and significant observations 6.Recommend a scoring range for the category

WSQA Return Examiner Independent review steps - process 1.Review related criteria 2.Read the category under review, take notes 3.Select relevant key factors from list 4.Identify & record processes 5.Record observations on ADLI: Approach Deployment Learning Integration 6.Identify any significant or missing responses

WSQA Return Examiner Record observations on “ADLI” “Approach” refers to Methods used to accomplish the process How appropriate the methods are to the item requirements The effectiveness of their use The degree to which the process is repeatable and is based on reliable data and information (i.e., systematic)

WSQA Return Examiner Record observations on “ADLI” “Deployment” refers to the extent to which The approach is applied in addressing relevant and important item requirements It is applied consistently It is used by all appropriate work units

WSQA Return Examiner Record observations on “ADLI” “Learning” refers to Refining the approach through cycles of evaluation and improvement Encouraging breakthrough change to your approach (innovation) Sharing refinements and innovations with other relevant work units and processes

WSQA Return Examiner Record observations on “ADLI” “Integration” refers to the harmonization of Plans Processes Information Resource decisions Actions Results Analyses to support key, organization-wide goals

WSQA Return Examiner Record observations on “ADLI” Questions to ask in analyzing “Integration” Do the individual components of a performance management system operate as a fully interconnected unit? Is the approach aligned with the organizational needs identified in the Organizational Profile?

WSQA Return Examiner Process diagnosis: key concepts Integration examples: Alignment of objectives and action plans with strategic challenges AND mission, vision, values Alignment of product/service delivery methods with KEY customer requirements Alignment of key process measures with KEY customer requirements

WSQA Return Examiner Process diagnosis: key concepts 1. Approach 3. Learning 4. Integration 2. Deployment PROCESS

WSQA Return Examiner Building the ADLI worksheet ItemProcessKF’sADLI 1.1a(2)Foster/ require legal & ethical behavior #, #, # Annual overview of obligations, online training, post-course survey + Covers staff, board, partners, volunteers + Daily Huddle addresses concerns quickly - None seen + Addresses HIPAA, patient confiden- tiality How would you score this single response?

WSQA Return Examiner Independent review steps - results 1.Review the Criteria to identify required results 2.Review the relevant process in the application to identify KEY results important to the applicant (use KF’s, application, your diagnosis) 3.Review the Results in the application What IS there? What ISN’T there that should be? 4.Evaluate & score on the basis of LeTCI

WSQA Return Examiner Results evaluation: key concepts Levels Numbers that position results on a meaningful measurement scale Permit evaluation relative to Past performance Projections Goals Appropriate comparisons

WSQA Return Examiner Results evaluation: key concepts Trends Numbers that indicate direction and rate of change Provide a time sequence of performance Require at least three data points Criteria call for trend data on Product, service performance Customer, workforce satisfaction Financial performance Market performance Operational performance (cycle time, productivity)

WSQA Return Examiner Results evaluation: key concepts Comparisons External requirements External benchmarks Similar OR different organizations with similar characteristics

WSQA Return Examiner Results evaluation: key concepts Integration Results align with key factors, e.g., Strategic challenges, Workforce requirements Vision, mission, values Results presented for Key processes Key products, services Strategic accomplishments What examples can you think of? Strong integration? Not-so-strong integration?

WSQA Return Examiner Results evaluation: key concepts Strong integration: results presented for Key areas addressing strategic challenges, Key competitive advantages Key customer requirements Not-so-strong integration: Results missing for the above Results presented that the Examiner can’t match to process items or Organizational Profile

WSQA Return Examiner “Key” results Refers to elements or factors most critical to achieving the intended outcome In terms of results, look for Those responsive to the Criteria requirements Those most important to the organization’s success Those results that are essential elements for the organization to pursue or monitor in order to achieve its desired outcome

Break time!

WSQA Return Examiner 2009 Stages of the Assessment Process

WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

WSQA Return Examiner 2009 WSQA Consensus: Step-by-step

WSQA Return Examiner 2009 Stage 2 Consensus Webinar Comments from all Independent reviews Ensure comments reflect scoring. - what is preventing applicant from moving to next level? Is this explained in the comments 46

WSQA Return Examiner Consensus preparation and call(s) Each Examiner responds with agreement, suggestions for changes, or disagreement, stating rationales Category Lead facilitates consensus Team Lead and/or PEPD mentor leads final scoring discussion Can take anywhere from 4-8 hours (depending on what?)

WSQA Return Examiner Consensus and conclusion If using online tool during consensus, changes may be made in real time If not, Category Leads update comments & scores after call according to consensus decisions Category Lead notifies Team Lead and Scorebook Editor when done Scorebook Editor compiles consensus scorebook with final edits, notifies WSQA Consensus scorebook should be finalized ASAP

WSQA Return Examiner Comment, scoring considerations “Benefit of the doubt” Give credit for what’s in the application Don’t penalize for incidental exceptions Not every process must show complete “DLI” Not every results example must include “TCI”

WSQA Return Examiner Comment, scoring considerations For LITE, only one score for entire results Category 7 Propose a score for results related to your categories Be prepared to reach team consensus for all of Category 7 Using scoring language can help the applicant understand comments and score

WSQA Return Examiner Examiner training Completing the stage 1 grid Using the gaps and notes tab Comment writing Consensus Meetings Scoring consistency Integration Learning Comparative Data Considerations for small organizations Opportunities

WSQA Return Examiner 2009 Baldrige National Quality Program National Examiner Questions 2009 Harry Hertz Responses

WSQA Return Examiner 2009 Further clarification of “performance projections” vs “goals” Inclusion of goals in results category – why are they not included in the evaluation? Why are projections expected for all measures in all scoring ranges? Why are projections not in the Results Criteria? Question 1

WSQA Return Examiner 2009 Why don’t we align results items with process items? Currently 1.1 is aligned with 7.6 instead of being aligned with 7.1. Question 2 Question 3 l Without comparisons, how do you interpret levels? Without a point of reference, how can an Examiner understand a “level”?

WSQA Return Examiner 2009 Question 4 l How are key themes assessed (from applicant’s perspective)? Question 5 l How do we evaluate applicants that are part of a larger system? For example an applicant might say that they use a process established by the system for which they are a part.

WSQA Return Examiner 2009 Why don’t we ask applicants to define/describe their work system(s), e.g., in 6.1.a.1, “What is/are your work system(s)?” Question 6 Question7 l Clarify the difference between work systems and work processes. There appears to be overlap and the category is labeled process management.

WSQA Return Examiner 2009 Core Competencies

WSQA Return Examiner Information resources WSQA comment guideline handout Your back-up, team lead, PEPD mentor Reference materials in Baldrige web site WSQA Website

WSQA Return Examiner What we’ve covered today WSQA overview Baldrige criteria changes WSQA ethics review WSQA Lite versus Full process WSQA process review Questions from National Training

WSQA Return Examiner THANK YOU!! Your support and participation as Examiners helps us all by helping WSQA fulfill its mission!