1. Copyright 2002-2004 Minnesota Council for Quality1 The 2004 MQA Site Visit

2. Copyright 2002-2004 Minnesota Council for Quality2 Today’s Agenda Purpose of the Site Visit Elements of a Successful Site Visit Site Visit Roles and Responsibilities Site Visit Preparation Site Visit Process and Products What Happens Next

3. Copyright 2002-2004 Minnesota Council for Quality3 Baseline Questions Is this applicant use the Narrative or Baldrige- Express? Which Criteria are used? How long is the site visit? (Usually 3 days.)

4. Copyright 2002-2004 Minnesota Council for Quality4 Purpose of the Site Visit Understand key organization factors Verify and clarify facts and site visit issues Prepare the feedback report Score the organization Assess commitment, sustainability, and momentum Present exit presentation

5. Copyright 2002-2004 Minnesota Council for Quality5 Elements of a Successful Site Visit People ProcessProducts Roles, Ground rules, Communication Assignments, Controlled agenda, Consensus building Exit presentation, Feedback report, Judges’ presentation

6. Copyright 2002-2004 Minnesota Council for Quality6 Evaluator Team Role & Responsibilities Gathers and documents facts for feedback report Communication with applicant is equally critical –Take time to explain and educate –You represent the Council and help sell the Council –Help make it a fun, pleasant experience

7. Copyright 2002-2004 Minnesota Council for Quality7 Applicant Representative Role & Responsibilities Contributes information for findings Arranges availability of interviewees and materials Helps team complete its agenda Applicant Representative usually provides: List of key contacts Organization chart Facility layout Optional requested material

8. Copyright 2002-2004 Minnesota Council for Quality8 Presentation to Judges Information Flow

9. Copyright 2002-2004 Minnesota Council for Quality9 Site Visit Sequence Advance Preparation Final Preparation Site Visit 1b. Exit Presentation 1a. Executive Preview 2. Feedback Report 3. Presentation to Judges

10. Copyright 2002-2004 Minnesota Council for Quality10 We are at step 2.12

11. Copyright 2002-2004 Minnesota Council for Quality11 Advance Preparation Dates set Expectations set Site visit issues compiled – SV Issue sheets –Prepare list of documents needed and individuals to be interviewed on site – use a matrix to help coordinate requests for meetings and documents Site visit agenda set Logistics arranged –Hotel –Emergency contact number 612-462-3577 –Name tags (as needed) –Expense accounting

12. Copyright 2002-2004 Minnesota Council for Quality12 Site Visit Planning Worksheet Site Visit Date:________________ Site Visit Location:________________ Evaluator:________________ Category:2 and 5 Item Ref #: 2.2 and 5.2 Issue: Verify the content of the Human Resource Plan (2.2) and clarify its use in identifying the strategic training needs for the organization (5.2) Person(s) to Interview: HR Manager Document(s) or Information to Review: Human Resource Plan Sample Questions: Tell me about HR plans you create. How do you use these? What is an example of training that was computed in the past year? Findings: Verify or Clarify

13. Copyright 2002-2004 Minnesota Council for Quality13 Focus of Site Visit Issues Information that is missing or vague. Strengths or weaknesses identified from narrative, Baldrige Express report, results data, or organizational profile. Deployment of the required practices. Cross-cutting issues or themes Site Visit Issues should be written on the “vital few” – there should not be more than 3-4 per item.

14. Copyright 2002-2004 Minnesota Council for Quality14 Site Visit Techniques Acceptable Techniques Reviewing data, reports, documents hard-copy or electronic as provided directly by the applicant Interviewing individuals and teams Receiving presentations by participating organization Unacceptable Techniques Conducting focus groups or surveys with customers, suppliers, or dealers Disrupting work processes Interviewing sources or collecting data beyond that provided by the applicant This includes conducting library or internet research and visiting web sites. This includes conducting library or internet research and visiting web sites.

15. Copyright 2002-2004 Minnesota Council for Quality15 “ Typical” Site Visit Issues Verify/Clarify Role and activities of senior management (driving customer focus, empowerment, innovation, learning) Degree of involvement and empowerment of employees Reliable and accessible data and information Use of facts in decision-making Focus on customers Systematic approaches to management Training effectiveness, on-the-job reinforcement Alignment of recognition and rewards to support strategy, customer focus, and key business outcomes Use of strategic objectives, action plans, and related measures to align work at all levels—links to performance feedback and employee rewards Evidence of evaluation and improvement Integration of all processes—both core business and support Supplier involvement in performance improvement activities

16. Copyright 2002-2004 Minnesota Council for Quality16 “Typical” Walk-around Questions What are the values, mission, and vision of the organization? What are the action plans for your work unit? What are the goals? What role do you play in achieving them? What kind of training do you receive? Is it useful to your work? How is training reinforced on the job? How are you empowered to take actions needed to carry out or improve your work? How are you involved in decision-making about work? What is recognized or rewarded here?

17. Copyright 2002-2004 Minnesota Council for Quality17

18. Copyright 2002-2004 Minnesota Council for Quality18 Final prep meeting held evening before site visit –Review SV issue sheets –Set up key themes worksheet/storyboard –Establish walk-around questions – Finalize the agenda, assignments, pairs Each Evaluator brings draft of site visit issue worksheets, to prepare a composite set Applicant representative meets with the team Aid in planning the time on site Share general schedule conflicts Final Preparation

19. Copyright 2002-2004 Minnesota Council for Quality19 Final Preparation (continued) Review and update documentation list as needed Review and update list of individuals to interview on site This will help you identify and manage hand-off issues. Finalize the matrix of document and meeting requests the team created earlier in the processes

20. Copyright 2002-2004 Minnesota Council for Quality20 Site Visit Interview Matrix Individual to InterviewSite Visit Issue Ref. Sheila Knight, Marketing1.1.1, 2.2.1, 3.1.1, 3.1.2, 3.1,3, 3.2.1, 3.2.4, 4.1.1, 4.1.3, 4.2.1, 4.2.2, 7.1.3 Ron McDonald, Administration1.1.1, 1.1.3, 1.2.1, 1.2.2, 7.2.3 Mary Stewart, Employee Development1.1.1, 2.1.1, 5.1.1, 5.1.3, 5.2.1, 7.3 all Alex Hamilton, Operations1.1.1, 2.1.1, 6.1.3, 6.2.1, 6.2.2, 7.5 all John Jones, Finance4.1.4, 4.2.1, 4.3.2, 7.2 all Leader/manager of Process Improvement Team & members of PI Team 6.1.5 Individual responsible for coordinating leadership communication 1.2.5, 2.2.2 Individual responsible for Support Services Process Improvement Meetings with Leadership 6.2.5

21. Copyright 2002-2004 Minnesota Council for Quality21 Sample Site Visit Schedule MondayVanessa (1), (2) 7.2 Walt (3, 7.1) Roy & (4, 5, 7.3) Deana (6, 7.4) 7:00Breakfast 7:30Travel to Site 8:00Opening Meeting 9:00Ron McDonald, CEO & Leadership Team Meeting Worker Services Team Team Meeting 10:00Category Lead Meeting, then Quick Caucus 11:15One on ones with Eos during lunch breaks 1:30Nate O. Forsight Planning VPMary Stew, Dir IT Development 2:35Caucus 3:00Customer Team * Sheila Knight (Mktng) Alex Hamilton, Operations 4:15Walk around, Shift Change Interviews 5:00Caucus, revise plan in detail for Day 2

22. Copyright 2002-2004 Minnesota Council for Quality22 Site Visit Agenda Opening meeting –Introductions to the applicant (MCQ/Team Leads) –Participant presentation (up to one hour) Facilities tour (optional) Scheduled meetings with key applicant resources –Allow time for travel when necessary –Schedule applicant interviewees Scheduled times for walk-around interviewing of applicant employees Scheduled times for team discussion (caucuses) Executive preview and exit presentation

23. Copyright 2002-2004 Minnesota Council for Quality23 Managing Issues and Handoffs Create 7 flip charts, one for each category. Ask Evaluators to use yellow sticky notes to record issues they have to hand off to the category owner in the “open” area and red for new issues. When the information has been collected, record the answer and move the sticky to “closed” portion. This way everyone can keep track of open and closed issues and effectively manage handoffs. Category 1 Open  Closed  Verify deployment between shifts Ask CEO about employee focus (4)

24. Copyright 2002-2004 Minnesota Council for Quality24 Key Themes Worksheet/Storyboard A key theme is a Strength or OFI that is common to more than one Item or Category, is especially significant in terms of the applicants Key Factors, and/or addresses a Core Value of the Criteria. Provides key points and an overall summary of the team’s evaluation of the applicant. Divided into three sections to address three questions concerning important Strengths, significant OFIs, and key Results Will be used to help prepare the exit presentation

25. Copyright 2002-2004 Minnesota Council for Quality25 Evaluator Team Caucuses Compile findings Review feedback comments Update SV issue sheets and key themes Hand-off any outstanding questions Remember - applicant representative does not participate except for logistical issues and planned review of key themes Be sure to schedule enough time for caucuses plus some personal break time

26. Copyright 2002-2004 Minnesota Council for Quality26 Structure for Interviewing Begin the interview with: –Introduce yourself and why you’re there We will respect confidentiality of information We will be looking for both Strengths and OFIs –Ask if they have questions –Explain why you are taking notes –Describe the time limits End the interview with: –Do you have any questions for me? –Thank them for their time and honesty

27. Copyright 2002-2004 Minnesota Council for Quality27 Effective Interviewing Tips Refocus when they ramble Ask open-ended questions, probe when necessary, bring it to a conclusion Script your lead and follow-up questions Be frank and respectful –Use the applicant’s terminology –Actively LISTEN to the applicant Be patient with shy interviewees

28. Copyright 2002-2004 Minnesota Council for Quality28 Asking Questions Use open-ended questions Tell me about … How do you … Help me understand … Please walk me through … Avoid Leading questions. Why don’t you – At my office we do it this way. What do you do? Have you always done it this way? Why did you change? How did you know what to change? Don’t you think it would be a good idea to re-engineer your manufacturing process? How do you determine how frequently to check this process? Why do you only audit this process every two years? How do you know customer requirements are correctly defined? Why don’t you involve customers in your design-build teams?

29. Copyright 2002-2004 Minnesota Council for Quality29 Typical Daily Progress (Day 1) Advanced level of understanding of business factors Close some Site Visit Issues (typically Cat. 1 and 2) Establish remaining agenda Brief applicant on progress and outstanding issues Category Owners begin editing their comments

30. Copyright 2002-2004 Minnesota Council for Quality30 Typical Daily Progress (Day 2) Confirm key themes Close 100% of remaining Site Visit Issues Identify remaining information/interviews needed (if any – there should not be any) Finalize plan for Day 3 Brief applicant on progress, outstanding issues, and plans for Day 3 Finalize 100% of the feedback comments

31. Copyright 2002-2004 Minnesota Council for Quality31 Typical Daily Progress (Day 3) Close remaining issues (if any) Prepare and deliver executive preview and exit presentation Finalize feedback report –Category Owners recommend final wording for comments –Entire team reviews and signs off on items as completed Identify any Pretty Good Practices Score Assess commitment, sustainability, and momentum Complete logistics (hotel checkout, return of applicant materials, collection of shredding, 360 feedback)

32. Copyright 2002-2004 Minnesota Council for Quality32 Preparing the Exit Presentation Review the key themes worksheet –Significant Strengths, including Pretty Good Practices –Significant Opportunities for Improvement Prepare the PowerPoint presentation Identify presenters

33. Copyright 2002-2004 Minnesota Council for Quality33 Executive Preview Applicant’s option Team Leads only Council representative also present Summary review of exit presentation content Opportunity for clarifying questions Expression of thanks for cooperation and participation

34. Copyright 2002-2004 Minnesota Council for Quality34 Exit Presentation One hour Moderated by MCQ representative Audience determined by applicant Review key themes –Significant Strengths and Pretty Good Practices –Significant Opportunities for Improvement Clarifying Q&A Thanks

35. Copyright 2002-2004 Minnesota Council for Quality35 Scoring Assign scoring facilitator Determine order for scoring items –Allow 5-10 minutes maximum per item Review scoring guidelines, terminology (refer to Clarifying Requirements for Basic-, Overall-, and Multiple-Level Scoring) For each item –Category Owner reviews high points of comments –Team members individually determine scores –Team displays scores on facilitator’s signal –Team negotiates variances Be mindful of item point values Don’t bog down Record percent scores on scoring matrix Remember--It’s not over till it’s over

36. Copyright 2002-2004 Minnesota Council for Quality36 Preparing the Feedback Report At site –Category owners prepare final wording of comments –Team reviews and signs off on comments –Team reviews judges’ questions After site (within one week) –Feedback editor collates comments into feedback template –Feedback editor writes executive summary based on Key Themes –Feedback editor distributes report to team for feedback –Team leader forwards finished feedback to lead judges Next steps (approximately two weeks) –Judges review and edit final report and forward to Council –MCQ forwards finished report to applicant

37. Copyright 2002-2004 Minnesota Council for Quality37 Reminder About Comments Review writing rules for all comments (refer to your training manual) Category 7 comments must: –Address results, not approach, deployment, integration, or learning –Reflect the measures the applicant says are important (expected results table) –Be stand-alone and quantified –Address specifics of levels, trends, and performance relative to comparisons

38. Copyright 2002-2004 Minnesota Council for Quality38

39. Copyright 2002-2004 Minnesota Council for Quality39 Judges’ Presentation Agenda Key Factors – 2 minutes Key Themes: Process Strengths – 3 minutes Key Themes: Process OFIs – 3 minutes Key Themes: Results – 6 minutes Significant Differences (if any) – 1 minute Commitment, Sustainability, and Momentum – 5 minutes Questions – 10 minutes

40. Copyright 2002-2004 Minnesota Council for Quality40 Judges’ Presentation - Commitment The extent to which leaders are willing to invest time, resources and energy in making the journey. An example of low commitment: leaders spend little time on quality or planning for improvements – few resources are made available for improving quality systems and approaches An example of high commitment: leaders consistently encourage and lead efforts to improve the organization – leaders plan for high levels of performance – leaders allocate resources to the effort

41. Copyright 2002-2004 Minnesota Council for Quality41 Judges’ Presentation - Sustainability The extent to which the organization is likely to continue making progress on its journey An example of low sustainability: The quality journey may be seen as just a management fad – “something the boss wants us to do.” If the leader of the quality effort were to disappear from the scene, the journey would stop. An example of high sustainability: “Managing quality is everyone’s job. We need to excel as an organization in order to survive and prosper. It’s not just the boss’ thing.” If there were a change in senior leadership, progress would continue.

42. Copyright 2002-2004 Minnesota Council for Quality42 Judges’ Presentation - Momentum The direction and rate of progress on the quality journey. An example of low momentum: There may be slow or no progress overall from year to year, although some pockets of rapid or steady growth may exist. The organization may be little aware of its opportunities for improvement or how to close the gaps. Perhaps, improvements in the past may have been lost and are no longer apparent. An example of high momentum: Innovation and improvement in business practice is frequent and pervasive. Baldrige scores may have been rising each year through multiple cycle of self-assessment and improvements.

43. Copyright 2002-2004 Minnesota Council for Quality43 What Happens After the Site Visit Final judging Applicant receives feedback report Improvement Planning Session Celebration!