1. 1 ©2002 Minnesota Council for Quality Evaluation Techniques for Team-Leaders and Judges August 2002

2. 2 ©2002 Minnesota Council for Quality Workshop Purposes Improve understanding of roles, responsibilities, procedures and expectations Increase awareness of resources Focus on planning, consensus, site visit, and report to Judges Convey changes to the process Improve ability to guide team

3. 3 ©2002 Minnesota Council for Quality Workshop Agenda Background: Schedule, Roles/Responsibilities Team Dynamics Consensus Process Site Visit: Preparation, Execution, and Exit Presentation Feedback Report Judging Process Lessons Learned

4. 4 ©2002 Minnesota Council for Quality Schedule This year, we began a “rolling cycle” for assessments From Letter of Intent, we have about 30-45 days to field a team Stage 1 (Independent Review) should take about 30 days Stage 2 (Consensus) should be completed about 14 days after Independent Review Stage 3 (Site Visit) should be completed within 28 days of Consensus

5. 5 ©2002 Minnesota Council for Quality Schedule, page 2 The Feedback Report is due to MCQ within 7 days of Site Judging will be within 28 days of Site (and may be “batched”) FYI: The last day an organization can submit the LOI and enter the 2002 cycle “batch” is November 1 (for early 2003 Evaluation and Judging using the 2002 Criteria)

6. 6 ©2002 Minnesota Council for Quality Roles Co-Leaders Mentors

7. 7 ©2002 Minnesota Council for Quality Co-Leader Responsibilities Plan and organize Facilitate team development and understanding of responsibilities Guide team’s work and team meetings Ensure production of high quality and timely feedback reports Serve as liaison with participants, Judges and MCQ staff

8. 8 ©2002 Minnesota Council for Quality Your Role: Provide Guidance Be aware of the Judges’ needs Evaluate in a holistic way  not rote or mechanistic  consider the needs of our customers Keep the “big picture”  view participant as a whole entity  consider the context of the participant

9. 9 ©2002 Minnesota Council for Quality Your Role: Provide Guidance Focus on the participant’s needs  fair evaluation  meaningful feedback to drive its improvement process Remember both customers require a thorough evaluation  Participants  Judges

10. 10 ©2002 Minnesota Council for Quality Judge/Team Partnership Direct points of contact  Before consensus meeting  Before site visit  After site visit Final edit of Feedback Report

11. Team Dynamics

12. 12 ©2002 Minnesota Council for Quality Evaluator Team Dynamics Forming Storming Norming Performing

13. 13 ©2002 Minnesota Council for Quality Leading Teams: Setting Ground Rules Use the Criteria as your evaluation anchor Maintain confidentiality Commit to deadlines and keep on-time Respect team members’ time, effort, and opinions; listen Communicate, communicate, communicate Be professional: you are representing the program Call “ELMO” (“Enough Already, Let’s Move On”) if the team appears to be stuck Have fun! [others?]

14. 14 ©2002 Minnesota Council for Quality Planning Meeting Team Formation, Roles Survey computer usage Team-specific Timeline and Deliverables Ground rules Site Visit Calendar

15. Consensus Process

16. 16 ©2002 Minnesota Council for Quality Consensus Planning Assign team responsibilities Consensus Matrices Category Summaries

17. 17 ©2002 Minnesota Council for Quality Feedback Report Process Final Summaries Draft Category Summaries Site Visit Worksheets Consensus Matrix Draft Comments Focus on Quality Comments FEEDBACK REPORT

18. 18 ©2002 Minnesota Council for Quality Focus on Quality Comments Address Criteria objectives Use Criteria language Use clear, concise, declarative sentences Do not be prescriptive Include “so what” statements, as appropriate Focus on what is important and relevant to the participant  Use Key Factors  Use specific examples

19. 19 ©2002 Minnesota Council for Quality A Word About Baldrige Express

20. 20 ©2002 Minnesota Council for Quality Consensus Tips and Traps

21. Site Visit Preparation

22. 22 ©2002 Minnesota Council for Quality Site Visit Planning session with Participant Reps

23. 23 ©2002 Minnesota Council for Quality Planning Session Considerations Complexity of Participant Organization  Number of Sites  Hours of Operation Experience and Expertise of Team Members  Category Leads  Written Comments On Site Logistics and Protocol

24. 24 ©2002 Minnesota Council for Quality Site Visit Planning session with Participant Reps Time management  Pre-visit revisit  Control the agenda  Caucuses / Rewrite time

25. 25 ©2002 Minnesota Council for Quality Example Schedule - Day 1

26. 26 ©2002 Minnesota Council for Quality Example Schedule - Day 2

27. 27 ©2002 Minnesota Council for Quality Example Schedule - Day 3

28. 28 ©2002 Minnesota Council for Quality Site Visit Planning session with Participant Reps Time management  Pre-visit revisit  Control the agenda  Caucuses / Rewrite time Story Boards

29. 29 ©2002 Minnesota Council for Quality Story Boards Role Model Processes Key Strengths Key OFIs Show- stoppers or Missing Links Major Gaps

30. 30 ©2002 Minnesota Council for Quality Site Visit Planning session with Participant Reps Time management  Pre-visit revisit  Control the agenda  Caucuses / Rewrite time Story Boards Scoring  Method for scoring  “Basic,” “overall,” “multiple” language

31. Executive Preview and Exit Presentation

32. 32 ©2002 Minnesota Council for Quality Executive Preview Story Board Category Summaries Exit Presentation

33. 33 ©2002 Minnesota Council for Quality Core Values  Strengths  Next Steps See Workforce Development Network model Exit Presentation Format

34. 34 ©2002 Minnesota Council for Quality Example Exit Presentation: Leadership Strengths  System in place  Clear statement and reinforcement of direction  Clear tie to community values  Strong stakeholder representation  Embedding of review processes Next Steps  Define measures of leadership effectiveness  Make improvements more systematic

35. 35 ©2002 Minnesota Council for Quality Example Exit Presentation: Continuous Improvement & Org. Learning Strengths  Embedded in culture as a value  Systems thinking in place  Evaluations performed Next Steps  Embed systematic refinement steps in all processes  Understand better how to use data for improvement  Strengthen processes for sharing learnings across units  Engage students in the process

36. 36 ©2002 Minnesota Council for Quality Example Exit Presentation: Management by Fact Strengths  Many sources of data, measurements  Recognition of need to systematically focus information  Embedding of review processes Next Steps  Identify clear process to tie data to goals  Increase/broaden accessibility of data to staff  Increase understanding of how to use data

37. 37 ©2002 Minnesota Council for Quality Site Visit Tips and Traps

38. Feedback Report

39. 39 ©2002 Minnesota Council for Quality Feedback Report Format Executive Summary Item  Strengths  Areas for Improvement Scoring Chart

40. 40 ©2002 Minnesota Council for Quality Feedback Report Guidelines Executive Summary should reflect Key Themes and Core Values Item Comments should be feedback quality (non- prescriptive, linked to KF, tied to Criteria, have “so what”) Results comments should have data included, Figure references, and so what (as appropriate) This is out product to the applicant and Judges!!

41. Feedback Report Tips and Traps FEEDBACK REPORT

42. Sanity Check

43. Judging Process

44. 44 ©2002 Minnesota Council for Quality Chief Judge Lead Judge Which one is which?

45. 45 ©2002 Minnesota Council for Quality Judges Role Review participant narratives Consult with Team Leaders Act as lead discussant as assigned Discuss narratives Resolve issues Assign recognition levels

46. 46 ©2002 Minnesota Council for Quality Judging Process for Team Leaders Present selected participants to Judges in requested format Answer questions Be available for subsequent questions via phone if necessary

47. 47 ©2002 Minnesota Council for Quality Judging Process for Lead Judge Participants considered by sector Review and briefly summarize assigned narrative to judges Review Item scoring Review strengths and vulnerabilities Identify issues Propose recognition level

48. 48 ©2002 Minnesota Council for Quality Judges Presentation Format 1.Overview of the participant 2.Strengths 3.Opportunities for Improvement 4.Strengths/OFIs in Results 5.Significant Differences on Team 6.Significant Issues/discussion

49. 49 ©2002 Minnesota Council for Quality 1: Participant Overview Brief Organization Description Key Business Factors  Products and Services  Major Markets  Employee Base  Customer Requirements  Competitive Factors

50. 50 ©2002 Minnesota Council for Quality 2: Significant Strengths Summarize significant strengths in terms of categories, items or processes; they should resemble Pretty Good Practices and Key Themes Try to get the summary down to a single slide for strengths Answer the Judges’ question: What are the most important strengths or outstanding practices of potential value to other companies that the team identified?

51. 51 ©2002 Minnesota Council for Quality 3: Significant Opportunities Summarize significant opportunities for improvement in terms of categories, items or processes Include major issues with linkage or alignment Try to get the summary on a single slide Answer the Judges’ question: What are the most significant concerns, risks, or vulnerabilities that the team identified?

52. 52 ©2002 Minnesota Council for Quality 4: Strengths/OFIs in Results Summarize significant strengths/opportunities for improvement in terms of results Try to get the summary on a single slide Answer the Judges’ question: Considering the participant’s key business factors, what are the most significant strengths, vulnerabilities and/or gaps (data, comparisons, linkages) found in the business results category?

53. 53 ©2002 Minnesota Council for Quality 5: Significant Team Differences Summarize significant differences on the Team Try to get the summary on a single slide Answer the Judges’ question: During your final team discussions were there any issues - strengths or vulnerabilities of the participant - which produced significant differences of opinion among site visit team members?

54. 54 ©2002 Minnesota Council for Quality 6: Significant Issues/Discussion Identify any “show stoppers” (often scored at 0-10%)  Perceived  Real Reveal any nagging disconnects during the site visit; may represent gaps in the system or topics you had a great deal of difficulty getting to the answers

55. 55 ©2002 Minnesota Council for Quality Judges Presentation Guidelines Be prepared ahead of time Follow the format Assist in the understanding the comments of the report Respond to questions Use clear, concise, objective responses Do not advocate for the Participant

56. 56 ©2002 Minnesota Council for Quality Judges Presentation Guidelines Relax - You are providing valued information!!!

57. 57 ©2002 Minnesota Council for Quality Lessons Learned Tried and True Methods Evaluation of Team Leader Workshop