EVALUATOR TIPS FOR REVIEW AND COMMENT WRITING The following slides were excerpted from an evaluator training session presented as part of the June 2011 CBFA Conference. Special thanks to Dr. Annette Craven for providing resources she received after participating in a recent national Baldrige examiner training.

Use of Standards & Criteria The Standards and Criteria serve as the basis for: Providing feedback to applicants in the form of Process & Results Strength comments OFI comments

Process Maturity

Individual Review

Individual Review Process Read Overview Read Standards One - Six

Evaluation Dimensions Approach Deployment Results Improvement (Learning)

Approach Appropriateness of methods Effectiveness of the methods The degree to which the approach is repeatable and based on reliable data and information (i.e., systematic)

Systematic Documented Repeated Evaluated Improved

Deployment (Implementation) the approach applied in addressing the requirements is relevant and important to the organization the approach is applied consistently the approach is used by all appropriate work units

Results Current performance Rate and breadth of your performance improvements Performance relative to appropriate comparisons and/or benchmarks Linking results measures to important customer, market, process, and action plan performance requirements identified in the Overview and in Approach-Deployment Items.

Where to Look for Results Results are found throughout the ACBSP Standards, for example: Student/Stakeholder Satisfaction (Standard 3) Student Learning Outcomes (Standard 4) Faculty Deployment & Qualifications (Standard 5) Business Process Results (Standard 6) Performance is generally presented in one of three ways Charts/Graphs Tables Narrative

What Should You Look For? Expected Results Appropriate Comparisons Important Segmentation

Appropriate Comparisons Performance is examined relative to competitors and/or other organizations providing similar products and services Include appropriate comparative data

Improvement (Learning) Refining the approach through cycles of evaluation and improvement Encouraging breakthrough change to the approach through innovation Sharing refinements and innovations with other academic departments or disciplines and processes in your school of business or institution

Writing Evaluator Observations Self-study Standards and Criteria Booklet ACBSP handouts including evaluator workbook

Observation Guidelines Observations should be: Standards and Criterion based Relevant Clear Concise Actionable

Observation Guidelines (Cont.) Observations should not: Parrot the self-study Be prescriptive Be judgmental Conflict with one another

Strength Observations: 3 Components Nugget what the business unit does to merit the score you assign to them Example(s) are the things the organization does with some level of proficiency Relevance Why the observation is important for the business unit (e.g. mission related, etc.)

Strength Observation Example The development of the budget is an integral part of the strategic planning process. This ensures that the budget reflects the business units strategic priorities. As a result, both long-term plans and shorter range operational activities are focused on the organizations mission, vision, and values.

OFIs Is an opportunity for improvement Is not necessarily a weakness Is what prevents an applicant from scoring at a higher level Is tied to the Standards and Criteria

Example of an OFI Observation the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

So Whats Explains the significance of the observation so that the business unit doesnt have to ask, So what? Can be used with both Strengths and OFIs though more commonly used with OFIs

Wheres the So What? the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

Wheres the So What? the business unit, through its strategic planning process has identified four key initiatives with associated goals. It is not clear, however, how these goals are being measured and the lack of an established timeframe make it difficult to gauge progress toward achieving them.

Guidelines for Results Start with a subject from the self-study or the Criteria Include the time frame you are writing about -- such as in 2009 or from 2005 to 2010 Include the actual numbers observed in the levels or trends Include figure references

Results Strength the business unit operates in a fiscally sound manner as evidenced by its Aaa and AAA bond ratings (Figure 7.3-1) and debt capacity (Figures and ). These results suggest that the business unit makes financial decisions with its long-term financial viability and sustainability in mind.

Results OFI Only 3 of the 13 figures in Standard 4, Criterion 4.3 contain comparative data. Without comparative data it is difficult to evaluate the business units current level of performance or the trends they report in these figures.

Whats Wrong with this Observation? The standard being addressed is in need of improvement. The OFI does not provide enough information – how does the business unit respond to the comment and how do the commissioners identify the weak area.

Whats Wrong with this Observation? An opportunity exists to broaden the advanced educational experience of the faculty by hiring and/or sending current faculty to a broader range of institutions than is currently the case. This may result in a broader base of knowledge for students in the Department of Business. This is a prescriptive comment – the Standards and Criteria do not prescribe which institutions faculty members should be hired from.

Whats Wrong with this Observation? Although the team was delighted with the depth of the understanding and the commitment to assessment, there were numerous closing the loop issues based on the self study report review. Upon examination and audit during the site visit, numerous closing the loop evidences were discovered. Team was delighted is judgmental and the issues discussed in the OFI were resolved while on site. This is not an OFI – it can actually be a strength.

Critique vs. information It is not clear Not addressing both feedback report customers: Commissioner and school/program Too much brevity Lack of clarity; So what? Program Evaluator Self-Study Review not converted to Feedback Report format Comments limited to they have a plan… Preliminary comments not prepared prior to site visit Common Mistakes

QUESTIONS TO ASK YOURSELF WHEN REVIEWING THE SELF-STUDY: 1.What approach is described? Is it systematic? 2.How is the approach deployed and to what extent for applicable stakeholder groups? 3.Are the results presented? 4. Is there evidence of learning or improvements?

QUESTIONS TO ASK YOURSELF WHEN REVIEWING THE SELF-STUDY: 1.Are the results presented? 2.Are there three to five data points? 3.Are the results segmented? 3.Are comparisons provided (e.g.,peer, national norm, etc.) ? 4. Are targets/benchmarks identified?