1. © 2003 (M. Bober) Presentation Guidelines Title (1 slide) Description of the evaluation (1-2 slides) Why conduct the study (1-2 slides) Evaluation approach or framework (1-2 slides) Evaluation questions (1-2 slides) Findings (1-4 slides, bulleted items, tables and/or graphs) Recommendations (1-2 slides)

2. © 2003 (M. Bober) Presentation Guidelines Handouts  Executive summary (3-page maximum)  Copy of slides (optional; 3 to a page) Timed Presentation: 10 minutes Questions: 5 minutes Transition to next presenter: 2 to 3 minutes

3. © 2003 (M. Bober) Welcome! Management Training: The Competitive Edge Prepared for the Executive Board Family Financial Services

4. © 2003 (M. Bober) Description: What is Family Financial Services?  Family Financial Services, Inc. is a leader in the first and second mortgage business –Based in New York and New England –Can help you understand what a lender considers when you submit an application –Can show you how to improve on an imperfect credit history –Can help you distinguish between first and second mortgages

5. © 2003 (M. Bober) Description: Why conduct this study?  To determine how well newly introduced management strategies translate into increased revenue, strong customer loyalty, and high company morale  To ensure that Family Financial Services is adequately responding to its training and performance needs … and to its competition  To gather information about about the effectiveness of newly-implemented, mid-level management training

6. © 2003 (M. Bober) Evaluation approach: Measures  Among the study’s several measures were these: –Participant reactions to the five workshops presented over a three-month period of time –Participant learning gains –Alignment of objectives, content presentation, practice, and assessment –Quality of additional resources provided “post hoc”

7. © 2003 (M. Bober) Evaluation approach: Underlying frameworks  Formative and utilization-focused  Implementation and management-oriented –Kirkpatrick (1996) guidelines (at Levels 1 and 2) incorporated

8. © 2003 (M. Bober) Evaluation approach: Major questions  How effective were the workshops in terms of: –Facilitation? –Environment? –Content? –Materials/resources? –Relevancy to actual job tasks?  How well did the workshop goals align with the corporate mission and vision?

9. © 2003 (M. Bober) Data gathering: An overview Mid-Level Management Team ObservationInterviews Extant —Ajax Customer Opinion Survey —GAO Welfare Placement Index —Performance Reviews (1997, 1998) —35 Regional Managers —10 Supervisors (2 per region) —75 Customers (15 per region) —Manager/Supervisor Interactions —Manager/Customer Interactions —Workspace —100 Staff (20 per region)

10. © 2003 (M. Bober) With whom did we speak?

11. © 2003 (M. Bober) Findings: Participant reactions

12. © 2003 (M. Bober) Findings: Participant reactions

13. © 2003 (M. Bober) Findings: Did learning take place? Achievement Gains by Group

14. © 2003 (M. Bober) Findings: Did learning take place?

15. © 2003 (M. Bober) Findings: Interpretation  Management training provided at Family Financial Services is cutting-edge –Overall reactions to the workshop series and its facilitation were overwhelming positive –Mean ratings of all items related to course facilitation were at or above 4.8 (on a five-point scale where 1 represented poorly and 5 excellent or superb)

16. © 2003 (M. Bober) Findings: Interpretation  Reactions to questions about potential supervisorial support were equally positive –Reactions to the workshop series and its facilitation were overwhelming positive –More than 90% of the 110 staff respondents claimed to have managers who are fair-minded, actively mentor, and provide constructive feedback

17. © 2003 (M. Bober) Recommendations  Team building –Provide a broader range of examples and practice opportunities; as presented, the concept is too abstract and subjective –Integrate team building into the formal performance appraisal system –Launch a listserv where team leaders can share successes and failures

18. © 2003 (M. Bober) Recommendations  Management selection –Use the ACME Personnel Screening battery with more consistency; establish a cut-off score of 75% –Incorporate a What’s wrong this this picture? scenario into the selection process –Promote from within whenever possible