1. The New Teaching Institute Program Evaluation Report Lin Chengyan Eric Whitmer Anastasia Kalmykova

2. Agenda Introduction Evaluation plan description Description of Materials Evaluated Data collection tools Results

3. Introduction Name of the Program: The New Teaching Institute. Time range: November,2014 -- March,2015. Purpose of the course: Incorporate the new technology into their teaching methods. Participants: 11 Faculty & Professors. Motivation for participants: – Upgrade Teaching Skills and Methods – Additional Reward.

4. Evaluation plan description The objectives of the course: Generate a fully functional hybrid course shell including a variety of course materials designed to engage students in achieving the course goals Generate concise learning goals and objectives appropriate for a hybrid course Generate assessments appropriate for evaluating the stated course outcomes Explain the ADDIE model of instructional design and its applicability in a hybrid course Evaluate the effectiveness of other instructors’ hybrid courses

5. Evaluation plan description Effectiveness of Course Efficiency of Course Usability of Course Appeal of Course Four main factors to evaluate the module

6. Description of Materials Evaluated Accessed via Blackboard  Start Here page Blocks provide sense of community and social presence for learners Blocks provide important information of the course Contact Information of the Course Facilitator Introductions Cyber Café Navigating the Course Syllabus Course Overview Netiquette UT Hybrid Course Rubric

7. Description of Materials Evaluated The module is organized in hierarchical structure represented in a form of tree. There are main sections menu of the module available on each page. Each section contains a set of topics with specific contents.

8. Data collection tools Entry Skills survey Usability survey Attitudes and Acceptance survey Course Review Rubric data Tool Administration Surveymonkey Data collection tools

9. Results of Evaluation of New Teaching Institute Course

10. How Effectiveness of NTI Module was Evaluated? Conducted a paired samples t-test MeanN Entry skills test Course review data 2.36 2.91 7777 Course review dataEntry skills test 3 2 1 Developed questions of the entry skills test based on the course objectives and course review rubrics.

11. What is the Result of NTI Module Effectiveness? The course achieved its objectives with the intended audience The objective “Evaluate the effectiveness of other instructors’ hybrid courses” was measured by indirect observation of work performance

12. How Efficiency of NTI Module was Evaluated? Course review data Calendar

13. What is the Result of NTI Module Efficiency?

14. How Usability of NTI Module was Evaluated? Developed survey questions Conduct the survey Analyze the results

15. What is the Result of NTI Module Usability ? QuestionUnsatisfactoryAcceptableExcellent 1. Comfortable to UseXXXX 2. Navigation works, easy to use, & consistent XXXX 3. Clear OrganizationXXXX 4. Practice Opportunities AppropriateXXXX 5. Expected Functions & CapabilitiesXXXX 6. Easily Accessible in the event accidental termination XXX 7. Overall SatisfactionXXXX

16. How Appeal of NTI Module was Evaluated? Developed survey questions Conduct the survey Analyze the results NTI Attitudes & Acceptance Welcome to the NTI Attitudes & Acceptance Survey

17. What is the Result of NTI Module Appeal? QuestionStrongly Disagree DisagreeUndecidedAgreeStrongly Agree 1. Maintains InterestXXXX 2. Helps Generate Fully Functional Hybrid Course XX 3. Assignments Effective & Assist in Learning XXXX 4. Course Follows Objectives XXXX 5. Helps to Evaluate Other Instructors' Hybrid Courses XXXX Part1

18. What is the Result of NTI Module Appeal? QuestionStrongly Disagree DisagreeUndecidedAgreeStrongly Agree 6. Provided Proper Tools to Generate Assessments for Evaluating Course Outcomes XX 7. Learner was able to Generate Concise Learning Goals & Objectives for a Hybrid Course XXXX 8. ADDIE Model Effective in Delivering Content XXXX 9. Course has been BeneficialXXXX 10. Course is ChallengingXXXX Part2

19. Pattern of Like or Dislike Expert experience Boring Entry skills survey Appeal survey No experienceInteresting

20. Recommendations Conduct the regular and systematic evaluation – Develop and conduct pre/post tests – Conduct entry skills survey – Conduct usability survey – Conduct attitudes and acceptance survey Make subsequent changes based on the results if following evaluation will indicate that the majority of learners have high level of entry skills we recommend: – Add more challenging practice items – Reduce time to complete course if following evaluation will indicate that the majority of learners have problem finding information we recommended: – Make navigation simpler

21. Thank you. Questions?