2. Enterprise-Based E-Learning Planning and Evaluation in a Large-Scale Operation © Copyright Nicola Martinez and Susan Oaks 2003. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

3. Presenters  Susan Oaks, Ph.D., Academic Area Coordinator, Writing, Literature and Communications; Chair, Curriculum Committee  Nicola Martinez, MA, Coordinator of Instructional Design and Curriculum Development Center for Distance Learning SUNY Empire State College Saratoga Springs, New York

4. Objectives  Demonstrate an integrated planning & tracking database for course development, review, and evaluation  Discuss context/issues leading to database  Discuss collaborative approach—and multiple uses of database—by faculty, instructional designers, term prep staff  Focus on quality assurance and applicability to other institutions

5. Who We Are: Empire State College  Locations around New York State  The Center for Distance Learning (CDL) serves adult students around the world

6. Mission Statement Empire State College is dedicated to:  enabling motivated adults  regardless of geography or life circumstance  to manage and master a rigorous academic program  and earn a degree

7. Areas of Study Areas of Study: The Arts | Business, Management & Economics | Community & Human Services | Communications, Humanities & Cultural Studies | Educational Studies | Historical Studies | Human Development | Labor Studies | Science, Math & Technology | Social Theory, Structure & Change

8. Center for Distance Learning  offers asynchronous online courses in five, 15- week semesters per year  leading enroller in the SUNY Learning Network  partner with several institutions such as eArmyU & Navy, public & private organizations, labor unions

9. Center for Distance Learning Course Statistics  approximately 5,000 students annually  200 distinct online courses  more than 350 course sections  over 6000 enrollments per term  46 courses developed for fall web delivery  50+ courses in development for Jan 04  existing courses scheduled for cyclical delivery

10. Context: Rapid Online Growth  Increase in f/t faculty from 8 to 15 (and growing) within past two years  Hire two Coordinators of Instructional Design and Curriculum Development to shepherd faculty developers through course development process and coordinate instructional support  Move from online business degrees to online degrees in many areas of study  Move all existing courses online (over 200)

11. Courses in development Some courses in development this term:  Artistic Expression in a Multicultural America  Introduction to Visual Literacy  Dance Across World Cultures  Food and Drink in Cultural Context  Reflective Learning  Biology of the Brain  Cognitive Psychology  Minds and Machines  Urban Studies  Chemistry in Context

12. Previous Course Development System  Elected Curriculum committee (4 faculty) + 1 instructional designer  All faculty review & comment on course proposals  Curriculum committee review, vote, and respond  E-mail records of issues/outcomes

13. Problems with Previous System  Not representative of all broad disciplinary areas  Sparse review & feedback on proposals  E-mail records time consuming; inconsistent depending on committee chair  Course development relates to other systems (instructor training, book ordering, quality evaluation); maintenance of multiple, discrete systems

14. New System: CourseTrak A Collaboration Between:  The Center for Distance Learning Dean: Meg Benke  The Center for Learning and Technology Director: Evelyn Ting  The Office of Educational Technology Vice President: Patricia Lefor

15. New System: CourseTrak  Integrates information for all players: instructional designers, faculty, support staff  Creates a public record from course proposal through review and offerings  Fosters collaboration through its public nature  Facilitates evaluation & review

16. CourseTrak: Faculty Perspective  Reminds course developer/Area Coordinator of required information,objectives, content and learning activities  Requires rationale for new courses in terms of academic skills, student population, etc.  Invites other faculty collaboration  Facilitates quality review process  Enlists staff help with materials ordering, scheduling, etc.—integrates systems

17. CourseTrak: Instructional Design Perspective For large scale development of pedagogically and academically sound courses infused with:  Active learning  Media-rich content  Student interaction  Opportunities for Teaching, Social and Cognitive Presence

18. Constraints WHILE EXPERIENCING A RAPID VOLUME INCREASE IN ENROLLMENTS, COURSES, AREA COORDINATORS, AND ADJUNCT FACULTY. Increased volume of courses developed per term:  from 5-8 for Fall 02  to 46 for Fall 2003  to 50 + for Spring 2004

19. Our Approach  Develop a new standard for course information documents and implement them in the courses  Create a new design for CDL web courses to increase navigability and overall consistencies in look, feel, and format  Enrich courses with multimedia components and other graphic enhancements  Include library referenced materials in all courses, with at least one library-based learning activity per course  Identify and implement best practices in the pursuit of academic and pedagogical excellence in web course development for adult learners taking distance learning courses

20. A Consultative Approach Strategies for the development and achievement of these objectives included consulting with:  Curriculum Committee  CDL Faculty  The Curriculum Development Team  The L-Guide Focus Group  The Center for Learning and Technology Library And Multimedia Instructional Design Staff.

21. Course Ratings  We established a three-tiered categorization of courses based on the inclusion of the above recommendations and the application of a new course design.  All new and revised courses are reviewed an assigned the following ratings:

22. Course Rating System Rating Legend Tier 3Exemplary course Tier 2Acceptable (some criteria not met, revisions needed) Tier 1Not Acceptable NRNot Rated Table 1. Tier 1 Ratings Legend

23. Course Rating System Table 1. Tier 1 Ratings Legend  Tier information and classification is generated as a result of the information captured in The Empire State College Online Course Evaluation Checklist.  All courses are considered unrated until they have been reviewed with this instrument.  The evaluation piece associated with The Empire State College Online Course Evaluation Checklist happens at several levels.

24. Online Course Evaluation Checklist Table 1. Tier 1 Ratings Legend  Administered during and after a new course has been developed, before delivery  Before an old course is revised to identify specific areas targeted for improvement  After an old course has been revised to ensure that it is complete and meets all baseline criteria for a Tier 3 course  This will also be used, in the future, to evaluate instruction

25. The Course Resource Needs Analysis Table 1. Tier 1 Ratings Legend  is used to establish the resources, support, services, and document creation needed for a course in development or revision.  An instructional designer conducts the needs analysis in consultation with the course developer (s).

26. The Course Resource Needs Analysis Table 1. Tier 1 Ratings Legend Information captured in the Course Resource Needs Analysis Form is used to generate work requests for  technical assistance  library assistance  multimedia learning object creation  digital media research and implementation  and instructional design/pedagogical support, among other things.

27. Focus on Quality Assurance  Establish Baseline Criteria for Courses  Evaluate and Review Courses  Track Performance  Capture Data  Make Necessary Adjustments to Courses  Create a Data Driven Analysis  Develop a Report  Make Recommendations  Improve Process and Performance Using Results

28. Applicability to Other Institutions This is an example of an in-house, web based system  designed to meet our course planning, revision, and evaluation needs  and tailored to the culture and goals of our institution

29. Applicability to Other Institutions Points to Consider in the Development of such a system:  Level of Institutional Support  # of faculty developing courses  # of new faculty needing intense support  How to integrate the system with other relevant systems  Access Control and Security Issues  Analyze your assumptions

30. Contact Information Nicola.Martinez@esc.edu Susan.Oaks@esc.edu Empire State College: www.esc.eduwww.esc.edu Center for Distance Learning: www.esc.edu/cdl www.esc.edu/cdl