1. Looking for the gaps: Using a limiting factors approach when evaluating online learning programs Dr. Susan Lowes Teachers College/Columbia University VSS/2009

2. “Limiting factors” approach Term comes from the field of ecology, where often cannot see program results for years Term comes from the field of ecology, where often cannot see program results for years Goal is to identify the current unmanaged factors that are likely to prevent the project from achieving its objectives Goal is to identify the current unmanaged factors that are likely to prevent the project from achieving its objectives Shift focus from looking at success (or only at success) to looking at what is preventing greater success—the limiting factors Shift focus from looking at success (or only at success) to looking at what is preventing greater success—the limiting factors

3. Evaluation questions Are there unmanaged “limiting factors” that raise doubts about whether a program can achieve its goals? Are there unmanaged “limiting factors” that raise doubts about whether a program can achieve its goals? Are there unmanaged “limiting factors” that are preventing a successful program from achieving even greater success? Are there unmanaged “limiting factors” that are preventing a successful program from achieving even greater success?

4. Limiting factors in online programs For evaluators, online programs have two key characteristics: For evaluators, online programs have two key characteristics: Distributed players Distributed players Hidden activities Hidden activities The challenge is to see how all the players, distributed over space and over time, work together The challenge is to see how all the players, distributed over space and over time, work together

5. Distributed players

6. One curriculum designer with distributed teachers One curriculum designer with distributed teachers One administrator with distributed teachers, students, and parents One administrator with distributed teachers, students, and parents One teacher with distributed students and parents One teacher with distributed students and parents One student with distributed teachers One student with distributed teachers One site coordinator with distributed teachers One site coordinator with distributed teachers One parent with distributed teachers One parent with distributed teachers Distributed over space

7. Distributed over time Responsibilities are distributed over time Responsibilities are distributed over time Interactions are distributed over time Interactions are distributed over time

8. Hidden activities Student learning Teacher Administrator Other students Site coordinator Curriculum designer

9. Examples of hidden activities Looking at teaching and learning online is challenging and very time consuming Looking at teaching and learning online is challenging and very time consuming Curriculum designers cannot see teachers teaching, only the residual artifacts Curriculum designers cannot see teachers teaching, only the residual artifacts Administrators cannot see teachers teaching or students learning, only the residual artifacts Administrators cannot see teachers teaching or students learning, only the residual artifacts Site coordinators cannot see teachers teaching or students learning, and sometimes not even the residual artifacts Site coordinators cannot see teachers teaching or students learning, and sometimes not even the residual artifacts Students cannot always see each other learning, since increasingly use email, Facebook pages, Skype, etc. Students cannot always see each other learning, since increasingly use email, Facebook pages, Skype, etc.

10. In online programs, limiting factors are therefore found in the gaps

11. Examples of gaps between players Between an instructional designer works without input from a teacher, designing courses that are not teachable Between an instructional designer works without input from a teacher, designing courses that are not teachable Between a teacher who cannot get his/her students to interact online and those students Between a teacher who cannot get his/her students to interact online and those students Between that teacher and the “manager” of the students, generally the site coordinator Between that teacher and the “manager” of the students, generally the site coordinator Between the student who cannot complete an assignment, has a computer crash, is taken off for a week by his/her parents and the teacher Between the student who cannot complete an assignment, has a computer crash, is taken off for a week by his/her parents and the teacher Between an administrator who puts students into a course and the teacher who feels he/she should not be there Between an administrator who puts students into a course and the teacher who feels he/she should not be there

12. IBOCOURSEDEVELOPER COURSE DELIVERER TEACHERSITECOORDINATOR Develop requirements document Do course needs assessment Make decision to offer course Hire teacher-designer Hire instructional designer Design course Approve course content Evaluate course structure in terms of requirements document Hire course teacher Train course teacher Take teacher training Hire and train site coordinator Take site coordinator course Recruit students Register students Administer readiness pre-assessment to students Supervise course teacher Teach course Supervise site coordinators Supervise students Evaluate course interaction in terms of requirements document Evaluate site coordination in terms of requirements document Assess student satisfaction Administer internal assessments Assess teacher satisfaction Administer external assessments Grade external assessments And limiting factors arise because the activities are distributed over time TIME