1. Stephen Jones, David Monson, & Scott Howell Brigham Young University Educause, October 4, 2002 Atlanta, Georgia

2. Copyright Statement  Copyright Scott L. Howell, Stephen M. Jones and David W. Monson, 2002. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

3. Three Presentation Objectives/Questions 1.Stephen Jones - Introduction 2.Scott Howell What happens pedagogically to teaching and learning when a large-scale hybrid course redesign comes to campus? (Pew) 3.David Monson What literally happens to teaching and learning when a CMS system comes to campus? 4.Stephen Jones What are faculty and students saying about the impact of the CMS upon their teaching and learning?

4. Stephen Jones Off Campus On Campus

5. Independent Large Study Course Redesign Off Campus On Campus Stephen Jones Strategy—leverage technology- enhanced learning for campus, improve GE courses, experiment with cost/space savings Strategy—”extend the blessings of learning,” anywhere, anytime 350 Courses, 120 web-based 25K enrollments (Semester Online)  Independent learning model  Inexpensive, scalable  Content is of high quality  Award winning  Faculty can’t edit  Doesn’t fit well on campus  Hybrid learning model  Expensive  Media rich, interactive  Award winning  Faculty can’t edit  Doesn’t fit well off campus  Long production cycles

6. Independent Large Study Course Redesign Off Campus On Campus Stephen Jones (Semester Online) Faculty-developed using Blackboard

7. Off Campus On Campus Stephen Jones Independent Large Study Course Redesign (Semester Online)  Hybrid model  Inexpensive  Faculty controlled  Developed through iteration  Leverages existing content  Emphasizes student learning activities over expensive media development Faculty-developed using Blackboard

8. Off Campus On Campus Stephen Jones Result Faculty gain confidence in teaching with technology, develop and maintain their own courses, and leverage the power of the CMS for improved learning. Independent Large Study Course Redesign Faculty-developed using Blackboard (Semester Online)

9. Stephen Jones Independent Large Study Course Redesign (Semester Online) Off Campus On Campus The content faculty create can be further developed for on and off campus use. Faculty-developed using Blackboard

10. Scott Howell

11. “The move toward a hybrid education has been quieter than much-hyped efforts to create completely virtual programs. Graham B. Spanier, president of Pennsylvania State University, calls the convergence of online and resident instruction "the single-greatest unrecognized trend in higher education today," and he touted it as part of the vision for his university in a speech last year.”

12. “'Hybrid' Teaching Seeks to End the Divide Between Traditional and Online Instruction By blending approaches, colleges hope to save money and meet students' needs” Jeffery R. Young Chronicle of Higher Education, 3/22/02

13. “Within five years, you'll see a very significant number of classes that are available in a hybrid fashion," says John R. Bourne, a professor of electrical and computer engineering at Franklin W. Olin College of Engineering who is editor of the Journal of Asynchronous Learning Networks. "I would guess that somewhere in the 80- to 90- percent range of classes could sometime become hybrid." And he says he expects to see more students choose to take online courses even if they live on campus.”

14. “"A strong case is beginning to be made on the basis of research evidence that many students learn better online than face-to-face, and therefore a mixture is the best way," says Mr. Dede. "What proportion that mixture should be would vary from course to course.”

15. The Pew Program in Course Resign  A $6 million award program conducted by the Center for Academic Transformation at Rensselaer Polytechnic Institute with support from the Pew Charitable Trusts.  Encourages colleges and universities to redesign their instructional approaches using technology to achieve cost savings as well as quality enhancements.  Focuses on large-enrollment, introductory courses, which have the potential of impacting significant student numbers and generating substantial cost savings.

16. The Pew Program in Course Resign  BYU Awarded $200,000 in Round 3 (2001) to REDESIGN English 115: Freshman English Composition Course

17. Show PEW Video (5 min 45 seconds)

18.  Instructor Quality and Curriculum Varied Narrowly Before RedesignAfter Redesign  42 Classroom Meetings  170 Classrooms (7140 classroom hrs/yr)  2 Faculty/Student Conferences/Yr (15 Minutes)  154 Graduate Instructors/Yr (Classroom Size = 20)  0% Cost Savings  Instructor Quality and Curriculum Varied Widely  Instructor Time Significantly Higher  10 Classroom Meetings  45 Classrooms (1360 classroom hrs/yr – 74% Reduction)  120 Graduate Instructors/Yr (Classroom Size = 25)  40% Cost Savings  Instructor Time Significantly Lower  4 Faculty/Student Conferences/Yr (30 Minutes – 400% Increase) English 115 (per year)

19.  25% reduction in instructor time CategoryOnlineTraditional Preparation101.9 min/week117.56 min/week Instruction38.5 min/week132.82 min/week Grading146.25 min/week164.87 min/week Office Hours0 min/week147.69 min/week Email110.38 min/week17.31 min/week Conferencing75.63 min/week73.08 min/week Development27.38 min/week39.49 min/week Total500 min/week692.82 min/week Two recent findings: First Finding

20. Two recent findings: Second Finding  “Based on our initial pilot we are encouraged that overall paper quality is higher in the online versus the traditional version of the course. We are also encouraged that there was a significantly higher rating for the introductions and conclusions and focus and organizations. Because these are global writing skills that were a focus in the online modules we developed, we feel the online portions of the course helped improve student writing.”

21. David Monson

22. Course Management System Study  Conducted in the summer of 1999  Research team from the McKay School of Education’s Instructional Psychology and Technology Department  Email survey of 270 randomly selected faculty  12 In-depth Interviews  Usability testing of selected commercial products

23. Purpose of the Study Identify a suite of online course management tools that: collectively meet the instructional needs of faculty at Brigham Young University, address the administrative needs that support online instruction, and are compatible with present and future university technology standards for online instruction.

24. Study Goals  Determine present and future needs of BYU faculty for online course management through a campus-wide survey and in-depth interviews of faculty.  Use data about faculty needs to identify an ideal feature set for a suite of tools and support for online course management.  Identify what is being done at BYU to provide faculty with online course management support.

25. Study Goals  Identify what is being done at top universities to provide faculty with online course management support.  Evaluate commercial online course management products using the feature set as criteria.  Identify the top products and conduct usability testing with faculty.  Recommend to the university possible courses of action for providing faculty with online course management tools and support.

26. Functionality Ranked by Current Use  Communication  Syllabi  Announcements  Instructional resources in the classroom  Feedback to students  Faculty-to-student collaboration  Learning resources from other sources  Research resources

27. Findings Surveys and interviews showed the primary concern of faculty: lack of time to use the tools. lack of funds that might be used to pay students to do to the work for them, saving faculty time. university training and support of the tools provided. ease of use.

28. Usability Tests Usability tests with BYU faculty from each college and school on-campus were conducted on the top three commercial products: Blackboard CourseInfo WBTSystems TopClass Educational Technologies WebCT Results: all faculty ranked CourseInfo first for product ease of use.

29. Blackboard Adoption and Support Hardware-Software Support Marketing

30.  23.9% of sections are using Bb (2,003 of 8,365)  43.4% of enrollments are also Bb enrollments (82,820 of 190,767)  81% of students are enrolled in a Bb course (29,563 of 36,492)  65% of faculty are using Bb (1,626 of 2,492) Blackboard Adoption and Support

31. Stephen Jones

32. Growth of Hybrid Courses  80% of BYU students took one or more courses during the last year that were a mix of F2F and online.

33. Faculty User Types  716 courses created, Winter 2002  228 (32%) were empty course “shells”  488 had active users  72 “Power” Users14.8%  180 Moderate Users36.9%  236 Light Users48.4% (Winter 2002)

34. Blackboard TM Survey, Winter 2002 Faculty 100 most active faculty users 60 responses Students All students in those classes 1697 responses

35. Blackboard TM Feature Use Data  Announcements78.3%  Course Documents76.0%  Course Information61.7%  Assignments49.6%  Staff Information45.5%  External Links30.3%  Quizzes Online24.0%  Discussion Board19.3%  Groups2.3%  Virtual Classroom0.6% (Winter 2002)

36. StudentsFaculty  Improve access to course materials83%85%  Improve class communication (one and two way)70%83%  Provide more flexibility for instructor and students63%76%  Instructor just trying out Blackboard TM 26%27%  Academic dept. encourages instructor to use it20%20%  Student demand16%4% Why is Blackboard being used? (Check all that apply) Student / Faculty Comparison

37.  Yes78.0%  No17.3%  Undecided4.7% Overall, would you prefer instructors use Blackboard TM in their courses? Blackboard TM Student Survey

38. Blackboard TM Instructor Survey  “Many students like Blackboard TM courses. They enjoy working at their own pace, any time, day or night. They like not having to attend class. But many miss the face to face interaction and discussion and the discipline required of a traditional class environment. “ – Audiology & Speech Pathology Professor How do you think Blackboard TM has been received by your students? (Winter 2002)

39. Blackboard TM Student Survey  Yes57%  No41%  Unanswered 2% Do you think Blackboard TM is changing the way instructors teach their courses? (Winter 2002)

40.  “It creates a more encompassing learning atmosphere. The teachers can refer the students to much more material and in a much more interactive environment. Blackboard is awesome.”  “Materials are more readily available. Communication is easier. Quizzes online save valuable class time. More time is spent learning in class and not being tested.”  “In a positive manner, it allows for more discussion among classmates. We can share ideas, and then learn from the ideas of others. On a negative side, some teachers have put so much on the Blackboard TM, that it is assumed that we should find all of the answers.” Blackboard TM Student Survey

41. Blackboard TM Instructor Survey  87% indicated a change in their practice  13% indicated no change What is the most significant teaching practice you have (or will) change now that this tool is available? (Winter 2002)

42. Blackboard TM Instructor Survey  Post materials online - 40% Student comments  “I can access grades, quizzes, tests and many other teaching materials online that were otherwise unavailable to me before.”  “I can do it at any computer on campus. I don’t have to carry around tons of papers, and I know I won’t lose it.” What is the most significant teaching practice you have (or will) change now that this tool is available? (Winter 2002)

43. Blackboard TM Instructor Survey  Increase faculty-to-student and student-to- student collaboration and communication - 32% Student comments  “I can contact other students in the class much easier through Blackboard. Group projects are much smoother when we have the use of Blackboard.”  “Using Blackboard has increased my team member skills.” Faculty comment  “I communicate with my students more frequently now via Blackboard than I had in the past.” What is the most significant teaching practice you have (or will) change now that this tool is available? (Winter 2002)

44. Blackboard TM Instructor Survey  Provide quizzes and homework online - 10%  Use to increase pre-class preparation - 5% What is the most significant teaching practice you have (or will) change now that this tool is available? (Winter 2002)

45.  1. Group problem-based learning activities with online collaboration. 2. Organizing out-of-class time. 3. Preparing students for class discussion-- they post reactions to reading before coming to class.” – Political Science Professor  I am able to encourage students to learn from each other and to share at a deeper levels. My goal with this course has always been to encourage the use of critical thinking skills. Using Blackboard TM has allowed me to facilitate this type of interaction on a regular basis. Also, I am able to adjust course content to meet the current needs of the student interns. -- Family Science Professor What is the most significant teaching practice you have (or will) change now that this tool is available? Blackboard TM Instructor Survey

46. Are you meeting less often in class and more often online? StudentsFaculty  Yes26% 18%  No74%82% Student / Faculty Comparison

47. Blackboard TM Student Survey  “Instead of three times a week, we meet once. But this is great, I have no complaints. I'm learning just as much if not more. I can go at my own speed.”  “My Org. Behavior class has all of our assignments and quizzes on BlackboardTM, with optional classroom lectures six times throughout the semester. I liked it a lot, I could complete my quizzes and things on my own time.” Student comments (Winter 2002)

48. Blackboard TM Instructor Survey  “We'll be in class two days a week instead of three. One day a week is an 'e-day.‘” – Political Science Professor  “Spring Term 2002 I will use the virtual classroom in three 1/2 hour periods during the week and allot additional time for electronic communication with students.” – Zoology Professor  “Yes, I meet more often online. Monday we meet in person, Wednesday in the virtual classroom and Friday my students post to the Discussion Board. – English Instructor (Winter 2002) Faculty comments

49. Blackboard TM Student Survey  Yes 13%  Not changed78%  Decreased 7%  No response 2% (Winter 2002) Overall, is your workload as a student changed by an instructor’s use of Blackboard TM ?

50. Blackboard TM Student Survey  “I believe that they think it's easier for us to get more done so all of my teachers who use it pile more work into it.”  “I do more in my on-line class then my friend does in the same class, because there are always tests to make sure that we understand the material and are staying caught up.”  “I believe that with Blackboard TM, I am able to better manage my homework situation; therefore, I have been able to decrease my workload.”  “It is easier when I can use Blackboard TM. It saves my time a lot!” (Winter 2002) Student comments

51. StudentsFaculty  Positive comments68%78%  Negative comments2%12%  Neutral1%0%  No response29%10% Can you think of at least one example of Blackboard TM use that has helped your learning / teaching experience? (Winter 2002) Student / Faculty Comparison

52. Blackboard TM Student Survey  “I like being able to read other's responses to what we discussed in class.”  “It allows me to access information when I need it. Timing is very important... If I'm working on a project, but am behind a little I can access the information at my own pace.”  “It has made communicating with teachers, TAs, and classmates MUCH easier. Specifically when we are doing group projects Blackboard TM allows us to have a common meeting place.”  “It allows me to take quizzes at my convenience.” Student comments (Winter 2002)

53. Blackboard TM Instructor Survey  “I think I'm doing a better job of focusing the students' learning experience and delivering a specific and meaningful learning outcome.” – Organizational Leadership Professor Faculty comments (Winter 2002)

54. Blackboard TM Student Survey  58% responded with at least one comment  42% did not respond Can you think of at least one example of Blackboard TM use that has frustrated your learning experience? (Winter 2002)

55. Blackboard TM Student Survey  30.8% - technical problems  5% - Student dislike for online instruction  4.4% - Professors not updating courses enough to be helpful  2.9% - Teachers post assignments right before class  2.8% - Hard to get computer access Top-ranking categories of negative responses (Winter 2002)

56. Blackboard TM Student Survey  “I prefer more interpersonal communication as opposed to the mass communication style of Blackboard TM when it comes to my education.”  “Sometimes instructors expect that if they post something on Blackboard TM right before class we will have time to look at it. It seems to encourage professors to procrastinate.”  “…sometimes navigation of the site isn't that easy. Many professors put things in different locations.”  “It's only frustrating when I can't get into it. Like, when…the system is down for a while...” Can you think of at least one example of Blackboard TM use that has frustrated your learning experience? (Winter 2002)

57. Blackboard TM Instructor Survey  “Has provided ease in communicating with class through mass emails, tracking grades on-line, ect.” – Family Science Professor  “The last test… proved clearly that those who used the online materials did very well. Those who didn't had test scores MUCH lower.” – Italian Instructor  “It makes it so that I don't have to be personally available quite as often, which is definitely valuable when you are teaching 200 students in one course.” – Psychology Instructor Has this tool helped create any efficiencies in managing your courses? (Winter 2002)

58. Blackboard TM Instructor Survey  “I, and my TA's, use the Communication link to contact students who have unusually high absences. We encourage them to come and see us. In a large class (166 students) it is way too easy to get lost and lose interest…” - Political Science Professor  “Hybrid is the way to go - part online for efficiency, part classroom for personal touches.” - Counseling & Special Ed Professor General comments and observations about your Blackboard TM experience including the tool itself. (Winter 2002)

59. Seven Principles of Good Practice  Good Practice  Encourages Contacts Between Students and Faculty  Develops Reciprocity and Cooperation Among Students  Uses Active Learning Techniques  Gives Prompt Feedback  Emphasizes Time on Task  Communicates High Expectations  Respects Diverse Talents and Ways of Learning Arthur Chickering and Stephen Ehrmann AAHE Bulletin, October 1996

60. “My ASLP teacher has used the discussion board a lot, and I really think that helps in learning. I find it really beneficial to be able to see what other students think and what they are getting out of our assignments.” Freshman Contact, Cooperation, Active Learning, Feedback, Diverse Ways

61. Time on Task “It allowed me to listen in my religion class, and not get caught up in taking notes,...” Freshman “...we could go at our own pace and rate of understanding rather than having to keep up with the class (either being a faster or slower learner than the rest of the class.) Senior “...at least no time is wasted in class trying to figure out who has what grades.” Senior

62. “The more contact I make with them through technology, the more my students seem to believe that I am readily accessible to help them…” – English Instructor Virtual can = Available Contact

63. “Helped me to become more technically literate.” Freshman Diverse Ways

64. “If I can’t remember if we have a homework assignment. No Worries. I don’t stress about it and cry, because I know that I can always go to Blackboard and look up what my assignment is. It has saved me a lot of useless tears.” Freshman Reduced Stress

65. Conclusions, Observations  The ability to extend the classroom using a virtual environment can create a seamless, connected experience for students that  Helps organize their out of class time  Connects them to each other and the professor  Puts them in control, offers choice  Greatly improves their access to information

66. Conclusions, Observations  Teaching a class using a CMS causes faculty to  Think reflectively about their practice  Align content, assessment, and learning activities  Think about the type and quality of student-to- student interaction needed outside of class  Use assessment more strategically and make grades (and rubrics) more public

67. Video

68. Next Steps, Future Opportunities  Move from  Uploading documents to creating and publishing learning objects  Online content (“Putting my course online”) to communities of practice  Individual, opportunistic to institutional, strategic course development  Tool training to pedagogical dialogue, improvement  “Slide, slide, quiz” to adaptive assessment, intelligent tutoring  Move from hybrid / blended/ e- / online learning to improving learning  Focus away from the technology and toward the goal of improvement