1. 1 LMS Lecture-1: Assessing Course Management Systems week 1- Semester-2/ 2011 Dr. Anwar Mousa University of Palestine Faculty of Information Technology

2. 2 Assessing Course Management Systems Abstract The authors examine CMS as a new enterprise technology. Using a model of transformative assessment that frames value within alignment of institutional goals and mission, they define the questions that should be asked of each of the CMS stakeholders: teachers, learners, support services, leadership and the CMS vendor.

3. 3 Assessing Course Management Systems Introduction A few brief years ago, we knew not course management systems (CMS). Technology-enhanced learning often meant expensive initiatives developed by instructional designers and computer programmers. Those faculty incorporating technology into their curricula were usually entrepreneurs, risk-takers, or generously funded and safely tenured. Funding was for narrow science applications, research repositories, or large course redesign.

4. 4 Assessing Course Management Systems Few incentives were offered to take the risks associated with a technology-infused curriculum. Hagner (2001) reports that only a few faculty “entrepreneurs” were willing to take on the hard work and risk of failure associated with technology when the needed shift in institutional reward, collaboration, support, and expectation was missing. Then, something happened. Across public and private institutions large and small, faculty embraced course management systems. In a short period of time, small departmental installations over-extended capacity and quickly gave way to institutional enterprise applications. Faculty development centers became CMS support sites.

5. 5 Assessing Course Management Systems University technology staff became CMS experts. Instructional designers wrote thousands of best practices handbooks for effective discussion boards and online learning. Students began to ask if there was a CMS site for the course before registering. Universities began to create notation, policy, and process for technology-enhanced, hybrid, and online courses. In a culture where change never happens, something happened. What happened? What were faculty members doing within the CMS framework? Did we find a tool that promotes deeper learning and better teaching?

6. 6 Assessing Course Management Systems Did the CMS make ordinary tasks easier for the instructor? Did the student engage in the material in a more direct or engaged way? Most anyone associated with the use of CMS will tell you that they don’t know the answers, haven’t asked, and aren’t sure what to measure. Common thought is that if so many faculty are using CMS, this must be good for teaching. If the students are embracing the new modes of delivery, something about the CMS must be effective for learning.

7. 7 Assessing Course Management Systems Others express the feeling that technology alone is not enough, and perhaps less “enhanced learning” is happening within the CMS space than assumed. In both camps, assumptions guide current thinking about reasons for the quick and widespread adoption of CMS. What is missing is assessment. With CMS now an integral part of much of our teaching, learning, and support endeavors, we must begin to seriously examine its value to the institution.

8. 8 Assessing Course Management Systems While higher education continues to embrace the CMS in enterprise technology, and in faculty development and support, the work to determine, understand, and justify its place and value to the institution’s strategic mission is still seriously missing. Questions of cost/benefit, function, and usage have yet to be addressed. In this chapter, we hope to refine some of these questions by examining the roles and needs of each of the stakeholders in CMS: teachers, students, support services, leadership, and vendors.

9. 9 Assessing Course Management Systems Seen from a model of “transformative assessment, assessment is the purposeful gathering and use of data to ensure that the application of findings and the dissemination of results will substantially change and enrich the learning experience. Transformative assessment strategies are based on institutional goals and missions and are implemented in an integrated way for all levels (the course, the program, and the institution). It is evident that consistent and consensus-driven goals of the stakeholders of CMS are not yet defined and that quality evidence still needs to be gathered to inform strategies for meeting these goals.

10. 10 Assessing Course Management Systems Transformative assessment requires an awareness of the need for alignment on the part of the institution. Within each of the stakeholder sections of this chapter, we offer possibilities for this alignment. We also examine the CMS issues facing those who define the missions of the university as they apply to CMS: teaching goals, learning outcomes, fiscal value, intellectual property rights, and enterprise values. Finally, transformative assessment also calls for improving the dissemination of findings to include, as much as possible, our governing community — where a need for understanding will increases with each passing day.

11. 11 Assessing Course Management Systems Teaching and CMS What attracts faculty to CMS? What features made it possible for such widespread adoption so quickly? What evidence exists that those features satisfy original expectations and aid in teaching? In what ways? Time saving? Reaching diverse students? Easier access to materials?

12. 12 Assessing Course Management Systems The literature is unclear regarding the known factors that continue to drive the increasingly costly delivery of CMS. We have self-reporting data on the most frequently adopted tools within the CMS. But, we have no evidence of the teaching and learning value attributable to faculty use. Despite the prevalence of CMS as a presumed enhancement to face-to-face courses, we see little evidence in the literature on what is being enhanced.

13. 13 Assessing Course Management Systems In the past, no significant difference (NSD) was enough, demonstrating twenty years of evidence that learning (as assessed by tests, grades, and measures of short-term demonstrations of memorizing course materials) does not suffer when done online or through technology mediation. It is now understood that online instructional methods neither, in general, obstruct nor help learning, but the widespread adoption of CMS in traditional instruction — and the costs in faculty time, technology resources, support services, and demands on the student — suggest that it is now crucial we ask more focused questions of CMS and the technology- enhanced model: how does it help, and what, exactly, does “help” mean?

14. 14 Assessing Course Management Systems If the value is not in the grade but depends on some measure of faculty engagement, student satisfaction, ubiquitous access, or incorporation of deeper learning theory, where is the evidence that strategic goals for these measures are being articulated, let alone reached? For instance: –How do we know that our students know what we hope they will know? –How are the tools in CMS changing the ways faculty teach and students learn, and how, subsequently, does that change how well and what they learn?

15. 15 Assessing Course Management Systems –How do those changes impact the services being reallocated to support CMS? –What was given up to bring these new services into information technology (IT), teaching and learning centers, faculty development programs, and the library? –How do we ensure that the changes are justified?

16. 16 Assessing Course Management Systems  Within this book, you’ll find much discussion of changes, features, and improvements desired for next-generation CMS.  Many of the ideas are based on hard, collaborative work by EDUCAUSE and the National Learning Infrastructure Initiative (NLII) to bring vendors, faculty, researchers, administrators, innovators, IT, and instructional designers together to determine consensus on what is needed in the next generation.  If we build it, they will come. We did, they did. Now what?

17. 17 Assessing Course Management Systems Alignment: Teaching and CMS  As the CMS community continues to work on better technology, reports on current usage suggest that faculty currently use few of the features available.  Faculty must move beyond intuitive value and explore ways the CMS permits faculty to serve the mission of engaged teaching and learning.  Faculty members embraced CMS independent of institutional support, but now struggle to find the time or resources to examine the role of CMS in better pedagogy.

18. 18 Assessing Course Management Systems  Innovation and adaptation to the changing landscape will not happen without institutional commitment to change.  For the next-generation systems to be truly effective learning environments, faculty must move from easy-to- use online organizers to effective pedagogical tools for assisting and assessing desired learning outcomes.

19. 19 Assessing Course Management Systems Learning and CMS  Evidence from the literature of the last ten years suggests agreement on a number of conditions that, when in place, contribute to the learner’s deep, engaged understanding of the content.  Carmean and Haefner (2002) examined some of the core research available on deeper learning, summarizing these conditions by suggesting that deeper learning takes place when it:  1) is social,  2) is active,

20. 20 Assessing Course Management Systems  3) is contextual (related back to the conditions and understandings of the learner),  4) requires learner ownership, and  5) engages the learner. They used these conditions to examine the tools commonly present in CMS and suggest practices for the tools that could allow the conditions that manifest in deeper learning. Theirs were merely suggestions on how the tools might be used in relationship to necessary conditions for deep learning, not evidence that the tools are being used effectively or that deep, engaged, lasting learning is taking place.

21. 21 Assessing Course Management Systems Is there evidence that the CMS tools are used to provide an environment where inquiry, not delivery, is promoted? Why should we expect any technology to learn, especially as we avoid the harder question: What do we mean by improve learning? What most important features or sets of features would help students deeper learning conditions to a structured, anytime/anywhere, and independently “improve learning”? In what context, and for whom? Can we apply these navigated environment? Does it allow for faculty members to assist students in exploring, questioning, and researching course content in ways that support diverse learning?

22. 22 Assessing Course Management Systems Alignment: Learning and CMS How could one measure whether CMS usage contributes to features of authentic learning? We suggest a two-fold approach: 1. Compare the features available against a set of agreed- upon, known conditions that need to be met for the learning outcomes established. For instance, is the learning activity designed to be social, active, relevant, engaging, or creating learner ownership?

23. 23 Assessing Course Management Systems 2. Observe student engagement as well as collect student responses to the use of CMS features in relationship to meeting those conditions. Did the discussion board provide opportunity for rich exchange of ideas? Did the announcement board allow for better navigation and ownership of the tasks at hand? Was the immediate quiz feedback used to encourage self-assessment and mastery of the material? It is not an empty course shell that improves learning, so we now must ask under what conditions the features available in standard CMS lend themselves to informed design of an engaging learning environment.

24. 24 Assessing Course Management Systems Did the usage of a CMS allow for recreating a more flexible course experience through online or hybrid offerings? Does less seat time and more independent inquiry make for more engaged learning, or merely fewer obstacles to getting a degree? Assessing learning should begin by clearly articulating learning outcomes and asking the students what they have learned. And, finally, learning assessment might also include simple satisfaction surveys and focus groups that explore the experience of being a student.

25. 25 Assessing Course Management Systems We will never know that our students know what we want them to know until we ask them, and, correspondingly, we may never know what our students want to know unless we listen to them. Support Services for CMS In technology use in teaching and learning, the strategic mission of university support services is to provide the resources to help faculty help their students. Some of these resources include stable systems, in-time training, and available assistance. With the growth of CMS, many services are being reworked to provide these resources.

26. 26 Assessing Course Management Systems As instructional support (in IT, the library, teaching and learning centers, and faculty development areas) is increasingly reallocated in support of CMS, what are these support units giving up to bring new CMS related services online? Are the right people and tools in place to effectively offer these services, both to faculty and to students? Are instructional support services ensuring that use of CMS is being deployed in alignment with institutional goals and in the support of best practices? Are support services implementing enterprise CMS with full consideration of stakeholders and are they being asked to assess return on investment?

27. 27 Assessing Course Management Systems Technology literacy is a valuable goal for student learning outcomes, but what services need to be in place campus-wide to assure that students are comfortable with CMS use, when not anticipated in course delivery? What is being done to bring students that need assistance with the technology up to speed? How is fear of technology hampering students and faculty from moving forward in the use of CMS, and what can support services do to mitigate the risk of technology failure in production usage of CMS?

28. 28 Assessing Course Management Systems Alignment: Support Services and CMS The role of support is complicated by the pace of change we see in student expectations. New students want and expect better use of technology in their coursework. Calhoun (2004) addresses this pressure, acknowledging the difficulty faculty experience in keeping up, without reward and with numerous barriers to thoughtful implementation.

29. 29 Assessing Course Management Systems In implementing technology, much responsibility for access, implementation, authentication, performance, data security, and instructional and pedagogical consulting now rests in the hands of system and instructional support services. Delivery of these services must be evaluated in alignment with clearly understood needs of the institution. Despite the pace of change, successful implementation and support for next-generation systems will depend on clear goals, thoughtful planning, and objective assessment of service to academic strategic plans.

30. 30 Assessing Course Management Systems Leadership and CMS As we discovered earlier in this chapter, other institutional stakeholders have been quick to respond to faculty interest in CMS. Great resources have been reallocated to allow any and every course a shell on the system. Simple servers have been converted to enterprise systems and integrated into portals. It would be rare to find such a large initiative taken on by another type of institution with no evidence of the value to bottom line, productivity, or mission. Why did leadership of academia move forward so quickly without evidence of value?

31. 31 Assessing Course Management Systems How many, long removed from teaching, took a hands-off approach only to now wonder what was wrought? Is leadership prepared to suggest negative return on investment is simply not a sustainable strategy? Technology units had previously been more comfortable supplying resources for research when suddenly a simple solution across the curriculum appeared. CIOs could now please the teaching community without the expenses previously associated with technology-enhanced learning.

32. 32 Assessing Course Management Systems Before the effects of rapid implementation could be seriously measured, IT leaders were moving forward to choose enterprise CMS. They would soon hire or reallocate system administrators, course creation staff, faculty trainers and application support staff. The commitment to support of CMS is now considerable, as is the unanticipated cost. And what about learning? Libraries, faculty development centers, and teaching and learning sites soon joined on. Did the campus leadership ever understand the consequences to the institution?

33. 33 Assessing Course Management Systems Were they consulted? Without careful assessment or consensus, institutions often committed great resources to a new support enterprise, not to mention to one or more course management systems from which it would be very difficult to separate themselves. In fact, a recent argument for investing in stock in a well- known CMS stated: “Since such a migration is typically a complicated and tedious task, customers have considerable motivation to stick with a given solution…. Consequently, Blackboard has a nice moat against competitors and the potential to raise prices as the product becomes indispensable to customers” (Gibbons, 2004, p. 1).

34. 34 Assessing Course Management Systems Based on decisions made with good intentions and little sense of consequence, course management systems became a part of the administrative cost within the technology and support services infrastructures. Serious concerns now facing administrators include: The increasing cost of CMS licenses. The costs of switching to open-source solutions, problematic for a host of reasons, not the least of which are code documentation, systems integration, reliability, and migration from systems that have established a loyal, even fanatic, user base. The support infrastructure needs that continue to grow as faculty increase use, questions, sophistication, and demand.

35. 35 Assessing Course Management Systems.The ownership of material when faculty develop content within the shell of a course they may have difficulty accessing later, due to authentication or archiving. The issue becomes more pressing when considering the high percentage of adjunct faculty now teaching in higher education. Does the administration have an obligation to make it clear to these individuals that it might be difficult, even impossible, to take their material with them when leaving the institution?

36. 36 Assessing Course Management Systems Alignment: Leadership and CMS How do we measure the value of these new enterprise CMS? How do we balance value against the current and often unknown inability of faculty to remove their own material from the course shell? Who is measuring the cost of the exclusion of the library from the electronic learning experience? Where are the policies outlining the rights of students to intellectual property on online work and discussion contributions from years before?

37. 37 Assessing Course Management Systems What would we do if assessment showed harm from loss of intellectual property as well as no significant difference, no greater satisfaction, and no worthwhile value from the change? It is imperative to the success of next-generation learning that university administrators step forward and do their job. Ask CIOs, deans, department chairs, and teaching support units to be responsible for assessment, return on investment, and a demonstrated commitment to meet clear, and measurable goals.

38. 38 Assessing Course Management Systems The Vendor The CMS market is narrowing, with most institutions having settled on one of three to five vendors, though there are still about 1,000 by a recent count (CMS Watch, 2004). Within these vendors, as with the few open-source alternatives being developed, there is a base set of features that are expected, relatively uniform, and functionally indistinguishable. Are they the right feature set? Are they well used? Are there best practices for the features that can inform us on better teaching and learning?

39. 39 Assessing Course Management Systems As experts argue for more sophisticated features, mainstream faculty argue for ease of use and a more intuitive, simple interface. Morgan’s (2003) EDUCAUSE Center for Applied Research (ECAR) report on Faculty Use of Course Management Systems surveyed 730 faculty members across the Wisconsin system and concluded that very few of the features within CMS are regularly used. This does not mean that the features being used (storing syllabus, posting announcements, course document functionality) are not of value, but what would evidence of greater value look like?

40. 40 Assessing Course Management Systems If faculty members are not using many of the current features (or use a few features to achieve multiple functions), what motivations exist for the vendor to continue building more features into increasingly complex systems? As new systems like ePortfolios emerge to address increasingly similar functions, when does feature-creep become system overlap? The CIO pays the bill, instructional designers argue for system-integration of theory, faculty request for simplicity, futurists ask for tools now used by the younger generation of students, and learning theorists ask for better integration of modules for outcome assessment.

41. 41 Assessing Course Management Systems Ignored on all sides are the students that must weigh time- demand over required use. Vendors work to locate consensus, common practice, and consistent feedback. But they are torn by the competing requirements of a diverse set of constituencies and wonder where the profit lies in an increasingly saturated market. Alignment: The Vendor and CMS Vendors must understand their constituents, not just the CIO wallet, and work diligently to collaborate with higher education to define the next-generation system.

42. 42 Assessing Course Management Systems Higher education, in turn, must assess value and features and reach consensus on what is needed. Enterprise systems will grow larger and more expensive as portals, features, tools, and processing power improve. What choices will we make on implementation?

43. 43 Assessing Course Management Systems Summary Change happens as a response to stimulus. This response can be reactive or a studied and sometimes difficult choice. The studied, institutional response to this change (transformative assessment) must carefully be implemented if the academy is to make decisions that radically affect so many stakeholders, as well as the direction and mission of the institution and the future of higher education.

44. 44 Assessing Course Management Systems John Tagg (2004) writes in a recent issue of About Campus, “Kellogg and Post aim to make cornflakes, Ford and General Motors aim to make cars, hospitals aim to make people healthier. Colleges, judged by the same standards and by the evidence of their own documentation, aim to have people take classes. It is as if Kellogg saw its function as grinding up great amounts of corn, or the RAND Corporation sought to fill as many pages as possible with reports. Kellogg knows what the corn is for. RAND knows what the reports are for. What are the classes for? (p. 3)”

45. 45 Assessing Course Management Systems It is time to ask what we’re doing, whether we’re doing it well, and whether the tools we’re purchasing to do it serve the goals associated with getting it done. The fundamental purpose of transformative assessment is to improve student learning outcomes, and it is clear that across the academy CMS is seen as a positive tool for doing so. Based on the commitment to CMS now in place across the institution, careful assessment and inquiry should help us determine its value to teaching, learning, and the changes needed in higher education.

46. 46 Assessing Course Management Systems To achieve transformation that will be responsive to the new student, new economy, and struggling academy, we must examine the design, planning, implementation, and evaluation of CMS for each of the constituencies involved: teachers, learners, support services, leadership, and vendors. To be useful in transforming the institution, assessment of the CMS must be studied in a new way — as a tool for learning. By asking the right questions and being willing to act on the answers, we create the possibility of graduating engaged, successful, and diverse learners.