1. STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot Collection Emrah Orhun, Ph.D. Troy State University Montgomery Alabama, USA

2. Outline Introduction Introduction Learning and Teaching Materials in The Merlot Collection Learning and Teaching Materials in The Merlot Collection Students’ Satisfaction with Merlot Materials in CIS Classes Students’ Satisfaction with Merlot Materials in CIS Classes Conclusions Conclusions

3. Introduction Difficulties in locating relevant and affordable materials of high quality Difficulties in locating relevant and affordable materials of high quality –World Lecture Hall at the University of Texas »83 categories to browse and search utilities to locate materials –WebCT (2002) »1,250 publisher course content packs in more than 50 subject areas »customizable materials around which to build courses combined with the functionality of WebCT's course management software.

4. Quality issue World Lecture Hall materials World Lecture Hall materials –courses offered at accredited colleges and universities around the world WebCT packs WebCT packs –editorial standards of the publisher and author of the text

5. MERLOT (Multimedia Educational Resource for Learning and Online Teaching) Web-based open resource designed primarily for faculty and students in higher education Web-based open resource designed primarily for faculty and students in higher education emphasizes the quality issue emphasizes the quality issue

6. MERLOT repository MERLOT repository –Over 8,000 free learning materials and support materials that can be integrated within the context of a larger course

7. Searching and Browsing in MERLOT

8. Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category

9. Figure 2. Search results for TECHtionary

10. Support resources –sample assignments –evaluations of the learning materials »discipline panel peer reviews »individual user reviews

11. Peer Review Process Discipline communities Discipline communities Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages. Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages.

12. Discipline teams Discipline teams (“MERLOT Editorial Boards”) –selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. –develop evaluation standards, –conduct peer reviews of learning materials within the discipline, and –make connections with their professional organizations

13. Peer review process Evaluated by at least two team members who have relevant expertise. Evaluated by at least two team members who have relevant expertise. Review process stages Review process stages –triage review to identify worthy candidates –review of candidates –composite review report sent back to the authors for feedback and permission to post the review on MERLOT. –approved review is posted to the MERLOT Web site by the discipline leader

14. The evaluation criteria: The evaluation criteria: –quality of content, –potential effectiveness as a teaching tool, –and ease of use.

15. Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed A review must average three stars in order to be posted to the site. A review must average three stars in order to be posted to the site.

16. Figure 3(a). Peer Review of TECHtionary (Part a)

17. Figure 3(b). Peer Review of TECHtionary (Part b)

18. Figure 3(c). Peer Review of TECHtionary (Part c)

19. Figure 4. Member Comments for TECHtionary

20. Current state of MERLOT

21. Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002) TotalAdded during last 365 days Added during last 30 days Materials68631991229 Members86134142600 Peer Reviews 47936120

22. Table 1 Growth of MERLOT materials, members, and peer reviews ( February 27, 2003 ) TotalAdded during last 365 days Added during last 30 days Materials83312338284 Members126375527712 Peer Reviews 778862 Member Comments 179030518 Assignments322591

23. Table 1 Growth of MERLOT materials, members, and peer reviews for IT Category TotalAdded during last 365 days Added during last 30 days Materials55927421 Members52425930 Peer Reviews 3630 Member Comments 218996 Assignments1190

24. May 25, 2002 September 26, 2002 February 27, 2003 Materials686373958331 % Peer Reviews 7.08.59.3 % Member Comments 21.220.821.4 % Assignments 3.43.83.9 Ratio of Peer Reviews, Member Comments, and Assignments to Materials

25. Problem Percentage of reviewed materials low Percentage of reviewed materials low Potential Solution Peer review by community of learners Peer review by community of learners Cause Editorial peer review process too slow Editorial peer review process too slow

26. STUDENTS’ SATISFACTION WITH MERLOT MATERIALS IN CIS CLASSES

27. Goal of Study To find out to what extent MERLOT provides relevant and quality materials for the CIS courses. To find out to what extent MERLOT provides relevant and quality materials for the CIS courses.

28. Questions Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? Is the MERLOT collection ‘usable’ by the students? Is the MERLOT collection ‘usable’ by the students? Does the MERLOT collection induce the students to become members of learning communities? Does the MERLOT collection induce the students to become members of learning communities? Can peer review by students substitute for peer review by editorial boards? Can peer review by students substitute for peer review by editorial boards?

29. Questions re-formulated How do students rate their satisfaction with MERLOT collection? How do students rate their satisfaction with MERLOT collection? How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members? How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members?

30. Teaching with the MERLOT Collection in CIS Courses Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: –CIS 5547 CARApplied Systems Analysis –CIS 5547 CBRApplied Systems Analysis –CIS 3332 Software Engineering I –CIS 3357 Logical Structures of Computer Design Overall, 41 students participated in the evaluation. A total of 252 evaluations were made. Overall, 41 students participated in the evaluation. A total of 252 evaluations were made.

31. Evaluation of web-based learning materials 1.1. Visit MERLOT (www.merlot.org) site and register as a member. 2.2. For each chapter covered in the course: 3.a. Find a relevant learning material in the IT subcategory. 4.b. Review the material using the guidelines provided by MERLOT. In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT. 5.c. Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all. 6.d. Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment. Student Assignment

32. ChapterTopicTitleURL Learning outcome Satisfaction with MERLOT collection for this task 5= very good 1=very poor 1 Players in the System Game Enterprise Resource Planning Supersite http://www.erp supersite.com http://www.erp supersite.com Provides analysis on ERP trends 3 2 Information System Building Blocks Interface Design for Educational Multimedia http://www.cdl. edu.rachel/PD F/intdesmm.pd f http://www.cdl. edu.rachel/PD F/intdesmm.pd f Describes the principle of interface design 4 3 Information System Development Risk analysis and management for Projects (RAMP) http://www.ra mprisk.com/fro ntindex.htm http://www.ra mprisk.com/fro ntindex.htm Comprehensive outline of RAMP 5 Example of Relevant MERLOT Material

33. ChapterTopicTitleURL Learning outcome 1 Players in the System Game New responsibilities Fall Heavy on IT Workers www.informatio nweek.com www.informatio nweek.com Talks about the new job responsibilities of current and upcoming IT employees. 2 Information System Building Blocks TP Middleware keeping up with Mainframe Revival www.adtmag.co m www.adtmag.co m Explains the functionality of the product TP middleware on mainframe systems. 3 Information System Development Tips for a successful Software Distribution Project. www.computerw orld.com www.computerw orld.com Shares other companies success stories of in relation to Software Distribution Project. Example of New MERLOT material posted

34. How do students rate their satisfaction with MERLOT collection?

35. COURSE Numbe r of Studen ts Numbe r of Topics Numbe r of Evaluat ions Averag esdev CIS3357 Logical Structures of Computer Design87493.81.3 CIS5547 (A+B) Applied Systems Analysis22141653.80.8 CIS3332 Software Engineering I114383.81.0 Students’ mean satisfaction ratings for MERLOT collection

36. ChCh Topic N Averag esdevMinMax Percent rated 4-5 1Binary Numbers & Codes 84.71.43587.5 2Digital Circuits 84.61.53575.0 3Combinational Circuits 84.31.91562.5 4Sequential Logic Systems 84.21.91562.5 5Registers & Counters 83.82.11550.0 6Memory & Programmable Logic 53.21.51540.0 7Register Transfer & Computer Operations 43.81.91575.0 CIS3357 Students’ satisfaction with MERLOT collection

37. ChCh Topic N Averag esdevMinMax Percent rated 4-5 1Process 134.20.81576.9 2Project Management 124.20.42591.7 3Requirements Analysis I 113.81.21572.7 4Requirements Analysis II 103.61.21570.0 5Software Architecture 134.20.81576.9 6Detailed Design 124.20.42591.7 CIS3332 Students’ satisfaction with MERLOT collection

38. ChCh Topic N Avera gesdevMinMax Percent rated 4-5 1Players in the System Game223.60.82559.1 2 Information System Building Blocks 203.70.82555.0 3Information System Development193.61.21563.2 4Project Management104.10.73580.0 5Systems Analysis123.80.63575.0 6Requirements Discovery123.90.83566.7 1 Application Architecture and Modeling 103.90.73570.0 CIS5547 Students’ satisfaction with MERLOT collection

39. AObject-Oriented Analysis and Modeling93.70.73555.6 BObject-Oriented Design and Modeling93.90.83566.7 1313 Output Design and Prototyping94.00.53588.9 1414 Input Design and Prototyping94.20.445100.0 1515 User Interface Design93.90.83566.7 1616 Systems Construction and Implementation 74.10.93571.4 1717 Systems Operations and Support84.10.83575.0 CIS5547 Students’ satisfaction with MERLOT collection (contd.)

40. How different are students’ evaluations from Editorial Peer Reviews?

41. Type************ Tutorial File System Design for an NFS File Server Appliance File System Design for an NFS File Server Appliance Cascading Style Sheets, Level1 Cascading Style Sheets, Level1 Electronic Commerce Tutorial Electronic Commerce Tutorial FTP Tutorial FTP Tutorial Learning Microsoft Word 2000 Learning Microsoft Word 2000 Web-Database Integration Web-Database Integration Mulder's Cascading Style Sheets Tutorial Mulder's Cascading Style Sheets Tutorial Sharky's Netscape Frames Tutorial Sharky's Netscape Frames Tutorial Tje ABCs of Building a Web Site Tje ABCs of Building a Web Site Data Communication s Data Communication s ADSL Computer Knowledge Virus Tutorial Computer Knowledge Virus Tutorial Reference Material Study Guides and Strategies Study Guides and Strategies Tips for Using Blackboard Tips for Using Blackboard Animation TECHtionary Warriors of the net Warriors of the net Distribution of materials in IT

42. Material Type ************ Tutorial ADSL Reference Material Web-Database Integration Web-Database Integration Animation TECHtionary Distribution of materials according to type and rating (excluding materials with no user comments with 3 categories) Sample of Materials from IT Category

43. Sample of Students CIS3332 SOFTWARE ENGINEERING (16) CIS3332 SOFTWARE ENGINEERING (16) CIS5547 SYSTEMS ANALYSIS (27) CIS5547 SYSTEMS ANALYSIS (27)

44. Case Editorial Board Rating Member Rating Students’ Rating (mean) TECHtionary 3.0*****3.75 Web-Database Integration Web-Database Integration 3.67***3.82 Students’ Ratings

45. Mean Std. Dev. Std. Error VarianceCoef.Var.Count 3.754.421.086.17711.21824 MinimumMaximumRangeSum Sum Squared # Missing 2.54.41.990.1342.330 Students’ ratings for TECHtionary Editorial Board Rating: 3.00

46. Mean Std. Dev. Std. Error Variance Coef. Var. Count 3.829.545.119.29714.23821 MinimumMaximumRangeSum Sum Squared # Missing 2.84.92.180.4313.760 Students’ ratings for Web-Database Integration Editorial Board Rating: 3.67

47. Is the mean student rating different from the rating given by the Editorial Board? TECHtionary – –Different (one-group t-value = 8.773, p=0.0001) Web-Database Integration – –Insufficient evidence (one-group t-value = 1.333, p=0.1975)

48. Conclusions Students are satisfied with the quality and quantity of learning materials they found in the IT category. Students are satisfied with the quality and quantity of learning materials they found in the IT category. Current data suggest that students rate materials differently than editorial boards. Current data suggest that students rate materials differently than editorial boards. The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards. The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards.