STUDENT SATISFACTION WITH WEB-BASED EDUCATIONAL RESOURCES FOR COMPUTER & INFORMATION SCIENCE: Teaching with Merlot Collection Emrah Orhun, Ph.D. Troy State University Montgomery Alabama, USA

Outline Introduction Introduction Learning and Teaching Materials in The Merlot Collection Learning and Teaching Materials in The Merlot Collection Students’ Satisfaction with Merlot Materials in CIS Classes Students’ Satisfaction with Merlot Materials in CIS Classes Conclusions Conclusions

Introduction Difficulties in locating relevant and affordable materials of high quality Difficulties in locating relevant and affordable materials of high quality –World Lecture Hall at the University of Texas »83 categories to browse and search utilities to locate materials –WebCT (2002) »1,250 publisher course content packs in more than 50 subject areas »customizable materials around which to build courses combined with the functionality of WebCT's course management software.

Quality issue World Lecture Hall materials World Lecture Hall materials –courses offered at accredited colleges and universities around the world WebCT packs WebCT packs –editorial standards of the publisher and author of the text

MERLOT (Multimedia Educational Resource for Learning and Online Teaching) Web-based open resource designed primarily for faculty and students in higher education Web-based open resource designed primarily for faculty and students in higher education emphasizes the quality issue emphasizes the quality issue

MERLOT repository MERLOT repository –Over 8,000 free learning materials and support materials that can be integrated within the context of a larger course

Searching and Browsing in MERLOT

Figure 1. Entries in the Applications sub/subcategory of Information Technology subcategory under Science and Technology category

Figure 2. Search results for TECHtionary

Support resources –sample assignments –evaluations of the learning materials »discipline panel peer reviews »individual user reviews

Peer Review Process Discipline communities Discipline communities Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages. Biology, business, chemistry, engineering, health sciences, history, information technology, mathematics, music, physics, psychology, teacher education, and world languages.

Discipline teams Discipline teams (“MERLOT Editorial Boards”) –selected by the MERLOT partner organizations in accordance with the MERLOT guidelines. –develop evaluation standards, –conduct peer reviews of learning materials within the discipline, and –make connections with their professional organizations

Peer review process Evaluated by at least two team members who have relevant expertise. Evaluated by at least two team members who have relevant expertise. Review process stages Review process stages –triage review to identify worthy candidates –review of candidates –composite review report sent back to the authors for feedback and permission to post the review on MERLOT. –approved review is posted to the MERLOT Web site by the discipline leader

The evaluation criteria: The evaluation criteria: –quality of content, –potential effectiveness as a teaching tool, –and ease of use.

Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed Each dimension evaluated separately (on a scale of 1-5 stars, with 5 being the highest), and an overall rating is computed A review must average three stars in order to be posted to the site. A review must average three stars in order to be posted to the site.

Figure 3(a). Peer Review of TECHtionary (Part a)

Figure 3(b). Peer Review of TECHtionary (Part b)

Figure 3(c). Peer Review of TECHtionary (Part c)

Figure 4. Member Comments for TECHtionary

Current state of MERLOT

Table 1 Growth of MERLOT materials, members, and peer reviews (May 28, 2002) TotalAdded during last 365 days Added during last 30 days Materials Members Peer Reviews

Table 1 Growth of MERLOT materials, members, and peer reviews ( February 27, 2003 ) TotalAdded during last 365 days Added during last 30 days Materials Members Peer Reviews Member Comments Assignments322591

Table 1 Growth of MERLOT materials, members, and peer reviews for IT Category TotalAdded during last 365 days Added during last 30 days Materials Members Peer Reviews 3630 Member Comments Assignments1190

May 25, 2002 September 26, 2002 February 27, 2003 Materials % Peer Reviews % Member Comments % Assignments Ratio of Peer Reviews, Member Comments, and Assignments to Materials

Problem Percentage of reviewed materials low Percentage of reviewed materials low Potential Solution Peer review by community of learners Peer review by community of learners Cause Editorial peer review process too slow Editorial peer review process too slow

STUDENTS’ SATISFACTION WITH MERLOT MATERIALS IN CIS CLASSES

Goal of Study To find out to what extent MERLOT provides relevant and quality materials for the CIS courses. To find out to what extent MERLOT provides relevant and quality materials for the CIS courses.

Questions Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? Is the quality and quantity of learning materials in the MERLOT collection sufficient to satisfy CIS students’ knowledge/learning needs in specific courses? Is the MERLOT collection ‘usable’ by the students? Is the MERLOT collection ‘usable’ by the students? Does the MERLOT collection induce the students to become members of learning communities? Does the MERLOT collection induce the students to become members of learning communities? Can peer review by students substitute for peer review by editorial boards? Can peer review by students substitute for peer review by editorial boards?

Questions re-formulated How do students rate their satisfaction with MERLOT collection? How do students rate their satisfaction with MERLOT collection? How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members? How do students rate the learning materials as reviewers? Do students’ ratings of the learning materials differ from the ratings given by Discipline Panel members?

Teaching with the MERLOT Collection in CIS Courses Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: Summer-Fall 2002 semesters in the following CIS classes at Troy State University Montgomery: –CIS 5547 CARApplied Systems Analysis –CIS 5547 CBRApplied Systems Analysis –CIS 3332 Software Engineering I –CIS 3357 Logical Structures of Computer Design Overall, 41 students participated in the evaluation. A total of 252 evaluations were made. Overall, 41 students participated in the evaluation. A total of 252 evaluations were made.

Evaluation of web-based learning materials 1.1. Visit MERLOT ( site and register as a member For each chapter covered in the course: 3.a. Find a relevant learning material in the IT subcategory. 4.b. Review the material using the guidelines provided by MERLOT. In the table provided below, state what learning outcomes are supported by the material and post your review as a Member Comment on MERLOT. 5.c. Rate your satisfaction with MERLOT collection on a scale 1-5, with 5 implying that you were fully satisfied with the quality and quantity of learning materials you found in connection with your task, and 1 implying that you were not satisfied at all. 6.d. Find a relevant online learning material not contained in MERLOT, post it to MERLOT and add a Member Comment. Student Assignment

ChapterTopicTitleURL Learning outcome Satisfaction with MERLOT collection for this task 5= very good 1=very poor 1 Players in the System Game Enterprise Resource Planning Supersite supersite.com supersite.com Provides analysis on ERP trends 3 2 Information System Building Blocks Interface Design for Educational Multimedia edu.rachel/PD F/intdesmm.pd f edu.rachel/PD F/intdesmm.pd f Describes the principle of interface design 4 3 Information System Development Risk analysis and management for Projects (RAMP) mprisk.com/fro ntindex.htm mprisk.com/fro ntindex.htm Comprehensive outline of RAMP 5 Example of Relevant MERLOT Material

ChapterTopicTitleURL Learning outcome 1 Players in the System Game New responsibilities Fall Heavy on IT Workers nweek.com nweek.com Talks about the new job responsibilities of current and upcoming IT employees. 2 Information System Building Blocks TP Middleware keeping up with Mainframe Revival m m Explains the functionality of the product TP middleware on mainframe systems. 3 Information System Development Tips for a successful Software Distribution Project. orld.com orld.com Shares other companies success stories of in relation to Software Distribution Project. Example of New MERLOT material posted

How do students rate their satisfaction with MERLOT collection?

COURSE Numbe r of Studen ts Numbe r of Topics Numbe r of Evaluat ions Averag esdev CIS3357 Logical Structures of Computer Design CIS5547 (A+B) Applied Systems Analysis CIS3332 Software Engineering I Students’ mean satisfaction ratings for MERLOT collection

ChCh Topic N Averag esdevMinMax Percent rated 4-5 1Binary Numbers & Codes Digital Circuits Combinational Circuits Sequential Logic Systems Registers & Counters Memory & Programmable Logic Register Transfer & Computer Operations CIS3357 Students’ satisfaction with MERLOT collection

ChCh Topic N Averag esdevMinMax Percent rated 4-5 1Process Project Management Requirements Analysis I Requirements Analysis II Software Architecture Detailed Design CIS3332 Students’ satisfaction with MERLOT collection

ChCh Topic N Avera gesdevMinMax Percent rated 4-5 1Players in the System Game Information System Building Blocks Information System Development Project Management Systems Analysis Requirements Discovery Application Architecture and Modeling CIS5547 Students’ satisfaction with MERLOT collection

AObject-Oriented Analysis and Modeling BObject-Oriented Design and Modeling Output Design and Prototyping Input Design and Prototyping User Interface Design Systems Construction and Implementation Systems Operations and Support CIS5547 Students’ satisfaction with MERLOT collection (contd.)

How different are students’ evaluations from Editorial Peer Reviews?

Type************ Tutorial File System Design for an NFS File Server Appliance File System Design for an NFS File Server Appliance Cascading Style Sheets, Level1 Cascading Style Sheets, Level1 Electronic Commerce Tutorial Electronic Commerce Tutorial FTP Tutorial FTP Tutorial Learning Microsoft Word 2000 Learning Microsoft Word 2000 Web-Database Integration Web-Database Integration Mulder's Cascading Style Sheets Tutorial Mulder's Cascading Style Sheets Tutorial Sharky's Netscape Frames Tutorial Sharky's Netscape Frames Tutorial Tje ABCs of Building a Web Site Tje ABCs of Building a Web Site Data Communication s Data Communication s ADSL Computer Knowledge Virus Tutorial Computer Knowledge Virus Tutorial Reference Material Study Guides and Strategies Study Guides and Strategies Tips for Using Blackboard Tips for Using Blackboard Animation TECHtionary Warriors of the net Warriors of the net Distribution of materials in IT

Material Type ************ Tutorial ADSL Reference Material Web-Database Integration Web-Database Integration Animation TECHtionary Distribution of materials according to type and rating (excluding materials with no user comments with 3 categories) Sample of Materials from IT Category

Sample of Students CIS3332 SOFTWARE ENGINEERING (16) CIS3332 SOFTWARE ENGINEERING (16) CIS5547 SYSTEMS ANALYSIS (27) CIS5547 SYSTEMS ANALYSIS (27)

Case Editorial Board Rating Member Rating Students’ Rating (mean) TECHtionary 3.0*****3.75 Web-Database Integration Web-Database Integration 3.67***3.82 Students’ Ratings

Mean Std. Dev. Std. Error VarianceCoef.Var.Count MinimumMaximumRangeSum Sum Squared # Missing Students’ ratings for TECHtionary Editorial Board Rating: 3.00

Mean Std. Dev. Std. Error Variance Coef. Var. Count MinimumMaximumRangeSum Sum Squared # Missing Students’ ratings for Web-Database Integration Editorial Board Rating: 3.67

Is the mean student rating different from the rating given by the Editorial Board? TECHtionary – –Different (one-group t-value = 8.773, p=0.0001) Web-Database Integration – –Insufficient evidence (one-group t-value = 1.333, p=0.1975)

Conclusions Students are satisfied with the quality and quantity of learning materials they found in the IT category. Students are satisfied with the quality and quantity of learning materials they found in the IT category. Current data suggest that students rate materials differently than editorial boards. Current data suggest that students rate materials differently than editorial boards. The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards. The present limited study does not provide evidence for the notion that peer review by students can substitute for peer review by editorial boards.