Assessment of Web-based Education Tools: Are your Students Learning Through Technology? Andrea Irby Undergraduate Academic Programs North Carolina State University

Background NC State is a large, public, research one institution. 19,000 undergraduates; 9,000 grads. Under pressure in late 1990s to grow Possible expansion of Distance Education Virtual Advising Center created 1999

Why Technology? Enables us to transcend many barriers Physical campus Race Country, nationality “Saving Face” Student development theory – meet them where they are Enables us to be more effective and efficient especially in times constrained resources Why Technology?

Types of Information Exchange Synchronous vs. Asynchronous Data - ways of expressing things Information - the arrangement of data into meaningful patterns Knowledge - the application and productive use of information Information versus Knowledge

Technology in Education Can transform key relationships Personalization: Targeting the student vs. the department Engaging the student: making applications interactive Student Decision Support: Information becomes knowledge Create community: cultivate student relationships Carl Jacobson, 2000

Engaging Students On-Line Web-sites On-line Course Management Systems /Listservs On line Chat or Instant Messenger type tools Bulletin Boards and Blogs Social Networking sites “Students will gravitate toward the technology medium that best meets their needs” Karen Thurmond- Univ of Memphis

Focus on Web-sites 2nd Generation Web-sites: Information based 3 rd Generation are interactive, knowledge based How do we know if/ how many students come to our website? Are they able to find the information they need? Is it clear and meaningful to them? Do the receive our intended information?

How to Know Log Data Analysis (data mining) Usability Studies Accessibility Testing

Log Data Analysis Data (log) is created each time a person visits a web- page. Analytic Tools then either pull data from the server that hosts the web-site or read it from html code on each web-page.

Sample tools Google Analytics (reads code)- free SAS Web Analytics (pulls from server) TopShareWare (pulls from server) htm

Easy to Read Reports (in real life )

Information Received Visitors/Users Repeat Visitors/Cached Visitors How long on site Traffic by time of day and month Where enter, exit, or drop out of site Browsers, operating systems Where they are logging in from Campaign/Action Tracking

Virtual Advising Center data mining 2000: 72,419 annual visitors 26,795 repeat visitors 176 avg. visitors per day 1,021,947 hits to site 2006: 285,053 annual visitors 108,320 repeat visitors 780 avg. visitors per day 8,269,215 hits annually

Virtual Advising Center data mining Monday-Thursday busiest days. Visitors on site for avg. of 2 minutes now about 5. Path in and out; pages turned. IP locations –whose accessing site on campus, off campus, around the world. Heaviest traffic times per day.

How we used this information Sent surveys out on busiest days Used busiest times of day to determine staff coverage for chat Used IP address/location information to help IT staff on campus place kiosks in high demand locations To enhance relationships with local feeder community colleges

Usability Testing Observed “Focus” Groups for web-sites Real data versus self reported data Examines how people use some human-made object Controlled experiment Small, cross section of users Trained users or “Hallway” users

Usability Testing Pre-test questionnaires get input on subject’s familiarity with web-site Actual test involves a series of short, simple tasks the subject is asked to complete Observations are recorded by “tester” Patterns of how subject completes each task are noted Emotional responses key to watch

Set up Location can be: In a computer lab with multiple teams In a usability lab At your own computer desk

Usability Testing Post tests – record overall impressions of web-site and experience; reflection Very Informative Web-site on Usability Testing: If testing uncovers difficulties, redesign Test in small groups and often For virtual advising center, its how we keep students at the forefront of our design

Accessibility Inclusive web-communities allow people with diverse abilities to be engaged in on-line learning Web-Content Accessibility Guidelines WEBCONTENT /#def-checkpoint 3 Priorities for Website Accessibility

Accessibility Series of Checkpoints Priority 1- Must satisfy these checkpoints or one or more groups will find it impossible to get and use the information Priority 2- Should satisfy these design elements Priority 3- May satisfy design elements Results in A, AA, AAA website ratings

Accessibility Test your site with on-line tools Test your site with screen readers/JAWs On-line web resources to help you: /tools.php

Evaluating Learning Web-site must be interactive tests, essays, portfolios, chats, Determine what we want students to learn Get student input Variety of ways: focus groups, check lists, rubrics and content analysis Have more than one staff member working on assessment

What we’ve tried and learned Focus Groups- what we may see as learning, they see as “help”. Over time, they may recognize what they have learned: raises question of timing of assessment Electronic communication trails are rich with evidence of learning. Students are readmitting, graduating, learning about themselves, picking majors, staying, expanding their academic portfolios.

Evaluating Learning Need to develop a rubric for evaluating rich content. George Mason Business School example Databases/Microsoft Access tool Technology is a tool – through which students can learn and for helping us with assessment.

Discussion? Questions?