1. The Vision of How to Assess Student Learning in Technology-Enriched Classrooms
Stan North Martin, Joni Spurlin, Dianne Raubenheimer, Brad Mehlenbacher, and Deena Murphy-Medley
NC State University 
© NC State University, 2006.

2. Session Outcomes
By the end of the session, participants will be able to
Consider different approaches to assessment of the interrelationships of technology, teaching and learning.
Review and revise their own assessment processes.
Acquire resources to develop assessment and research practices on their campus.

3. Session Overview
ClassTech and the Learning in a Technology-Rich Environment (LITRE) Quality Enhancement Plan
Assessment plan & results from 2004 –2005
Adding pedagogy and theory to assessment
Vision of Assessment
Framework
Assessment plan overview
Details of Student Learning Assessment including interview process and observations
Questions from audience

4. ClassTech’s Roots (1)
Pedagogical needs, supported by technology, have emerged
Requests for technology percolating up to deans, dept. heads
$525M in new capital projects will increase number of classrooms by 20%
NC State’s quality enhancement plan for Learning in a Technology Rich Environment (LITRE)
Convergence of several things taking place on campus….
Faculty are incorporating new styles of teaching: inquiry guided learning, increasing use of laptops and PDAs.
Even many of those using traditional lecture-style methods are working to engage students. They’re asking for data projectors, instructional computers, document cameras, VCR/DVD players.
A $468 million bond initiative and other resources has caused the university to step back and evaluate classroom needs and to develop a set of classroom standards. These standards are being re-evaluated again as we continue to build and renovate new learning spaces.
The University’s reaffirmation of accreditation process called for a quality enhancement plan. The concept of improving our already strong technology infrastructure to further improve teaching and learning was adopted and a plan came together during an 18 month intensive campus process.
A faculty survey conducted as part of the LITRE plan development reinforced the need for classroom improvement. When faculty were asked what factors would make it easier to use technologies in their courses, the most common response was that technologies needed to be available and supported in their classrooms.

5. ClassTech’s Roots (2) Implement basic technology Support the users
Maintain the equipment
Assess the outcomes
Equipment usefulness, functionality, training, support
Pre-semester Open House training sessions
Meet-n-greet sessions during first several days of classes
- Met more than 80% of instructors within first two days of classes
Online and on-site materials
Telephones in most rooms
Weekly room checks
Availability of support staff 8 – 8 p.m.

6. Meanwhile...
Preparing NC State’s Quality Enhancement Plan: Learning in a Technology Rich Environment (LITRE)
Scholarly inquiry focused on enhancing the technology-rich learning environment
Investigative process through which new approaches to student learning, using technology, are proposed, vetted, empirically evaluated, and if the evaluation results indicate, deployed and routinely assessed
Evidence would be collected and analyzed to inform future projects
The assessment of ClassTech became a first-round project under the LITRE umbrella.

7. 2004-05 Assessment Questions
Assess satisfaction of students, faculty and technical staff with the use of this technology in academic settings.
Identify technical issues involved in using technology in the classroom, for students, faculty, and technical staff.
Evaluate the impact of using technology on faculty workload, pedagogy, attitudes, and amount of material delivered.
Assess faculty and student perceptions of the degree to which technology, used inside and outside the classroom, has impacted learning.

8. ClassTech Assessment Methods
Online faculty survey of those teaching in ClassTech rooms
Several in-depth interviews of faculty
Quick in room surveys of students and faculty
Interview of ClassTech technical support staff
Log of equipment usage from a subset of ClassTech rooms
UPA surveys of sophomores and graduating seniors and the annual ResNet user survey
The response rate for spring 2005 was 47% (n=116) compared to 28% (n=35) in fall 2004
Nine interviews conducted in spring 2005 primarily by Dr. Karen St. Clair of Faculty Center for Teaching and Learning
Quick surveys carried out in 10 classes, generally in the classes of faculty who were interviewed
Actual equipment usage data gathered automatically via some in-room control systems
University Planning & Analysis and ResNet regularly conducts surveys and several new questions were added to address classroom technology

9. ClassTech Results –
Impact of technology on workload, pedagogy, attitudes, amount of material delivered....
Overall favorable perceptions of technology use on faculty workloads
No significant evidence of direct impact on pedagogy or assessment methods
Almost half (48%) of faculty respondents felt pace was faster with technology
Half said technology allowed for wider variety of topics
Most (61%) said they were able to cover material in more depth
Faculty survey results: 47% response rate *of those teaching in ClassTech rooms*--limited group of faculty, most of whom specifically requested the teaching space with technology.
Initially took more time to create material, but then more useful, easier to update, re-purpose
Most faculty using newer technology but same teaching methods. Some exceptions exists.
When comparing this class with same or similar class in room without technology....
Pace: Pretty much the other half (49%) said pace was the same, with less than 3% saying it was slower.
Variety: Split between more variety and the same
Depth: one respondent said less depth

10. ClassTech Results –
Faculty & student perceptions of how technology use inside/outside the classroom impacts learning....
Most faculty surveyed or interviewed had difficulty directly addressing the impact of technology on learning outcomes
In-class student surveys indicated 81% felt it positively affected their learning
UPA student surveys generally positive or no difference on how technology impacts learning
Responses to open ended-questions attempting to elicit technology’s impact on student learning generally resulted in greater elaboration of what they did with the technology.
Those that did elaborate indicated it :
enabled students to more easily follow class discussion,
Assisted students in visualizing and understanding course concepts
Improved participation in class
Students generally feel positive about use of technology in the classroom, though multiple comments in the ResNet survey about frustration with faculty not able to use the technology well. (E.g. Death by PowerPoint, tinkering with the equipment to get it to work, etc.)

11. ClassTech Results –
Faculty & student perceptions of how technology use inside/outside the classroom impacts learning....
Most faculty felt students were more involved in learning process when using technology compared to teaching without it
Almost 60% felt students were more involved...

12. ClassTech Results –
Satisfaction of faculty, students & technical staff with use of technology in academic setting....
Faculty surveyed—only those using ClassTech rooms, so biased—overwhelmingly appreciated and made extensive use of the tools: 76% said having networked computer was essential (51%) or important (25%) in teaching their class
Most students surveyed in classes prefer moderate (71%) or extensive (21%) use of instructional technology
Most students agreed (61%) or strongly agreed (13%) that the instructional technology accommodated their needs and learning styles.
See full results
High correlation among those faculty’s self-reported use of equipment and saying having the equipment was important for their class teaching.
There is a general trend in students’ appreciation for faculty using technology in the classroom, to accommodate their learning styles, or for the visual stimulation. It’s becoming an expectation, but definitely cautions exist about poor use of technology.
More extensive results and graphs are online.

13. ClassTech Results – Difficulty of Assessing Impact on Learning
Surveys and interviews tried to find out how and why faculty used technology in their class
Attempted to determine if any impact on course objectives, desired learning outcomes, methodology
However,
Faculty were able to talk much more about what they did with the technology and how it impacted the way they taught or used their time
Much more difficult to get at how it impacted student learning
E.g.
“What, specifically, do you want students to know, do or feel (e.g., attitudes) when teaching with the technology (student learning outcomes)? “
“How do you measure the learning that you expect to occur when you teach with the technology in the classroom (student learning outcomes)?”
“Did any of the methods to measure the learning, incorporate the technology in the classroom (clickers, displaying questions on document camera)? “
Through the survey and interview process we found it very difficult to talk to faculty and get meaningful data on “how technology impacts student learning”

14. Therefore….
Beginning Fall 2005, added members to assessment committee who understood more about pedagogy and theory
Assessment Committee:
Stan North Martin, Associate Director, Computing Services
Joni Spurlin, University Director of Assessment
Brad Mehlenbacher, Associate Professor, Education
Dianne Raubenheimer, Director of Assessment, College of Engineering
Deena Medley-Murphy, Graduate Student

15. Instruction and Learning with Technology: Classroom Dimensions
Activities/Attributes
Learner Background & Knowledge
Biological (age, gender, race), Abilities (cognitive, physical), Literacies (computer, domain, textual, visual), Socioeconomic (income, geographic, organizational), Personal (learning style, attitude, motivation, self-monitoring)
Learner Tasks & Activities
Represent task + set information goal, Navigate to related topics, Scan information, Understand information, reading to learn, to do, to analyze, to compare, confirm, correct, submitting to computer (ie., goal, intention, action, interpretation, evaluation)
Social Dynamics
Instructor presence, Cognitive + social learner identity, Boundaries, Frequency + duration of communications, Responsiveness, Group management and self-assessment
Instructor Activities
Content, set objectives, information exchange requirements, topic pacing, sequencing, adaptation to audience, methods of evaluation, strategies for topic elaboration
Learning Environment & Tools
Selection of instructional materials, e.g., reading and writing tools, individual and shared documents, viewing and dissemination methods, atmosphere that promotes mentoring and open exchange of ideas and discussion; ergonomically designed for optimal usability

16. Instruction and Learning with Technology: Terminology
Stressing problem-based goals and authenticity in projects and learning activities. Adaptation to audience, communication of content, generating objectives, drawing on prior knowledge, information exchange requirements, strategies for topic elaboration, topic pacing and flow, sequencing, methods of evaluation (Savery & Duffy, 1995, 1996).
Learning (individual and distributed cognition)
Comprehension (selection, working memory, cognitive workload)
Integration with existing knowledge structures (strategies for potential storage in long-term memory, information mapping, organization, schemata)
Development of new connections between the new information and existing state of understanding (associative meaning, mental models), and
Elaboration toward a richer understading of the subject matter in question, leading to expert understanding and/or behaviors (Kirsh, 2005; Simon, 1979).
Technology
“Like a textbook or any other cultural object, technology resources for education … function in a social environment, mediated by learning conversations….” (Bransford et al., 2000, p. 26).
Learning Environments are efficient (in terms of resources, task support, and time), ergonomically effective (have the capacity for producing desired results), economical (in terms of time and resources for learners and instructors), educational (in facilitating and promoting learning), and equitable (equally usable to all learners) (Mehlenbacher, et al., 2000, 2002, 2005).

17. Framework based on literature Assessment Plan for 2005-2006
Vision of Assessment
Framework based on literature
Assessment Plan for

18. Draft Framework (see handout)
Learning Environment
Examples:
Instructional practices inside learning spaces:
Instructor or learner centered
Instructor behavior/presence
Learner Engagement
Tasks and activities (in and out of learning spaces)
Learning techniques
Learning environment and tools
Source of information
Effective use of learning spaces
Adaptive for differences in learners
Instructional practices outside of class-time
Feedback/communication with learner
Assessment
WHAT and HOW WELL do students learn?
Learning outcomes (learning comprehension to elaboration)
Student work quality
Outcomes added, modified based on use of technology
Evaluation of environment/ technology
Satisfaction
Costs v. benefits (time/resources)
Persistence in discipline/ degree
Technology
Functionality and support
Usability
Reliability
Amount used
Learning space characteristics
Class size
Usability of space
Laboratory, classroom, etc
Instructor characteristics
Literacy with instructional technologies
Educational philosophy
Management style
Learner background/ characteristics
Knowledge/skills/literacies
Attributes (demographics)
Attitudes
Learning style
Functional use of technology
Data access, analyses
Communication
Document Preparation
Construction

19. Modified ClassTech Assessment 2005-2006 (see handout)
How does use of technology impact course’s:
pedagogy
faculty workload
faculty attitudes
amount of material delivered
How does having the technology in the classroom affect:
use of class time and assigned coursework
how students learn (following LITRE defined outcomes)
student achievement of course and program objectives
Are students, faculty and technical staff satisfied with the use of this technology in academic settings?
What are the challenges of using technology in the classroom for students, faculty, and technical staff?

20. Assessment Methods
ClassTech Assessment Team to define model that incorporates
technology
pedagogy
learners
environment
outcomes
Survey faculty who use rooms (More in-depth in spring)
Automatic tracking of usage in some of the rooms
Interviews with faculty and classroom observations in 20 courses
Sample of student work in observed classes to assess student learning
Student surveys in selected classrooms
Focus group with technical staff
Review of problem call tracking logs
Model developed in the fall based on literature and research of the team members.
Longer faculty survey each spring, include questions related to the dimensions of our assessment model.
Built an application that captures equipment usage data from one of the control systems and matches it to the room schedule.
Significantly modified this year: --This spring using a purposeful sampling technique to determine which faculty to interview and classes to observe. Plan to gather student work samples to measure learning.
Call tracking logs can help determine type, quantity and severity of maintenance and support problems being encountered.

21. Details on One Method Classroom Observations and Interviews
Spring 2006

22. Assessment Spring 2006
Developed research design, interview protocol, and observation instrument – fall 2005
This is a case study design with cross-case analysis
Purposeful sample of 20 faculty using ClassTech classrooms
Video-taped an instructor teaching in a ClassTech room
Trained 4 research observers to use the interview protocol and observation instrument
Each research observer watched the video, scored the classroom using the observation instrument
Calculated inter-rater reliability for 4 observers
Made adjustments to observation instrument base on discussion amongst observers
Re-watched and re-scored video and re-calculated the inter-rater reliability

23. Interview Protocol Questions focus on
Identifying learning outcomes
Role of technology in classroom
Relationship between technology and anticipated student learning outcomes
Faculty perceptions of the value of technology
Evidence of students achieving the learning outcomes
Interview data will be analyzed for emerging themes, with cross-case analysis.

24. Observation Instrument
The observation instrument is a fairly complex rubric containing items in different categories, including
Learning direction
Instructor behavior
Lesson implementation
Instructor activity based on SOLO taxonomy
Technology use by instructor
Student learning techniques
Student activities based on SOLO taxonomy
Technology used by students during class
Learning environment and classroom characteristics
Observers rate frequency of occurrence
Data analyzed for trends in various categories

25. SOLO taxonomy (Biggs, 2003) SOLO category Representation
Type of outcome
Unanticipated extension
Create Synthesise
Hypothesise Validate
Predict Debate
Theorise
Logically related answer
Apply Outline
Distinguish Analyse
Classify Contrast
Summarise Categorise
Multiple points
Explain Define
List Solve
Describe Interpret
Single point
State Recognise
Recall Quote
Note Name

26. Student Artifacts
Artifacts providing evidence of student learning outcomes include
Homework
Test items
Classroom assessment techniques
Student work is graded
Tasks and artifacts will be rated by the research team for SOLO taxonomy level
Data will be triangulated with observation and interview data

27. Current status Interviews conducted with most of the 20 instructors
About half of the observations have been done.
Student work starting to be collected.

28. Questions?

29. Resources
Session documents & other resources:
Resources on assessment of technology related to student learning:
Quality Enhancement Plan for Learning in a Technology-Rich Environment at NC State:
LITRE Goals and Assessment Plan:
2003 LITRE Faculty Survey Report:
Classroom NC State:

30. Contact Information
Stan North Martin
Associate Director, Computing Services
Information Technology Division
Joni E. Spurlin, Ph.D.
University Director of Assessment
University Planning and Analysis
C. Dianne Raubenheimer, Ph.D.
Director of Assessment
College of Engineering
Brad Mehlenbacher, Ph.D.
Associate Professor
Education