Dr. Stephen Landry, CIO Dr. Eric Fountain, IT Assessment Coordinator Danielle Mirliss, Instructional Designer Using External Data Sources to Increase Confidence in Assessment Conclusions

Who We Are … Seton Hall University Mid sized, Catholic affiliated university in suburban NJ, 15 miles from New York City. 4,400 Undergraduates, 50% residential. Two campuses: Main Campus in South Orange, NJ and Law School in Newark, NJ. Approx. 350 Full Time Faculty on the South Orange Campus. Research Intensive (Research II) Institution.

Institutional SHU SHU made a major commitment to institutional change in its 1996 Strategic Plan Seton Hall became a ubiquitous computing campus, providing laptop computers to the learning community as part of tuition and fees Seton Hall increased enrollment while increasing academic requirements for admission Seton Hall added new programs and faculty Seton Hall increased emphasis on assessment and accountability

Ubiquitous SHU Seton Hall University’s Mobile Computing Program is designed to ensure all members of the learning community have access to learning resources which are increasingly online. Program has three interdependent components: Access: The University licenses students the use of a laptop as part of their tuition and fees; Curricular Integration: The University provides support and incentives to faculty to use technology in T&L; Services and Support: Network (Wired and Wireless), Help Desk, Laptop Repairs, Insurance, Training, etc.

Implementing Ubiquitous Computing 1994 – 1996: Development of SHU Strategic Plan 1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School 1997: Mobile Computing Pilot Project expands to 300 incoming Freshman 1998: Mobile Computing Program launched for all incoming Freshman 2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented 2002: SHU graduates first mobile class 2004: Tablet PC Pilot Project

Mobile Computing and Curriculum Development Seton Hall recognized that access to technology is necessary for the changes envisioned in the learning environment, but not a sufficient condition. Teaching, Learning and Technology Center Provides support and training for faculty in their use of technology in their teaching Internal Grant Programs: Curriculum Development Initiative Faculty Innovation Grants Implementation of Classroom and Network Technology Teaching, Learning, and Technology Roundtable

Mobile Computing and Assessment In 1997 Seton Hall recognized it had a unique assessment opportunity Mobile Computing Assessment Program has two objectives: To assess the impact of ubiquitous computing on the learning environment To provide prompt feedback to the Mobile Computing Program planning team to address potential implementation issues with the program

Assessment Methodology and Resources

Foundations of the Assessment Movement Research Traditions Contributing to the Assessment Movement: Student Learning in College Retention and Student Behavior Evaluation and Scientific Management Mastery Learning

A B C D A.Reflect B.Plan C.Act D.Observe Assessment Process Cycle

Types of Assessment Focus GroupsePortfoliosSurveys Faculty Fellows Tracking Systems Performance Evaluations Skill Demonstration

National Studies Studies Focused on Students: National Survey of Student Engagement ( Higher Education Research Institute ( CIRP Freshman Survey: profile of entering freshman class. CSS College Student Survey: follow up of CIRP cohorts. YFCY Your First College Year Flashlight ( Studies Focused on Faculty: Faculty Survey of Student Engagement (

National Studies Studies Focused on Technology: EDUCAUSE Core Data Survey ( Campus Computing Project ( Cost of Supporting Technology Services (

Assessment in the Teaching, Learning and Technology Center

Current Initiatives Mobile Computing Assessment Team Project Assessment TLTR-Assessment Committee

Mobile Computing Assessment Team Cross-Campus, Cross-Discipline Provost’s Office Institutional Research Faculty TLTC Multiple Projects Mobile Computing Assessment Survey Extant Data Analysis Large-Course Redesign Classroom Study Multi-Campus

Mobile Computing Survey R1. What is the effect of the Mobile Computing Program on student satisfaction? R2. What is the effect of the Mobile Computing Program on technology use? R3. What is the effect of the Mobile Computing Program on desired educational outcomes? Research Questions

Results Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University, Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University, Making good use of the technology available to all students, at least in terms of certain types of technology use, and. Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment. The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally:

Web-based Online Courses Mixed mode Hybrid Courses Enhanced Courses with electronic resources and exercises Course Level Assessment Strategies

Collaborative effort between Provost’s Office, faculty, and TLTC Student and Faculty focus groups Development and implementation of pilot survey Hybrid Course Assessment

Tablet PC Highlights

MCAT – Future Plans Expand Faculty Participation Expand Membership to Include Student Affairs and Student Government Organize and Publicize Results Approach Question of Classroom Use FlashLight Comparison Studies

Using External Data Sources to Increase Confidence in Assessment Conclusions An Example from Seton Hall University using results from the 2003 National Survey of Student Engagement

What NSSE Measures Student exposure to and participation in educational practices that promote engagement and student learning Interaction with faculty in & out of the classroom Active & collaborative learning High expectations and academic challenge Enriching educational experiences Student perception of a supportive campus environment helping them to succeed

NSSE Questions - Active & Collaborative Learning

NSSE Active & Collaborative Learning Seton Hall Freshmen vs Peers & National

NSSE Questions - Faculty Student Interaction

NSSE 2003 – Faculty Student Interaction Seton Hall Freshmen vs Peers & National

NSSE Technology Qs Technology Use SHU StudentsSHU Carnegie Peers Time online for academic - 6+ hours / week 55% 2003 SHU FR40% Doct-Intensives FR 51% 2003 SHU SR44% Doct-Intensives SR Time online for any reason - 6+ hours / week 88% 2003 SHU FR77% Doct-Intensives FR 81% 2003 SHU SR72% Doct-Intensives SR

NSSE Technology Qs Technology Use & Active Learning SHU StudentsSHU Carnegie Peers How often required to use IT - % Very Often 59% 2003 SHU FR49% Doct-Intensives FR 61% 2003 SHU SR56% Doct-Intensives SR Used T to make presentations - % Very Often 47% 2003 SHU FR34% Doct-Intensives FR 56% 2003 SHU SR53% Doct-Intensives SR

NSSE Technology Qs Technology Use & Faculty Contact SHU StudentsSHU Carnegie Peers ask instructor to clarify assign - % Very Often 51% 2003 SHU FR30% Doct-Intensives FR 57% 2003 SHU SR39% Doct-Intensives SR to express ideas to instructor - % Very Often 22% 2003 SHU FR13% Doct-Intensives FR 28% 2003 SHU SR16% Doct-Intensives SR

NSSE Technology Qs Technology Use & Active/Collaborative Learning SHU StudentsSHU Carnegie Peers Work in teams in class using IT - % Very Often 27 % 2003 SHU FR14 % Doct-Intensives FR 25 % 2003 SHU SR19 % Doct-Intensives SR Communicate with classmates on-line to complete academic work - % Very Often 51% 2003 SHU FR27 % Doct-Intensives FR 49 % 2003 SHU SR39 % Doct-Intensives SR

NSSE Technology Qs Technology Use & Active Learning / Reflection SHU StudentsSHU Carnegie Peers Gain new insights into course material from on-line discussions - % Quite a bit+ 43% 2003 SHU FR37% Doct-Intensives FR 31% 2003 SHU SR32% Doct-Intensives SR Express ideas to instructor by not comfortable saying in class - % Very Often 22% 2003 SHU FR13% Doct-Intensives FR 28% 2003 SHU SR16% Doct-Intensives SR

Conclusions

Questions? Contacts: Dr. Stephen Landry, CIO – Dr. Eric Fountain, IT Assessment – Danielle Mirliss, Instructional Designer –

Resources Internet Resources for Higher Education Outcomes Assessment AAHE Assessment Forum - Assessment in Higher Education -