1. An Evaluation Of The Effectiveness Of Digital Tool Use In Undergraduate History Classes
Scott Elliot, SEG Measurement
Gerry Bogatz, MarketingWorks
Cathy Mikulas, SEG Measurement
Paper presented at E-Learn 2016 World Conference
Washington, D.C.
November 14, 2016

2. Overview of Presentation
Background and Overview of Effectiveness Study
Results of Effectiveness Study
Instructor and Student Feedback

3. Use of Digital Tools in Higher Education
Utilized in online, blended, and face-to-face courses
Support teaching and learning
Multiple devices used 24/7
General findings support that tools can increase engagement, increase learning, and provide new models for education
Allow for reduced costs and more efficient use of instructional time
Allow for increased access to educational resources

4. Effectiveness study

5. About the Treatment: MindTap®
Online learning solution developed by Cengage Learning
Course Management System
Digital textbook
Interactive chapter assignments/problem sets
Multimedia
Assessments
Gradebook
Performance monitoring
Subject for this study: History
Use regularly for at least 10 weeks of a typical 15-week semester

6. Research Questions
Do students in history classes that use MindTap earn higher grades than students in comparable history classes that do not use MindTap?
Is MindTap differentially effective among students of different genders and ethnicities?
Is MindTap differentially effective between face-to-face or online courses?

7. Study Design Overview Quasi-experimental (depicted on next slide)
Both Treatment Group and Control Groups had access to online tools
Students were all in History courses that used the same textbook(s)
Analysis of Comparability of Pretest performance
Monitor fidelity of MindTap use throughout the semester
End of Course Grades served as outcome
Collected evaluations of MindTap from instructors and students

8. Study Design by Study Group
Treatment Group
Critical Thinking Pretest
Use of Textbook and MindTap
Student Final Course Grades
Control Group
Critical Thinking
Pretest
Use of Textbook and other online tools
-- No MindTap Use --
Student Final Course Grades

9. Profile of Participating Instructors
13 instructors teaching 21 classes in 12 institutions
Equal representation of 2- and 4-year institutions
50% suburban, 33% rural, 17% urban locations
62% male instructors, 38% female
46% more than10 years of teaching experience, 46% between 4 and 10 years of experience, remainder less than 4 years of experience
38% online courses, 52% face-to-face

10. Student Demographics Variable Treatment Control Gender Female 46% 51%
Female
46%
51%
Male
48%
Not Reported 6% 3%
Ethnicity
Caucasian
57%
60%
African American
21%
17%
Hispanic 8%
10%
Other Races or Mixed Race 7% 9% 4%

11. Use of Digital Tools Beyond MindTap
Digital Tool (Number of Instructors)
Treatment
Control
Lecture capture system 2 1
Student response system
Social networking 3
Google apps for education
Video activities or assessment 5
Web collaboration tools
YouTube
Khan Academy
Other

12. Analysis Comparison of starting ability Analysis of Covariance
Compared treatment and control groups’ grades while controlling for initial ability
Evaluated interaction effects between study group and gender
Evaluated interaction effects between study group and ethnicity
Evaluated interaction effects between study group and course type

13. Initial Comparability of Study Groups
Study Group
Critical Thinking
Test Mean
Critical Thinking Test Standard Deviation
Treatment Group
14.61
4.63
Control Group
14.42
5.06
There were no statistically significant differences in the means (F=0.126, p=.723).

14. Results
Students using MindTap outperformed their comparison classes. (F=13.302, p<.001). Effect size = .31

15. Results – By Subgroups Gender Ethnicity Course Type
Equally effective for males and females (F=2.4, p=.122)
Ethnicity
While ethnicity was not a significant factor, there was an interaction effect found for study group and ethnicity in the history courses (F=5.654; p<.001).
There is an insufficient number of students to fully compare study group differences across all of the ethnicities.
The Caucasian students who used the digital tool had higher grades than the Caucasian students who did not use the digital tool.
Course Type
Equally effective for face-to-face and online courses (F=.132, p=.717)

16. Effect Size
A common metric that can be used to evaluate the amount of growth across studies, even when different measures are used.
The Effect Size for the use of MindTap was .31 (similar to moving from 50th to 62nd percentile)

17. Instructor and Student Feedback

18. Instructors’ Evaluation of the Effectiveness of MindTap
History instructors reported that MindTap improves both teaching and learning
Rated their overall experience an A (excellent) or B (very good).
In addition, all instructors gave the program a rating of A or B for:
Meeting the needs of a diverse group of students
Making teaching easier and better
Impacting student engagement
Contributing to student learning

19. Instructors’ Evaluation of the Effectiveness of MindTap – Influencing Instruction
History instructors reported that MindTap greatly or somewhat influenced the way they taught by:
Providing useful tools and simulations to encourage deeper analysis of the issues
Helping students analyze information more completely through the variety of assignments MindTap provides
Making instructors’ work easier by providing well thought-out simulations that help to introduce and analyze issues
Giving instructors more confidence in asking students challenging questions
Giving students the foundation on which to build critical thinking
Giving students practice restructuring their knowledge, reading primary sources, considering cause and effect, and seeing the patterns of history

20. Instructors’ Evaluation of the Effectiveness of MindTap – improving students’ abilities
History instructors reported that MindTap greatly or somewhat contributed to the improvement of students’ ability to:
Analyze information and arguments
Evaluate evidence
Draw conclusions and make deductions
Separate facts from opinions, values, beliefs, and attitudes
Interpret data
Make inferences
Understand causality

21. Sampling of Instructors’ Comments
MindTap is a remarkable product. As structured, it engages the students in meaningful reading, quizzes, and real learning. With MindTap my students are learning far more. It will be a central and important part of my courses from now on.
I have seen a marked improvement in the quality of student responses to analytical questions, demonstrating an advancement of their skills as they increasingly engage with the MindTap materials.

22. Students’ Evaluation of MindTap
MindTap helps history students learn the following aspects of critical thinking (a great deal or somewhat)
% saying a great deal or somewhat
N= 263
Critical thinking skills overall
71%
Analyzing information and arguments 62
Interpreting data
Separating facts from opinions, values, beliefs, & attitudes 61
Evaluating evidence
Drawing conclusions and making deductions
Making inferences 60
Writing up information and data 59
Understanding causality 58
Understanding experimentation and the scientific process
Analyzing the credibility of sources 56

23. Summary
Students who used MindTap in history classes earned higher grades than students who used the same textbook without MindTap (effect size .31)
Faculty found MindTap to be effective and useful
Students indicated that MindTap helped them to learn important skills required for success in their history classes

24. Please contact Scott Elliot at selliot@segmeasurement.com
Questions?
Please contact Scott Elliot at