1. First Two Years Project Bringing Theory to Practice
Cathy Crosby-Currie
Christine Zimmerman
Bringing Theory to Practice
March 2007

2. Modeling the Multiple Influences on Civic Development and Well-Being

3. Astin’s Theory of Involvement: I-E-O Model
Research designs reflect underlying conceptual models of the relationship between the variables

4. Terenzini’s General Conceptual Model of College Influence on Student Learning
Terenzini, P., Springer, L., Pascarella, E., Nora, A.(1995). Influences Affecting the Development of Students’ Critical Thinking Skills. Research in Higher Education, 36,

5. Methodological Considerations

6. Experimental v. Quasi-Experimental Design
Key difference between experimental and quasi-experimental
Researcher’s control over the “input” variable
Experimental: YES! -> cause/effect conclusions
Quasi-Experimental: NO! -> examine relationships
Control v. comparison groups

7. Experimental v. Quasi-Experimental Design
Quasi-experimental power comes from:
Ability to detect change through design
e.g., interrupted time series design
Equivalence of comparison group to experimental group

8. Longitudinal v. Cross-Sectional Designs
Comparing groups of different ages at one point in time
Convenient but lacks statistical and conceptual power
Longitudinal:
Comparing individuals to themselves across time
Multiple cohorts is ideal

9. St. Lawrence’s Quasi-Experimental, Longitudinal Design
Participants:
Two Cohorts – Selected Students from Classes ‘09 and ’10
Experimental Group – students in Brown College
Second year added a second experimental group
Comparison Group – non-equivalent group matched on key variables of interest
Comparison Group Sample II.xls

10. St. Lawrence’s Quasi-Experimental, Longitudinal Design
Data Collection
Pretest (9/05 & 9/06)
Posttest (2/06 & 2/07)
Follow-up (4/07 & 4/08)

11. Challenges of Quasi-Experimental and/or Longitudinal Designs
Creating comparison group(s)
Participant attrition
Communication incl. letter from president
Personalized letters &
Contacting students multiple times/multiple ways
Accommodate students’ schedules
Institutional Review Board Approval
Reframe as a positive contribution to your research not a hurdle to overcome

12. Challenges of Measurement
Valid and Reliable Measures
Direct - Indirect Measures
Quantitative - Qualitative Data
Process - Outcomes Measures

13. Reliability … is the consistency or repeatability of responses
Random error (noise)
Systematic error (bias)

14. Reliability (cont.) Ways to increase data reliability:
Clear directions
Clear questions
Consistent order of questions
Clear survey layout
Trained proctors/interviewers
Consistent data entry and scoring

15. Reliability (cont.) How to assess the reliability of your instrument:
Pilot-test your study
Test-retest your survey
Focus-group survey or interview questions
Include similar questions in same questionnaire

16. Validity
… the extent to which the instrument truthfully measures what we want to measure
How well does the instrument content match what we want to measure?
Do respondents interpret the questions correctly?
Do respondents’ answers reflect what they think?
Are the inferences we make from this study accurate? Can they be generalized?

17. Validity (cont.) How to establish validity:
Use multiple measures and multiple methods
Derive measures from literature review & existing research / participate in national survey instruments and tests
Expert review
Pilot-test your own survey

18. Direct – Indirect Measures
Direct: tangible, actual evidence
Indirect: proxy for what we try to measure
Direct Measures
Indirect Measures
Portfolio
Essay/reflection
Performance task/test
Actual student behavior
Self-reported behavior, attitudes, gains
Grades
Participation rates
Time spent at task

19. Qualitative – Quantitative Measures
Qualitative: unit of data = words
Quantitative: unit of data = numbers
Qualitative Measures
Quantitative Measures
Focus groups
Structured interviews
Self-reflections/diaries
Open-ended survey questions
Surveys with closed questions (Likert scale, check list, etc.)
Grades
Actuary data such as participation rates, attendance, etc.

20. Process – Outcomes Measures
Process Measures
What did we do? (=data to demonstrate the implementation of an activity/program)
Outcomes Measures
What are the results? (= data used to measure the achievement of an objective/goal)
Initial
Intermediate
Long term

21. Administrative Challenges And Best Practices
Buy-in and Support
Form campus partnerships early on
Build on existing data collections
Institutional survey cycles and survey timing
Copyrights of survey instruments
Liability for use of certain measures
Survey recruitment & retention

22. Select Survey Instruments and Literature

23. Sampling of Survey Instruments
Entering Student Survey
CIRP Freshman Survey (HERI, UCLA)
College Students Expectations Questionnaire CSXQ (Indiana)
Enrolled Undergraduate Students/Alumni
As a continuation of CIRP: Your First College Year/College Senior Survey
As a continuation of CSXQ: College Student Experience Questionnaire (CSEQ)
National Survey of Student Engagement (NSSE)
Consortia Senior and Alumni Surveys (e.g. HEDS. COFHE)

24. Sampling of Survey Instruments
Depression/Mental Health Measures
Beck’s Depression Inventory (BDI II)
Brief Symptoms Inventory (BSI)
Optimism/Pessimism/Happiness Scales
Mehrabian Optimism/Pessimism Scale
Alcohol/Drugs/General Wellness
CORE Alcohol And Other Drugs Survey
ACHA-NCHA

25. Sampling of Survey Instruments
Civic Development (from Lynn Swaner)
CASA TELEPHONE SURVEY INSTRUMENT
Other National Surveys
HERI Faculty Survey
Other In-House Institutional Surveys
Course evaluations
Program evaluations
Satisfaction studies

26. Select Literature
The National Center on Addiction and Substance Abuse at Columbia University (2003): Depression, Substance Abuse, and College Student Engagement: A Review of the Literature. Report to The Charles Engelhard Foundation and The Bringing Theory to Practice Planning Group.
The National Center on Addiction and Substance Abuse at Columbia University (2005): Substance Abuse, Mental Health and Engaged Learning: Summary of Findings from CASA’s Focus Groups and National Survey. Report to Sally Engelhard Pingree and The Charles Engelhard Foundation for the Bringing Theory to Practice Project, in partnership with the Association of American Colleges and Universities.
Swaner, L.E. (2005). Linking Engaged Learning, Student Mental Health and Well-being, and Civic Development: A Review of the Literature. Prepared for BTtoP

27. Select Literature
Pascarella, E., Terenzini, P.(1991). How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass.
Bringle, R. G., Phillips, M.A., Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, D.C. American Psychological Association
Suskie, L. (1996). Questionnaire Survey Research: What works. Tallahassee: Association for Institutional Research