Presentation is loading. Please wait.

Presentation is loading. Please wait.

California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO Educational Benchmarking (EBI) June 25, 2004 Fundamentals.

Similar presentations


Presentation on theme: "California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO Educational Benchmarking (EBI) June 25, 2004 Fundamentals."— Presentation transcript:

1 California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO Educational Benchmarking (EBI) June 25, 2004 Fundamentals of Structuring and Implementing Successful First Year Seminars

2 The statistical analysis for this presentation was based on the Educational Benchmarking (EBI) First Year Initiative (FYI) Assessment and conducted by Randy L. Swing, Ph.D. Co-Director, Policy Center on the First Year of College & Fellow, The National Resource Center on The First-Year Experience and Students in Transition in conjunction with Educational Benchmarking

3 Value of a Collaborative Group Shared mission and purpose Common problems and barriers Varied expertise Benchmarks for performance

4

5 Direct Quote: “The elements that research indicates are the most important to have as components of an effective first year program.”

6 Presentation Overview The elements that research indicates are the most important to have as components of an effective first year program. Principles for leveraging assessment to initiate and sustain improvement

7 FYS – Goals Retention 37% Academic Achievement 37% Participation in College Life 11% The Policy Center on the First College Year Ease transition to college Increase “student skills” Increase persistence rates Increase graduation rates

8 First-Year Initiative Survey FYI Developed by the Policy Center on the First College Year and Educational Benchmarking (EBI) 62 institutions (limited to 4-year institutions*) Over 30,000 students 7- point scale Learning Outcome Factors Administered in the last week of fall 2001 *4-year or 2-year regional campuses - a 2-year version is in development.

9 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

10 FYI Assessment and Research Context The following principles and practices provide the foundation to the research and the First Year Initiative Assessment (FYI)

11 Research/Assessment Context 1.Factors: A more powerful indicator than individual question results. 2. Regression: Allows you to identify predictors of performance 3.Predictor Factors: Factors, if improved, have the greatest impact on improving overall effectiveness

12 Factors Factors (also called “constructs”) are groupings of related questions that share a relationship. The basic assumption of factor analysis is that underlying dimensions, or factors, can be used to explain more complex phenomena.

13 Factor Means Course Learning Outcomes mean Course Improved Knowledge of Campus Services Course Improved Knowledge of Campus Policies Course Improved Connections with Peers Course Improved Connections with Faculty Course Improved Managing Time/Priorities Course Improved Critical Thinking Course Improved Study Strategies Course Improved Knowledge of Wellness Issues Course Improved Academic/Cognitive Skills Course Increased Out-of-Class Engagement /23/41/4

14 Factor Reliability Once it has been determined that a certain set of questions do share a relationship and therefore constitute a factor, there is an additional (and necessary) statistical test to assess the psychometric soundness of the factor.

15 A Cronbach’s Alpha of zero would mean that there is no internal consistency at all, i.e., subjects are likely to respond with anything from 1 to 7 on any of the questions in a factor with no discernable pattern. An Alpha of 1 would mean that every subject answered every question comprising the factor consistently (e.g., all subjects answered with all 7’s, or all 1’s). This is a highly unlikely event. Factor Reliability

16 FYI Factor Reliabilities Factor DescriptionsReliability Course Improved Study Strategies0.90 Course Improved Academic and Cognitive Skills0.89 Course Improved Critical Thinking0.91 Course Improved Connections with Faculty0.84 Course Improved Connections with Peers0.90 Course Increased Out-of-Class Engagement0.90 Course Improved Knowledge of Campus Policies0.90 Course Improved Knowledge of Academic Services0.87 Course Improved Managing Time and Priorities0.92 Course Improved Knowledge of Wellness0.90 Sense of Belonging and Acceptance0.90 Usefulness of Course Readings0.90 Satisfaction with College/University0.90 Course Included Engaging Pedagogy0.92 Overall Course Effectiveness0.92

17 Regression A correlation establishes the relationship between two variables. Regression analysis, by contrast, allows us to determine the relationship between some dependent variable (e.g. overall effectiveness) and multiple independent variables (e.g., engaging pedagogy, etc.).

18 Major predictors identify, in order of descending importance, the factors that have the greatest impact on overall effectiveness. While minor predictors are statistically significant, individually and combined they add little to the predictability of overall effectiveness. Regression

19 Predictors Therefore, the return on the investment for improving the major predictors would far outweigh the return on improving the minor predictors.

20 Horizontal Cross Bar set at Mean = 5.50 (75% satisfaction level) Vertical Cross Bar sets the line between Major and Minor Predictors of Overall Satisfaction Top Priority High Impact, Low Performance Monitor Low Impact, Low Performance Maintain Low Impact, High Performance Maintain or Improve High Impact, High Performance Impact: Predictors of Overall Satisfaction Performance 4 Quadrants Priority Matrix Executive Summary

21 Top Priority High Impact, Low Performance Monitor Low Impact, Low Performance Maintain Low Impact, High Performance Maintain or Improve High Impact, High Performance Impact: Predictors of Overall Satisfaction Performance Priority Matrix Executive Summary Engaging Pedagogy #1 Predictor of Course Effectiveness

22 Engaging Pedagogy Factor To what degree did the course include: –A variety of teaching methods –Meaningful class discussions –Challenging assignments –Productive use of classroom time –Encouragement to speak in class –Encouragement for students to work together –Meaningful homework

23 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

24 Four Types of Academic Seminars 1.COLLEGE TRANSITION THEME orientation/study skills/managing transitions 2.SPECIAL ACADEMIC THEME interdisciplinary/problem-focused/selected topic 3.DISCIPLINE BASED introduction to major/department/discipline 4.REMEDIAL/STUDY SKILLS study skills for a high risk population

25 Transition- Theme 75% What is a First-Year Seminar? A course offered specially for new students to assist with the transition into college. Four Types of Seminars Remedial 2% Discipline- based 10% Special Academic 13% 58% Cal

26 College Transition Theme courses deal directly with orientation to college and academic skills. This is the University 101 (USC) model. Special Academic Theme courses deal with a selected topic other than college transition. These are often taught as interdisciplinary seminars where a small group of students and a model learner/teacher use a variety of methods to investigate an important theme. Discipline based are often an introduction to a major or department. They are based in individual academic departments. Remedial/Study Skills, was considered, but there were too few cases to draw any meaningful conclusions.

27 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Theme Format: Percent of students with mean 5.50 or greater Special TransitionAcademic Discipline

28 FYI Finding Transition-theme and Special Academic- theme courses were about equal on learning outcomes and student satisfaction. Discipline-theme courses produced lower learning outcomes and student satisfaction. Remedial courses – too few in the study to draw valid conclusions.

29 Why? Engaging Pedagogy explains the difference. Transition 30.5% Special Academic36.5% Discipline 18.3%

30 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

31 Credit Hours 2001 Pilot Administration % of institutions (N=62) No credit 4 or more credits 1 credit 2 credits 3 credits Cal 21% 16% 32% 0%

32 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Percent of students mean 5.50 or greater Contact Hour(s) 1 Hr 2 Hrs 3 Hrs

33 1 contact hour courses  Orientation to Services 2 contact hour courses  Study Strategies  Peer Connections  Faculty Connections 3 contact hour courses  Academic Skills  Critical Thinking most effective with learning outcomes associated with a basic orientation to campus services. perform well at producing effective study strategies, peer connections, and faculty connections significantly impact academic skills (reading, writing, oral presentation skills) and critical thinking skills

34 FYI Finding The number of contact hours should match the intended goals of a first- year seminar.

35 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

36 Required/ Not Required 2001 Pilot Administration % of institutions (N = 62) CalFYI 42%37% No students in any section required to enroll 32%11% Some students in some sections required to enroll 16%35% All/most students in every section required to enroll 10%17% Mixed formats – no one format constitutes 80% 42% 16% 10% 32% Cal

37 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Required Not Required (Elective) Percent of students mean 5.50 or greater

38 Required Not Required Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Factor Means for Required/not required Controlled for Contact Hrs, Theme-types, and Grade-formats

39 Required Not Required Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Factor Means for Required/not required Controlled for student characteristics (gender, race/ethnic, HS Grades, and commuter/residential)

40 FYI Finding Courses that are NOT REQUIRED... produced greater outcomes than required courses. WHY? Engaging Pedagogy Required Not Required 4.48* 4.71* * after controlling for differences in grading and contact hours

41 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

42 Grading 2001 Pilot Administration % of institutions (N = 62) 82% Letter Graded 16% Pass/Fail 2% Mixed Cal 47%

43 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Graded Pass/Fail Mean Scores * controlling for Contact Hrs, Required, UGTAs, & Themes

44 FYI Finding Overall, grading format produces mixed results, even when controlled for Contact Hours, Theme-types, Required, and UGTAs. Key Finding: Graded courses are associated with higher scores on Engaging Pedagogy

45 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

46 Linked Courses 2001 Pilot Administration % of institutions (N = 62) Few or no sections linked Some Linked Most Linked 16%43% 42% Cal

47 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy Linked Not Linked Mean Scores controlling for Required, Grading, Contact, & Theme

48 FYI Finding Linking the seminar to other courses produced greater learning outcomes for: Academic Skills Study Skills Critical Thinking Engaging Pedagogy Limitations in this study reduce the capability to make definitive statements about the impact of linking courses.

49 Issues Addresses by FYI Findings Course Theme Credit/Contact Hours Required/ Not Required Letter Graded/Pass-Fail Linked Course Format Undergraduate Teaching Assistants

50 Use of Undergraduate Teaching Assistants Less than 80% of sections have UTAs Cal 58% 21% 16% No sections have UTAs More than 80% of sections have UTAs

51 Study Strategies Academic Skills Critical Thinking Faculty Connections Peer Connections Out-of-Class Policies/Procedures Campus Services Time/Priorities Wellness/Spirituality Belonging Course Satisfaction Engaging Pedagogy UGTA No UGTA Mean Scores controlling for Required, Grading, Contact, & Theme

52 FYI Finding Undergraduate Teaching Assistants are associated with higher mean learning outcomes - except for academic skills, critical thinking skills, and engaging pedagogy..... even after controlling for: Required/Not Grading format Theme Contact Hours

53 Summary: Engaging Pedagogy best predicts the learning outcomes and student satisfaction with the seminar

54 Student rating of Engaging Pedagogy: No significant difference by gender African-Americans, Latino, and Native Americans gave higher ratings than Whites and Asians “A” high school students gave lower ratings

55 Right Reason Help staff calibrate performance Identify where to focus effort Identify where training is needed Provide motivation to improve

56 Use assessment for Continuous Improvement, not Evaluation Don’t punish staff for past performance they are not capable of changing. Join with staff to use information to improve performance Right Reason

57 Right Method Credible Confidential Comparative Comprehensive Continuous Right Reason

58 Right Info reallocate budgets focus resources on a limited number of initiatives with the greatest probability of producing improvement Results should be of research quality, but analyzed and presented to provide decision- makers with the information they need to: Right Method Right Reason

59 Right Info Right People Put the information directly in the hands of the people in the best position to initiate and implement change Put information into the hands of the people responsible for results Right Method Right Reason

60 Right InfoRight Time Right People Immediacy of feedback brings relevancy and impact to results Provide results at a time staff can evaluate performance, focus on impact factors to create innovative initiatives to improve. The shortest assessment cycle leads to the most rapid improvement Right Method Right Reason

61 Right Info Repeatedly Right Time Right People Essential to establish expectation in staff they will be able to determine the impact of their efforts to improve through future assessments Assessment is a management tool not an isolated event Continuous assessment is paramount to continuous improvement Right Method Right Reason


Download ppt "California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO Educational Benchmarking (EBI) June 25, 2004 Fundamentals."

Similar presentations


Ads by Google