USING THE CLA TO INFORM CAMPUS PLANNING Anne L. Hafner Campus WASC Faculty Coordinator Winter 2008

BACKGROUND  Collegiate Learning Assessment ( CLA) created by Council on Aid to Education, non- profit in NY  Performance assessment of critical thinking, analytic reasoning and written communication (see example)  Assesses “value added” by a college  Value added: standardized gain score that controls for students’ academic abilities(and for selectivity and SES)

CLA MEASURES I  Analytic writing task: Make an argument: “in our time, specialists of all kinds are highly overrated. We need more generalists - people who ca provide broad perspectives.”  Critique an argument

CLA Measures - Performance Task  Provide students with a real world scenario. Students have 90 minutes to advise the mayor on crime reduction strategies & evaluate potential policies: a.Invest in a drug treatment program b. Put more police on the street  Students provided w document library including memos, statistics, data tables, news articles, etc.

Performance Task  Students are required to use an integrated set of critical thinking, analytical reasoning, problem solving, and written communication skills

DETAILS I  Common models: cross sectional (freshman and senior cohorts) or longitudinal (follow same kids)  Students’ SAT scores are collected to enable comparison of student’s “expected” CLA score vs. student’s actual CLA score

DETAILS II  Both freshmen & seniors are tested. After seniors were tested, a gain score from freshman to senior class can be estimated. This informs the campus how much “value” was added by attending CSULA  Because sample of 100 drawn may not be representative of the class, CAE makes an adjustment so that a school’s “actual” CLA score is compared to its “expected” CLA score (based on students’ SATs)

INTERPRETING DIFFERENCES  Differences between actual and expected scores are reported in 2 ways: by points on CLA scale and by standard errors in terms of 5 performance levels (well below expected, below expected, at expected, above expected, and well above expected) At expected is between -1 and +1 standard errors from expected.

CSULA Fall 06 Administration  Fall students tested  Problems with getting students to participate, paid incentives  One half of students took performance task, one half took writing task  CAE sent report back with some interpretation, also sent data set

ANALYSIS: HOW DID OUR FRESHMEN SCORE ?  After taking into account their abilities (SAT), our freshmen scored 1.4 s.e. higher than expected overall or above expected.  Students performed at expected on performance task and above expected on writing tasks.  CLA sample scored higher than freshman population on SAT and ELM, same on EPT

How CSULA Freshmen Did  Our college performed in the middle of the group of all schools nationwide that participated in CLA in 2006 (5th -6th decile)

Correlations  With dataset, correlations were run  CLA scale scores moderately to highly intercorrelated  Scale scores correlated moderately with EPT, SAT, HSGPA  Overall CLA performance level did not have any positive correlations other than with CLA scale scores

CONCLUSIONS I  CLA sample not equivalent to 2006 freshman class (CAE: not a problem)  CLA sample’s demographics similar to campus demographics  CSULA sample scored higher than expected on tasks  Overall, CLA performance level was not highly correlated with other tests

CONCLUSIONS II  Students with English as primary language do not score significantly higher than non- English speakers on CLA  CLA appears to be measuring something different from knowledge & skills measured by the SAT, EPT, ELM and grades

FINDINGS Summer 2007  In summer 07, we received our senior findings and “gain” score  Seniors scored “at expected”  Gain was 55 pts (less than expected) Campus is at 4th decile (between 30th-40th percentile nationwide)  CAE can examine if improvement in average student performance between entry and graduation is in line with similar gains of comparable students at other colleges

FINDINGS II  Seniors did less well than freshmen on performance tasks, and on critique an argument tasks.  CSULA’s 6 year graduation rate was “at expected’ 32%

MAJOR PROBLEMS  Staff time to recruit students and administer the test  High cost of enrollment incentives  Time required to take test (90 minutes)  Cost to take the test is inexpensive ($20)

STRENGTHS  Value added is major focus, rather than absolute performance  CLA is well developed and assesses two key skills: critical thinking/reasoning, and writing skills  Strong psychometric properties, valid and reliable, standardized gain scores enable comparisons  Can work with small sample

CRITICISMS  Issue of junior transfers not dealt with in models; different population may need to be also assessed  Uncertainty about validity (consequential) and reliability  Heavy reliance on SAT score as measure of academic ability

THE FUTURE  It may be preferable to embed CLA tasks into a freshman required course such as Intro to College or orientation  Senior level: could be embedded in capstone courses or made graduation requirement  CLA could be a useful measure of some GE outcomes

THE FUTURE II  Pressure on universities from US DOE for accountability led to Voluntary System of Accountability (VSA)  2007: CSU Chancellor mandated that all campuses use CLA and report annually for the VSA  CAE is working to create performance standards for CLA (e.g “Proficient” level); could be useful to colleges  CAE is working with CSU to deal with issue of junior transfers