Multi-State Collaborative For Learning Outcomes Assessment (MSC) An initiative sponsored by the Association of American Colleges and Universities and the State Higher Education Executive Officers Pilot Year Aggregate Results NOTE: These tables represent aggregate results of a pilot study involving samples of student work from 29 2-year and 24 4-year institutions in 9 states. Work products were assessed by faculty members using common rubrics developed as part of AAC&U’s VALUE project. See for more information and rubrics used. Because this was a pilot study, the results are generalizable for all students in each participating state or nationally. The pilot sample included work products from 2,642 students sampled from 53 institutions in 9 states.

MSC Pilot by the Numbers 53 institutions uploaded artifacts – Some consortia uploading – 52 public, 1 private Institutions in all 9 states of MSC represented By sector: – 24 four-year – 29 two-year By Carnegie Type: – Associates = 31 – Masters = 15 – Research = 7 2

MSC Pilot by the Numbers 7,215 pieces of student work were submitted – Students had to be 75% of the way to completion – 2,642 artifacts scored twice (36.6%) – 5.2% marked unscoreable More than 1,100 assignments submitted More than 100 scorers Approximately 5% were marked unscorable 3

Reporting To Come: On-line reporting interface This week: – Cover Sheet – Institutional level raw data in Excel – Aggregate results – Tool Kit – SPSS Syntax 4

Taskstream Demo What is to come... 5

This Week Cover Sheet Aggregate results in power point Institutional data file – Before analyzing institutional data – Structure of institutional data file Analyzing Institutional Data – Research Questions – Excel – SPSS 6

MSC Aggregate Tables Overview of the project (see also cover sheet) Data show MSC Sample is relatively representative of MSC Participants, – Slightly over representative of hispanic and traditional age students – Not representative of nation or states and may or may not be similar to your institutions sample Breakout of assignment subject and artifacts by learning outcome 7

Things to Check About your Data Is the sample representative of the population from which it was drawn? – Demographic characteristics (race/ethnicity, gender, age, Pell status, etc) – Student characteristics (student major, degree level, Full- time/Part time status etc.) the sample at your institution similar to the project sample? (using similar breakouts above) What type of assignments were submitted? What is the distribution of the data? – Is it normally distributed around a mean? – Are there outliers? 8

Questions you may consider asking How did certain artifacts score on the dimensions for each learning outcome? How did certain assignments on the dimensions for each learning outcome? How did faculty intent match scorer perceptions? How did students as a whole perform on the dimensions for each learning outcome? How do students at my institution compare to all students involved in the project? Is there a difference between how different groups performed? (independent sample t-tests) 9

Institutional Data File Structure Assignment Data Data Included in the Assignment Coversheet: There are multiple Artifacts for each assignment. Artifact Data Data included in the Artifact Upload File: There are multiple rows for each score. Score Data Scores by dimension submitted by scorers: There is only one score per rubric dimension

Excel Example

SPSS Example