Comparing Generic Student Learning Outcomes among Fresh Graduates in the Workplace Comparing Generic Student Learning Outcomes among Fresh Graduates in the Workplace Focusing on Faculty of Medicine’s Graduates

This report includes relevant sections of a research report submitted to the Senate Committee on Teaching & Learning for all of CUHK by Hazlett, Clarke Cheng Chun Yiu, Jack Cheung Wai Hung, Gordon Kwong Kai Sun, Sunny Plus further data from a 2 nd similar survey completed only by Medicine, i.e., medical alumni from its new curriculum as the surveys were done within one yr of graduation, the medical alumni were interns Hazlett, Clarke Cheng Chun Yiu, Jack Cheung Wai Hung, Gordon Kwong Kai Sun, Sunny Plus further data from a 2 nd similar survey completed only by Medicine, i.e., medical alumni from its new curriculum as the surveys were done within one yr of graduation, the medical alumni were interns

CUHK’s Objectives Determine if recent (within 1 yr of graduation) CUHK alumni concluded that their major undergraduate programme had developed their generic skills so that they could effectively meet job requirements in their place of employment Determine if these opinions of the alumni were consistent with those of students enrolled all CUHK programmes during 1 st & last yrs of the program (as assessed by the Student Engagement Questionnaire) Hong Kong employers (as assessed by the Hong Kong Education and Manpower Bureau) Faculty of Medicine’s Added Objective Determine if alumni from old and new curriculum were different in reporting how well their programme had developed these generic skills

Methods: Summary Summary Surveyed all alumni graduating in 2004/5 (N= 2646) and then repeated the survey on only medical graduates in 2005/06 (N=142) Focus of survey instrument: During my employment since graduation, the academic programme that I took in CUHK has enabled me to... e.g.,.. speak more effectively in Cantonese when communicating with clients and colleagues.” Instrument and survey method pilot tested for content & face validity Instrument’s construct validity established (construct validity = 0.91) Generalizability of results estimated in initial survey (found no evidence that non respondents had any different opinions) Methodology details and relevant analyses available

Results

Response Rates Overall Response Rate (CUHK) in 1 st Survey:1356 (51%) Fac. of Medicine Response Rate in 1 st Survey: 78 (54%) Fac. of Medicine Response Rate in 2 nd Survey: 73 (51%)

Results of validation studies were consistent with (and somewhat better than) the findings when the survey instrument was first pilot tested. Instrument met (exceeded) acceptable standards for construct validity - an oblique factor structure with nine correlated factors was readily interpretable - 9 factors corresponded to the a prior defined constructs, i.e., those which SEQ & EMB surveys were assumed to have in common - solution met criteria for a simple factor structure - nine factors represented over 53% reduction from the number of items used within the instrument but still accounted for > 84% of the total variance (approximating a construct validity coefficient of.91) Construct Validity Estimate

Comparisons among those who only responded after 1 to 3 follow up reminders and with those needing no reminders were used to estimate if the collected data were possibly representative - i.e., to establish if it was clearly inadvisable to generalize the findings to those who had not responded to the survey at anytime - no differences in opinions detected among those who responded to 1 st contact & those who were contacted a 2 nd or 3 rd time - altho’ no evidence to indicate that one can’t generalize the results to non respondents, it remains possible that those who did not answer were systematically different Generalizability

Faculty of Medicine’s Within Program Comparisons With reference to CUHK’s average for all respondents (i.e., includes those who did and did not require the skill in their job)

CUHK & Nursing Alumni Compared Level their Programmes Developed Generic Competencies

CUHK & Pharmacy Alumni Compared Level their Programmes Developed Generic Competencies

CUHK & Medical Alumni Compared Level their Programme Developed Generic Competencies

* * statistically significant at 0.05 (2-tailed) between old & new curriculum cohorts CUHK & Medical Alumni Compared Level their Programmes Developed Generic Competencies

Summary Expected differences among CUHK Faculties & programmes were often found, adding evidence to the validity level of the measured outcomes These results imply that caution is needed when attempting to make any direct comparisons among Faculties or among major programmes To evaluate the relative differences, note that on a scale from 1 to 5, the absolute differences among Faculties & progammes were not large and thus few differences are educationally meaningful

Comparative Findings for Faculties & Programmes Variations between programmes were more pronounced than among Faculties caution should be exercised as smaller programmes with fewer respondents can more easily exhibit wider variations than large programmes with larger number of respondents

Conclusions Recent alumni from 3 programs in Faculty of Medicine are quite similar to how other CUHK alumni report that their respective programmes’ have developed the graduates’ generic competencies Medical programme’s alumni who graduated under the new curriculum reported one statistically significant improved competency (use of computers) & that the other 8 generic competencies were directionally better than parallel measures reported by alumni from the old curriculum

Summary Although reported results are based on a survey instrument which met acceptable standards for construct validity, there were only reasonable consistencies between CUHK’s SEQ and this Fresh Graduates Survey’s measures for the following constructs a. Language proficiency b. Computer literacy c. Problem Solving Ability Thus, comparative alumni ratings are likely more accurately informative for these 3 above noted competencies

most constructs measured in this survey are generic to all programmes; thus, most should be included in the student learning outcomes of any major programme or Faculty these generic skills are those that both employers and the Educational Manpower Bureau regard as relevant skills for any university graduate entering the Hong Kong workforce these skills are those which CUHK has operationally defined as appropriate for its students to acquire while they pursue a chosen discipline-based or professional-based education the investigators hope that this feedback from new alumni will prove to be useful for the Faculty in its continuing efforts to enhance its respective curricula, instructional designs, learning activities and student assessments in order to achieve desired student learning outcomes Notwithstanding the Afore Noted Cautions