1. Written Communication & Critical Thinking Capabilities of College of Business Undergraduates

2. Introduction
Are undergraduates in the Belk College of Business developing the written communication skills & critical thinking skills we want to see students graduate UNCC with?
Evidence-based evaluation using multiple data sources:
Direct sources:
Embedded assessments in various core courses
ETS Heighten exams
Indirect sources:
Senior Survey
NSSE
Will show Heighten data and NSSE data today.

3. ETS HEIghten Written Communication Exam
HEIghten Written communication exam is 2 parts:
Part 1: Construct a persuasive response adopting a stance on a particular claim
Part 2: Two passages presented, multiple choice questions on each
- scored via proprietary algorithm
Scores presented a point scale.

4. Participants
Undergrads from COB courses sat for HEIghten WC exam in Spring 2016.
Excluded participants >30 yrs, GPA < 1.0, Current Hrs = 0
Freshmen (n = 115)
69 male
18 transfer students
GPA: M: 3.02 SD: 0.60
43 Dean’s or Chancellor’s List
Age: M: 19.4 SD: 0.81
Majors: Pre-Business Admin (89),
Pre-Account (18),
Pre-Econ (7)
Seniors (n = 98)
52 male
0 transfer students
GPA: M: 3.09 SD: 0.36
27 Dean’s or Chancellor’s List
Age: M: 22.6 SD: 1.4
Majors: Marketing (51),
Management (43),
other COB (4)
Note test-takers are concentrated in certain majors.

5. Sample Institutional Score Report from ETS
UNCC COB Seniors
The histograms show the distribution of individual students’ scaled scores within the Reporting Group (UNCC) and the Comparison Group (All students, all institutions).
No info is provided about which institutions are in the Comparison group, and there is no ability yet to self-select a peer comparison group.
2,162 students who have taken exam at 27 institutions

6. Evidence from HEIghten Written Comm. Exam
Seniors score significantly higher than freshmen. *
Bar graphs are means, error bars represent the standard error around the mean). Data are based on our in-house analyses using disaggregated student score data.
* p < 0.05

7. ETS HEIghten Critical Thinking Exam
Critical thinking exam is a test of logical & deductive reasoning
Provides students with lists of evidence and arguments
Asks questions about whether pieces of evidence support a conclusion or not, and how strongly
Asks students to identify underlying assumptions in an argument
Logic word problems
Test scope: addresses evaluation, interpretation, & inference skills
Multiple choice, scored via an algorithm
Scale is 150 to 180.

8. Participants
Undergrads from COB courses sat for HEIghten CT exam in Spring 2016.
Excluded participants >30 yrs, GPA < 1.0, Current Hrs = 0
Freshmen (n = 103)
65 male
26 transfer students
GPA: M: 3.03 SD: 0.57
36 Dean’s or Chancellor’s List
Age: M: 19.8 SD: 1.4
Majors: Pre-Business Admin (81),
Pre-Account (15),
Pre-Econ (5)
Seniors (n = 104)
51 male
1 transfer student
GPA: M: 3.2 SD: 0.37
37 Dean’s or Chancellor’s List
Age: M: 22.9 SD: 1.5
Majors: Marketing (59),
Management (26),
other COB (20)

9. Evidence from HEIghten Critical Thinking Exam
Seniors score numerically, but not significantly, higher than freshmen.
(Bar graphs are means, error bars represent the standard error around the mean).

10. ETS HEIghten Exams Limitations:
Small sample sizes for both UNCC and the national data limit the strength of conclusions about this exam.
Exams are narrow in types of abilities tested, and may not reflect what skills/abilities are emphasized in particular disciplines.

11. Evidence from NSSE
Triangulating students’ abilities: examining multiple measures
Many COB programs emphasize analytical skills, including quantitative reasoning, mathematical, and statistical skills.
Evidence for strengths in these area. * *
***Animation on this slide to reveal QR graphs***
For this study, we examined multiple measures. Where different measures pointed to the same conclusion, that adds to our confidence in that conclusion. The converse is also true. One of the measures analyzed was the National Survey of Student Engagement (NSSE). There were several items that pointed in a similar direction as the Heighten exam data. For instance, compared to UNCC students from other colleges (gray bar), students in COB report their coursework places less emphasis on inference skills. For instance, ‘How much coursework emphasized forming a new idea or understanding from various pieces of information.’
However, the NSSE also pointed out a major area of strength for COB students. Many COB programs emphasize analytical and quantitative reasoning skills. These are areas the Heighten Critical Thinking exam does not cover. NSSE data revealed COB students reported frequently engaging in quantitative reasoning skills in their coursework. For instance, ‘How frequently your courses asked you to reach conclusions based on your own analysis of numerical information. (NSSE likert scale: very often, often, sometimes, never)’
NSSE likert scale:
4 = very much
3 = quite a bit
2 = some
1 = very little
n=169
n=1138
* p < 0.05

12. Conclusions
Drawing conclusions on the basis of multiple measures can mitigate limitations inherent in one measure or methodology.
Importance of intentional design when using standardized exams.
Considering discipline-specific contexts.
Triangulation method helps overcome limitations inherent in embedded assessments, survey/indirect measures, and standardized exams.
Topic-specific standardized assessments currently all fall short in area of peer comparison groups. Important to have a strong design that allows for internal comparison to be made as well, such as freshmen/ senior comparison shown in this study.
It is also important to consider discipline-specific contexts. For instance, writing in COB may look different than writing in other disciplines. Some disciplines may emphasize different critical thinking skills to a greater or lesser degree.

14. Defining Effective Written Communication
Strong, effective writing may demonstrate many of these skills:
Knowledge of rhetorical conventions in different contexts
Awareness of audience & purpose of writing
Content development & organization
Correct/appropriate spelling, grammar, tone, style
Knowledge of the writing process (drafting, revising)
rhetorical conventions in different contexts: knowledge of genre and discipline conventions
Content development & organization: understanding how to incorporate source material, develop ideas using examples & evidence in a logical, compelling manner

15. Evidence from Embedded Assessments
Core Curriculum Learning Goal 2: Students will demonstrate the ability to communicate effectively.
- Subgoal on written communication
Students have excelled at previous assessments of writing:
COMM 3160 cover letter assignment, 91% of students (n=318) at target.
Assessment plan now updated to ‘Persuasive Analytical Report’ writing assignment
- this is also an EXCELLENT critical thinking assignment
Core curriculum assessed every other year, majors on opposite years.

16. Evidence from Senior Survey & NSSE* **
N=488
N=2051
N=73
N=471
** p <
* p < 0.05
Senior survey likert scale: 4 = very much, 3 = somewhat, 2 = very little, 1 = not at all
Data are from Seniors only.
non-COB: UNCC students from colleges other than COB

17. Written Communication in Belk College of Business
Students’ writing meets current targets (embedded assessments)
Evidence students are gaining in their writing abilities & knowledge of
writing process (ETS HEIghten exam)
COB seniors self-perceive less gains in writing (Senior Survey) and are
writing fewer pages in undergrad (NSSE) than other UNCC seniors
However, business communications may be shorter in length by design compared to papers in other disciplines.
Faculty are best judge of whether or not there are sufficient writing opportunities throughout the curriculum.

18. Defining Critical Thinking
Critical thinking may encompass some or all of the following steps:
A process of inquiry and hypothesis-generation
Gathering information & data of good quality
Evaluation of information’s credibility, validity, logical strength (etc.)
Analysis of quantitative and qualitative information & data
Interpretation of information’s significance & meaning
Drawing inferences from evidence, assessing alternatives, determining sufficiency of evidence, decision-making
Communication of thought-process and conclusions to others
Operational definition of the skills critical thinking entails.

19. Evidence from Embedded Assessments
Core Curriculum Learning Goal 4: Students will demonstrate the ability to use analytical skills in decision making.
- 6 subgoals
Assessed via problem sets & multiple choice questions in ACCT 2121, ACCT 2122, FIN 3120, OPER 3100
: 5 out of 6 subgoals at or above target
: 4 out of 6 subgoals at or above target
Core curriculum assessed every other year, majors on opposite years.

20. HEIghten Critical Thinking Scores and GPA
Similar correlation pattern for writing scores and GPA

21. Evidence from HEIghten Critical Thinking Exam
National data courtesy of Javarro Russel at ETS, personal communication.
National data provided via personal communication.

22. Evidence from Senior Survey
All students give fairly high ratings to item, ‘To what extent do you think your college education contributed to your knowledge, skills, and personal development in using critical thinking skills?’
COB seniors (green bar) rate this item lower than seniors in the other colleges (non-COB, gray bar). **
To what extent do you think your college education contributed to your knowledge, skills, and personal development in each of the following areas
Using critical thinking skills
Applying analytic skills
4 = very much 3= somewhat, 2 = very little, 1 = not all, 0 = don’t know
Green bars show the mean for COB students, gray non-COB students. While all students tend to endorse these items at pretty high rates, COB students do rate these items lower than students from other colleges.
Senior Survey
likert scale:
4 = very much
3 = somewhat
2 = very little
1 = not at all
N=485
N=2048
** p <

23. Senior Survey & HEIghten data
Seniors’ HEIghten CT scores positively correlate with their responses on these Senior Survey questions:
Overall satisfaction with instructors in my major departments’:
ability to motivate me to do my best
how quickly they provide feedback on my work
effectiveness in using instructional technology
Suggests effective teaching practices may foster development of students’ critical thinking skills.
Senior Survey likert satisfaction scale:
5 = very satisfied, 4 = satisfied, 3 = neither, 2 = dissatisfied, 1 = very dissatisfied

24. Evidence from NSSE
Compared to other UNCC students not in COB (n=1138, gray bars), COB students (n=169, green bars) self-report their courses place more emphasis on memorization and less emphasis on analyzing and forming new ideas. * * *
Memorizing course material. (College report says they are memorizing more than business majors at other institutions too; significant difference holds if just look at seniors as well)
Analyzing an idea, experience, or line of reasoning in depth by examining its parts.
Forming a new idea or understanding from various pieces of information.
NSSE likert scale:
4 = very much
3 = quite a bit
2 = some
1 = very little
* p < 0.05

25. Critical Thinking in Belk College of Business
Seniors show numerically, but not statistically, higher scores than freshmen on test of evaluation, interpretation, & inference skills (ETS Heighten).
Evidence students are gaining in ability & confidence with quantitative reasoning and quantitative analysis skills (NSSE, embedded assessments)
Good instructional practices may benefit students’ development of critical thinking skills (ETS & Senior Survey)