1. Using Portfolio Assessment to Discover Student Learning Barbara J. D’Angelo, Ph.D. Arizona State University Association for Business Communication Annual Convention November 2009

2. Background: Written Communication for Managers Junior-level business writing course Multi-section course 7-10 sections per semester + summer On-campus, hybrid, and online Redesigned Summer 2007 Industry focus groups Business and writing faculty input Role playing scenario: students create a company Standardized syllabus and assignments Portfolio for assessment End-of-semester self-evaluation of learning Scored by 1 writing faculty and 1 business faculty Barbara J. D'Angelo, ABC Convention November 2009 2

3. Research Questions Which assignments do students choose to include in their portfolios? How do students organize their portfolios as an argument to demonstrate their learning? What persuasive techniques do students use to support their argument for learning? What tone and style do students use? Do students present an argument for their learning or to enhance their grade? Barbara J. D'Angelo, ABC Convention November 2009 3

4. Methods Random sampling 3 portfolios from each section Fall 2008 and Spring 2009 34 portfolios analyzed Simple counting of assignments Content analysis of narrative statements Categories derived from Bower (2003): rhetorical appeals and argumentation Scott (2005): form, tone, and appeal Post-semester feedback from instructors See proceedings for details (or talk to me later or email me) Barbara J. D'Angelo, ABC Convention November 2009 4

5. Assignments Included in Portfolios Barbara J. D'Angelo, ABC Convention November 2009 5 Fall 2008Spring 2009 Memo1114 Manager Roundtable1012 Proposal10 “Branding”911 Letter814 Policy910 Audience Analysis61 Agenda47 Email45 Peer Review45 Research Summary33 Minutes27

6. Assignments Assignment inclusion not related to grading scale Inclusion seems based on Substantiality of writing or traditional genres Instructor emphasis: instructor feedback indicates this as high De-emphasis of meeting documents Lack of understanding or emphasis Increased emphasis during spring = higher inclusion = awareness of importance? Process documents Reluctance to show drafts or other “in progress” work? 6 Barbara J. D'Angelo, ABC Convention November 2009

7. Portfolio Organization 26 organized generically Introduction: reflection about course Body: described artifacts, claims and evidence Conclusion: summed up 7 organized contextually Body contained more context Best vs. worst Strengths vs. weaknesses Course objectives Had no influence on raters’ scores Barbara J. D'Angelo, ABC Convention November 2009 7

8. Persuasive techniques # of Portfolios# of Statements Ethos1419 Logos2351 Pathos1729 Nomos2131 Skills Supported2364 Skills Unsupported1839 Barbara J. D'Angelo, ABC Convention November 2009 8

9. Use of Persuasion Nomos and pathos not linked to “schmooze” Not directly linked to a claim or evidence More expression of student “feelings” Awareness of audience and course values Majority (23) of students supported claims 23 used logos to create claim with evidence to support it 13 supported all claims Unlike findings with FYC students Indication of more sophisticated writing from upper level students? 8 did not support any claims 9 Barbara J. D'Angelo, ABC Convention November 2009

10. Tone and Style 24 students used positive tone Excitement about role playing (pathos) 3 students used negative tone Dis-satisfied with grade or with instructor? 7 students used intermediate tone Neutral descriptive tone All students changed from “business” style to “student” style when completing the portfolio Barbara J. D'Angelo, ABC Convention November 2009 10

11. Learning or enhance grade? Learning 28 students made statements related to their growth Only 1 student directly referred to their grade Use of logos and support for claims indicates argument was for learning Enhance grade 19 portfolios potentially included “schmooze” 2 flattered course or instructor Rest thanked instructor 2 students appealed directly to instructor (using “you”) Result differs from FYC studies Barbara J. D'Angelo, ABC Convention November 2009 11

12. Use for course improvement More emphasis on meeting documents Importance to our employer stakeholders Additional requirement to write agendas added for each Manager Roundtable Requirement to evaluate agendas Change MRT topics to more completely support written assignments Instructors believed not enough time to go over concepts needed for assignments Strengthen research summary to emphasize tie between finding information-analysis-use for persuasion 12 Barbara J. D'Angelo, ABC Convention November 2009

13. Future Potential change: portfolio scenario to shareholder or other external audience Maintain emphasis on business style Continue to de-emphasize mechanics But help needed for ESL students Continue assessment to aggregate additional data Confirmation of findings related to use of logos/evidence Add industry evaluators to assessment to include external stakeholders 13 Barbara J. D'Angelo, ABC Convention November 2009

14. Questions? Presentation available at: http://www.public.asu.edu/~bdangelo/presentations/abc09.ppt Course syllabus and assignments available at: http://techcomm.asu.edu/curriculum/twc347 Email: bdangelo@asu.edu Barbara J. D'Angelo, ABC Convention November 2009 14