Multimodal Methodologies Part I and II Jeff Bezemer, Carey Jewitt and Gemma Moss MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS One

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS An understanding of the origins and scope of multimodality Explications of multimodal frameworks and key concepts Knowledge of the different ways in which multimodality has been taken up by a range of perspectives An outline of the procedures of multimodal analysis The aim of today is to provide:

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Why take a multimodal approach? What concepts and methods can be used to analyse interaction in digital environments? How can we get at how the use of different digital technologies shapes understandings and experiences? What challenges does multimodality raise for researching digital data and environments? Focusing questions:

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Part I 9.15 Introduction 9.30 Origins, Scope and Key concepts (Carey Jewitt) Multimodal perspectives: video data analysis (Jeff Bezemer) lunch Part II Presentation and data workshop: Analyzing multimodal digital texts (Gemma Moss) Program:

Multimodality: origins, scope and key concepts MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

6

7

Concepts: the level of mode MODE MEANING FUNCTIONS SEMIOTIC RESOURCE SEMIOTIC (MEANING) POTENTIAL MATERIALITY AFFORDANCE PROVENANCE MODAL LOGICS FUNCTIONAL SPECIALISM MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Concepts: Level of modal-interaction MULTIMODAL ENSEMBEL INTER-MODAL RELATIONS COHERENCE DISSONANCE MODAL DENSITY MODAL INTENSITY EQUAL-UNEQUAL DOMINANT – SUBORINATE ELABORATION EXPANSION REPETITION-EMPHASIS SALIENCE December 2011, Norway

Concepts: level of process of sign making SIGN MAKER INTEREST MULTIMODAL ORCHESTRATION MULTIMODAL DESIGN TRANSFORMATION TRANSDUCTION TRANSMODAL MOMENT TEXTUAL CYCLE MODAL ORIENTATION CONTEXT December 2011, Norway

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Social Semiotic Multimodal Analysis A Systemic Functional Approach (Multimodal Discourse Analysis) Multimodal Interactional Analysis Multimodal conversation analysis Four main perspectives on multimodality:

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS fine grained analysis: the need to understand the details of texts and interactions focus on meaning making as social and semiotic the demand to broaden conceptions of communication beyond language shared interests, and shared histories externally material focus Similarities

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Historical origins, influences and directions Extent of emphasis on context Degree of emphasis on modes and modal systems Attention to the agentive work The scale of attention Differences

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Organizational geography Temporality of events Recognizing events: patterns and customs Detecting incongruities and breakdowns Sequences of interaction, before, after and immediately following one another Conversation analysis & multimodality: analytical focus

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Systemic Functional Linguistics (Anders Bjorkvall, 2009 in HMA)

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS

Social Semiotic Multimodal Analysis

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Analyzing Multimodal Interaction Norris (2004)

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Norris (2009) Modal Denisity in HMA

MODE multimodal methodologies FOR RESEARCHING DIGITAL DATA AND ENVIRONMENTS Sensory anthropology (e.g. David Howes) Socio-cultural approaches and multimodality (e.g. Barbara Rogoff, Roger Saljo) Eye-tracking (Jana Holsvana) New literacy Studies (Kate Phal, Jennifer Rowsell) Historical analysis with a multimodal focus (e.g. Grosvenor) Other approaches