1. Dimensions of Media Object Compehensibility Lawrie Hunter Kochi University of Technology http://www.core.kochi-tech.ac.jp/hunter/

2. Dimensions of Media Object Compehensibility Lawrie Hunter Kochi University of Technology http://www.core.kochi-tech.ac.jp/hunter/ KUT Island of Shikoku

3. Kochi Niigata Osaka

4. A pattern language for MMC Source of insight: language / language learning Second language (L2) learning: a cognitive process? Comprehension of partially acquired L2: revealing of the nature of text/media. Language learning issues are germane to MMC.

5. Tempering: questions of significance and applicability for machine automation. A pattern language for MMC Intervention author's structural model of content information (for second language learning materials) Discussion of parameters of difficulty Ground: related issues in second language learning materials Exemplars ‘ considerate text ’ ‘ considerate multimedia ’ Generating parameters of difficulty in media object comprehension Background work towards a human-communication paradigm for the guidance of machines Frame the new multidisciplinary approach of machine-mediated communication Objective development of a pattern language for that multidisciplinary approach to MMC Focus factors influencing the difficulty of comprehension of media objects Question how media objects carry information.

6. L2 learning materials must all be more immediately apparent to the learner than in the case of materials for L1 medium learning scenarios The creation of second language (L2) learning materials demands document transparency : 1. document purpose 2. document content 3. target behavior 4. target lexical items

7. The creation of second language (L2) learning materials demands document transparency : L2 learning materials 1. document purpose 2. document content 3. target behavior 4. target lexical items work towards transparency is informed by difficulty-related issues inform human interaction with info media

8. Earlier work: an EAP tool *in a talk to KMI at the Open University David Kolb* re using hypertext to present scholarly text: "...the easiest ways of making a complex argument available in HT tend to move the text toward linear structures that do not take full advantage of the possibilities of linked text."

9. Earlier work: an EAP tool *in a talk to KMI at the Open University David Kolb* re using hypertext to present scholarly text: "...the easiest ways of making a complex argument available in HT tend to move the text toward linear structures that do not take full advantage of the possibilities of linked text." "...what the HT can do is present the argument, but also use linkage and juxtaposition to make the reader’s engagement with the argument more creative, self-conscious, and self-critical."

10. Earlier work: an EAP tool Lawrie Hunter re using hypertext to present technical L2 text: For the L2 reader, engagement can only be enhanced if the rhetorical and information structures are articulated.

11. Earlier work: an EAP tool Lawrie Hunter re using hypertext to present technical L2 text: For the L2 reader, engagement can only be enhanced if the rhetorical and information structures are articulated. What the HT can do for the NNR/W is tp present simultaneously the various faces of a research paper: the rhetorical moves; the bits of structured information; the text; necessary glosses. * NNR/W EAP = non-native reader/writer of English for Academic Purposes

12. Earlier work: an EAP tool Lawrie Hunter re using hypertext to present technical L2 text: For the L2 reader, engagement can only be enhanced if the rhetorical and information structures are articulated. What the HT can do for the NNR/W is to present simultaneously the various faces of a research paper: the rhetorical moves; the bits of structured information; the text; necessary glosses. And if the NNR/Ws design their personal interface, a negotiated pattern language of NNR/W EAP* will emerge. * NNR/W EAP = non-native reader/writer of English for Academic Purposes

13. Arguably important direction "Tomorrow's literacies... need to be process and systems literacies.” -John Thackara, In the Bubble: Designing in a complex world. MIT Press 2005.

14. Rhetorical structures Knowledge structures Cohesion devices Grammar (sentence surface structure) Background Extension DiversionsTrain of argument This is the domain of texture down here. This is the domain of structures up here. Rhetorical structure theory, systemic functional linguistics and knowledge structure mapping form a hierarchy of structures, whereas grammar and sentence diagrams reflect rules for texture management. Functional structures Structural view of writing

15. Rhetorical structures Knowledge structures Cohesion devices Grammar (sentence surface structure) Background Extension DiversionsTrain of argument False hierarchy: the train stops here. This is the domain of texture down here. This is the domain of structures up here. Rhetorical structure theory, systemic functional linguistics and knowledge structure mapping form a hierarchy of structures, whereas grammar and sentence diagrams reflect rules for texture management. Functional structures Structural view of writing

16. L2 reader needs analysis Knowledge Niche grammar structures Niche rhetorical structures General register repertoires (distinguishing formal academic from informal academic) Research Paper text structure and information structure Language skills Argument sequencing Info-structured sentence generation Mimicry of model language Facilities Concordance & collocation resource Bank of model research papers (annotated*) *c.f. Brown and Brown’s ‘annotation’

17. L2 reader wants analysis In a technical hypertext, L2 reader/writers want*: 1. Glossing (of 'difficult' terms and phrases) 2. Moves indicator 3. Lexia position indicator 4. PDF-drawer-like phrase recurrence tab 5. Register converter (e.g. research paper presentation script) 6. Information structure maps for atomic utterances 7. Overall argument map on every lexia (similar to Horn's argument maps or Rhetorical Structure Analysis?) *Based on a survey of 22 PhD engineering students

18. Technical hypertext design: WANTS NEEDS A pattern language? www.patternlanguage.com

19. Technical hypertext design: WANTS NEEDS …The language, and the processes which stem from it, merely release the fundamental order which is native to us. They do not teach us, they only remind us of what we know already, and of what we shall discover time and time again, when we give up our ideas and opinions, and do exactly what emerges from ourselves. -Christopher Alexander, The Timeless Way of Building A pattern language? www.patternlanguage.com

20. Do humans have a GRAPHIC THOUGHT FACILITY? The knowledge structure map is a matrix (confluence) for the situated learner* and the situated mentor to confirm context and the nature of "stolen property."** *Jean Lave **Duguid and Brown < $$$ !

21. Hunter’s knowledge structure map links < big Description Classification Degree comparison Attribute comparison SequenceCause-effect Contrast !

22. 2005 project: design level EEAP* students: HT designs for the analysis of technical academic papers. *EEAP = Engineering English for Academic Purposes, a subset of EAP, which is a subset of ESP (English for Specific Purposes) Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing. Presentation at HyperText 2005, Salzburg. http://www.lawriehunter.com/presns/%20HT05poster0818.htm http://www.lawriehunter.com/presns/%20HT05poster0818.htm

23. TEXT STRUCTURE Introduction Background Question Methods and materials Results Observations Conclusion INFOMAP(s) INFO STRUCTURE Describe Classify Compare Sequence Cause-effect Contrast UTTERANCE(s) In general, power plants boil some liquid to make steam, which rotates turbines, which generate electricity. Power plants boil a liquid to produce steam, which is used to rotate turbines, which in turn generate electricity. RHETORICAL MOVES Common knowledge Cite Report Explain Claim Question Qualify Evaluate Decide Infer Project

24. TEXT STRUCTURE Introduction Background Question Methods and materials Results Observations Conclusion INFOMAP(s) INFO STRUCTURE Describe Classify Compare Sequence Cause-effect Contrast UTTERANCE(s) Traditional power plants use fossil fuel heat or heat from nuclear fission to boil water and produce steam at 500°C. Older type power plants boil water with heat from fossil fuel combustion or nuclear fission to produce steam with a temperature of 500°C. RHETORICAL MOVES Common knowledge Cite Report Explain Claim Question Qualify Evaluate Decide Infer Project

25. TEXT STRUCTURE Introduction Background Question Methods and materials Results Observations Conclusion INFOMAP(s) INFO STRUCTURE Describe Classify Compare Sequence Cause-effect Contrast UTTERANCE(s) OTEC power plants use seawater heat to boil ammonia and produce steam at 20°C. OTEC type power plants boil ammonia with the heat of the sea to produce steam with a temperature of 20°C. RHETORICAL MOVES Common knowledge Cite Report Explain Claim Question Qualify Evaluate Decide Infer Project

26. RHETORICAL MOVES Common knowledge Cite Report Explain Claim Question Qualify Evaluate Decide Infer Project TEXT STRUCTURE Introduction Background Question Methods and materials Results Observations Conclusion INFOMAP(s) INFO STRUCTURE Describe Classify Compare Sequence Cause-effect Contrast UTTERANCE(s) Traditional power plants use fossil fuel heat or heat from nuclear fission to boil water and produce steam at 500°C, whereas OTEC type power plants boil ammonia using the heat of the sea to produce steam with a temperature of 20°C. Older type power plants boil water with heat from fossil fuel combustion or nuclear fission to produce steam with a temperature of 500°C, while OTEC power plants use seawater heat to boil ammonia and produce steam at 20°C.

27. Obstacle in 2005 project Massive diversity in learner perception of knowledge structures.diversity

28. Obstacle in 2005 project Massive diversity in learner perception of knowledge structures.

29. Rhetorical structures Knowledge structures Cohesion devices Grammar (sentence surface structure) Background Extension DiversionsTrain of argument False hierarchy: the train stops here. This is the domain of texture down here. This is the domain of structures up here. Rhetorical structure theory, systemic functional linguistics and knowledge structure mapping form a hierarchy of structures, whereas grammar and sentence diagrams reflect rules for texture management. Functional structures Structural view of writing

30. Grammar staging Information orchestration Rhetoric, flow Sentence level Prescriptive order charts (linear); sentence diagrams Knowledge structure maps Topic/stress and subject-verb distance gizmos Paragraph level Readability charts Knowledge structure maps Old/new and topic/stress gizmos Document level Readability outlines Knowledge structure maps Old/new and topic/stress gizmos

31. 2006~ new layer: READABILITY The missing link in technical academic writing: Gopen’s readability -subject-verb distance -topic position / stress position -old/new information placement

32. Background: readability work In the design of traditional high-text language learning materials, readability is a prominent concern. Reading difficulty has for some time been seen as depending on -word length -sentence length -text length -number of sentences per paragraph -vocabulary ‘ difficulty ’ More recent work has extended this list to include -subject-verb distance -adherence to old/new position conventions -topic position/stress position conventions Treated extensively in Hunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379.

33. Background: MM readability Treated extensively in Hunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379. Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments. Language learning and technology 1(1): 60-81.

34. 2006~ new layer: READABILITY Hunter’s new TAW syllabus: assume grammar

35. 2006~ new layer: READABILITY TexturalStructural GrammarLexical patterns RegisterKnowledge structures CohesionCoherence/readability Functional grammarInformation organization Rhetorical deviceRhetorical structure

36. Readability The creation of second language (L2) learning materials demands appropriate readability. 1. understandable by the learner 2. ‘ stretching ’ learner knowledge/skill 3. contextualized to support stretching 4. orchestrated with degrees of scaffolding

37. Considerate text Original framing: ･ well-written, ･ well-organized, and ･ signals the organization of its thought to the reader One inroad to readability is considerate text :

38. Considerate text Original framing: ･ well-written, ･ well-organized, and ･ signals the organization of its thought to the reader One inroad to readability is considerate text : More recent takes: -glossing -phrase boundary marking -de-idiomatizing -the Plain English movement -graphic organizers -text nouveau

39. Text nouveau is still text Text comprehension in multimedia environments is a rich variant, BUT : Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments. Language learning and technology 1(1): 60-81.

40. Text nouveau is still text Text comprehension in multimedia environments is a rich variant, BUT : Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments. Language learning and technology 1(1): 60-81.

41. Sharing considerate text Appropriateness of learning materials/tasks is very complex. Tagging of these materials & tasks is daunting. L2 learning objects : welcome to the TagTower of Babel! KUT English is a Moodle department.

42. Sharing considerate text Appropriateness of learning materials/tasks is very complex. Tagging of these materials & tasks is daunting. Fortunately, as David Weinberger points out*, there is a huge amount of metadata out there, but this allows multiple simultaneous organizations of content. *June 12, 2007 interview with IT Conversations http://www.itconversations.com/shows/detail1838.html Weinberger books The Cluetrain Manifesto Small Pieces Loosely Joined Everything is Miscellaneous

43. Considerate multimedia? Tentative definition: considerate multimodal objects are those which contain few non-essential obstacles to their comprehension. Considerate text in the context of M3C suggests the notion of considerate multimedia

44. Considerate multimedia? Tentative definition: considerate multimodal objects are those which contain few non-essential obstacles to their comprehension. Considerate text in the context of M3C suggests the notion of considerate multimedia Tentative definition 2: considerate multimodal objects are those which are tagged for various forms of comprehension difficulty.

45. Considerate multimedia?

46. Multimedia comprehensibility? Alternative approach: create a set of parameters for multimedia comprehensibility “Considerate multimedia” confronts vastly more complexity than considerate text One approach to comprehensibility: explore obstacles to comprehensibility, as has been done in readability work.

47. In the domain of multimodal computer-mediated communication, the question of readability translates as ease of comprehension: Multimodal equivalent of readability How easy is it for a human to extract all the information contained in a multimodal media object (MMO)?

48. To measure the ease of extraction of all the INTENDED information contained in a MMO, we need a characterization of the difficulty of extraction: Tentative list of sources of information extraction difficulty, for simplicity ’ s sake limited here to text objects graphic objects speech objects video objects and combinations thereof. Parameters of media object function This is a tentative, exploratory framing of MMO comprehensibility,

49. ParameterInstance/unit concept densityexophoric references per paragraph/page/frame metaphor densitymetaphors per scene/argument/minute phoneme densityphonemes per unbroken utterance* (e.g. Italian speech) phonemes per inhalation phonemes per word phonemes per minute mathematical symbol densitynumerals per page numerals per sentence formulae per sentence/paragraph formulae per argument noise densitysuperfluous signals per utterance, e.g. "...in 1960, oh, sorry, I meant to say in 1960...) readability stoppages** per sentence asides per sentence/message cognitive dissonances per utterance facial expression/statement conflicts reference transparencyanchoring devices per lexia anchoring devices per reference channel-channel synchronicitynumber of channel-channel synchronicities number of channel-channel asynchronicities message-message agreementnumber of message-message agreements number of message-message dischords *utterance: minimal spoken, written or graphical communication unit Parameters of media object function Learning object tag ‘shopping list’

50. Concept density* in text space Concept density* in aural time Concept density* in video space *concept density = number of exophoric references per sentence/minute/frame Parameters of media object function Concept density

51. Metaphors per sentence. Metaphors per argument. Metaphors per minute. Idioms per sentence. Idioms per argument. Idioms per minute. Parameters of media object function Metaphor density

52. Phonemes per unbroken utterance*. Phonemes per exhalation. Phonemes per word. Phonemes per minute. *e.g. Italian speech. Parameters of media object function Phoneme density

53. Parameters of media object function Superfluous signals/utterance Readability ‘stoppages’ per sentence Asides per sentence/message Cognitive dissonances per utterance Facial expression-statement conflicts ‘Noise’ density

54. Parameters of media object function Numerals per page. Numerals per sentence. Formulae per sentence/paragraph. Formulae per rhetorical move. Symbol density

55. Parameters of media object function Imperfect audio channel Imperfect text channel Imperfect visual channel Channel imperfections

56. Parameters of media object function Channel imperfections Nass and Brave, Wired for speech Reeves and Nass, The media equation Finding: humans retain more info from video with an imperfect audio channel

57. Parameters of media object function Channel-channel discord

58. Parameters of media object function Channel-channel synchronicity

59. Parameters of media object function Message-message agreements Message-message discords Message-message harmony

60. Both involve reduced text density and spatial highlighting of text, and suggest the question of a 'graphic thought facility' in humans. mangaknowledge structure maps Low phoneme density Isolated conversational text chunks: X idioms per sentence. Isolated descriptive text chunks: 0 idioms per sentence X metaphors per utterance0 metaphors per utterance X idioms per utterance0 idioms per utterance Graphical situating: narrative/mood Graphical situating: symbolized relations to other text chunks manga vs. knowledge structure maps To illustrate the use of the parameter approach, here is a comparison of two relatively similar types of media objects, manga and knowledge structure maps. Both involve reduced text density and spatial highlighting of text, and suggest the question of a 'graphic thought facility' in humans.

61. Once a comprehensive set of parameters of MMO comprehensibility has been developed, questions of application will arise. How can (should?) these parameters be situated among larger semantic frameworks? Which of these parameters are relevant to the development of machine- mediated communication? How can they be operationalized in computable form? Tempering: questions of significance and applicability for machine automation Generating parameters of difficulty in media object comprehension

62. Work on ontology-based research writing * : reforming how scientific research is written/read. EXPO* and the Robot Scientist Does the ontology EXPO feed back from a machine interface with a body of knowledge/practice to a solidification of human interface with that body of knowledge/practice? Daunting: ontology-based readability EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf

63. Work on ontology-based research writing * : reforming how scientific research is written/read. “Use of Natural Language is a great hindrance when using computers to store and analyse data hence the growing importance of text-mining. We argue that the content of scientific papers should increasingly be expressed in formal languages. Is writing a scientific paper closer to writing poetry or a computer program?” Daunting: ontology-based readability EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf

64. Work on ontology-based research writing * : reforming how scientific research is written/read. Can humans now experience knowledge differently, thanks to machine interface work, i.e. through a formal language imposed for the machine’s sake? Will this reform how we read? how we think? Daunting: ontology-based readability EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf

65. References [1] Elsayed, A. (2007) Machine-mediated communication: the technology. 6th IEEE International Conference on Advanced Learning Technologies, ICALT 2006, 5-7 July 2006, Kerkrade, The Netherlands. [2] Hunter, L. (2005) Technical hypertext accessibility: information structures and rhetorical framing. Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, Salzburg, Austria. [3] Kalyuga, S. (2006) Instructing and testing advanced learners: A cognitive approach. Nova Science Publishers. [4] Mann, B. (1999) An introduction to rhetorical structure theory (RST). http://www.sil.org/mannb/rst/rintro99.htm [5] Mohan, B.A.M. (1986) Language and content. Reading, MASS: Addison-Wesley. [6] Nass, C. and S. Brave. (2005) Wired for speech: How voice activates and advances the human- computer relationship. MIT Press. Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments. Language learning and technology 1(1): 60-81. Grow, G. (1996) Serving the strategic reader: cognitive reading theory and its implications for the teaching of writing. Viewed June 30, 2007 at http://www.longleaf.net/ggrow/StrategicReader/index.html http://www.longleaf.net/ggrow/StrategicReader/index.html Goldman, S.R., & Rakestraw, J.A. (2000). Structural aspects of constructing meaning from text. In M.L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. II, pp. 311-335). Mahwah, NJ: Erlbaum. The Plain English movement http://www.plainenglish.co.uk/index.htm (de-idiomatizing)http://www.plainenglish.co.uk/index.htm

66. References 2 Research via ontologies Ian Horrocks http://www.cs.man.ac.uk/~horrocks/http://www.cs.man.ac.uk/~horrocks/ EXPO Ontology of scientific experiments http://expo.sourceforge.net/http://expo.sourceforge.net/ Soldatova L.N., Clare A., Sparkes A. and King, R.D. (2006) An ontology for a Robot Scientist. Bioinformatics (Special issue ISMB) (in press).An ontology for a Robot Scientist. Soldatova, LN & King, RD. (2006) An Ontology of Scientific Experiments. Journal of the Royal Society Interface (in press).An Ontology of Scientific Experiments.Journal of the Royal Society Interface EXPO: An Ontology of Scientific ResearchEXPO: An Ontology of Scientific Research by Ross D. King & Larisa N. Soldatova, Department of Computer Science, University of Wales, Aberystwyth. Hunter Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing. Presentation at HyperText 2005, Salzburg. http://www.lawriehunter.com/presns/%20HT05poster0818.htm http://www.lawriehunter.com/presns/%20HT05poster0818.htm Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379. (text nouveau) WordbyWord http://www.core.kochi-tech.ac.jp/hunter/WordByWord/index.html (text nouveau)http://www.core.kochi-tech.ac.jp/hunter/WordByWord/index.html Text usability for non-native readers of English. Ueta, R, Hunter, L. & Ren, X. Proceedings, Information Processing Society of Japan, Vol. 2003.7. Pp. 199-200. (phrase boundary marking) Information Processing Society of Japan

67. Thank you for your kind attention. Don ’ t hesitate to write to me. Lawrie Hunter Kochi University of Technology http://www.core.kochi-tech.ac.jp/hunter