1. Collaborative Jazz Improvisation as an example of Cognitive and Network Extension David Mott, John Ibbotson, IBM 12 th May 2009

2. Structure Introduction –extended cognition –jazz playing Jazz as collaborative extended cognition Jazz as an networkable activity Experiments Relevance to ITA

3. Introduction

4. What is Cognitive Extension? Coupling with external artefacts –counting on fingers Socially extended –language to share thoughts, refine concepts, execute plans Sense of self –affected by interaction with others –confidence as an improver of performance [The Extended Mind, Clark and Chalmers]

5. Examples that convince me of Cognitive Extension Driving a car: –the car drives itself –I have the same feeling of vulnerability when I turn that I have with my own body Playing jazz music –(on a good day) the piano plays itself Consider the jazz example …

6. Leaving artefacts in the environment augment memory –memory == existence of an entity in a place –keys in same place –(plant triggers in memory) reminders/requests to do things –notes, paper trails for self and others

7. Using tools notepads calculators hoey [cognition in the wild]

8. Is jazz cognitive and capable of cognitive extension?

9. Jazz group instruments are tools need to collaborate need to problem solve

10. Use of patterns Some aspects of performance is realtime –no time to construct totally new melody lines –people use patterns previously learnt –but not necessarily exact note-for-note, so cognition is still required Other aspects are less realtime –who is doing next solo –this can be done by problem solving

11. Is playing jazz NEC? Is it Cognitive? –a complex problem solving activity involving sequencing, tempo, structure (you only have abstract chords for guidance), in real time –Improvisation requires more than just muscle memory Is it network? –must synchronise, respond, and lead other members of the group (different roles for same player at different times in an ad hoc way) –If you cant hear, you cant play –If the groove is good, you play better (mismatch of different players rhythms can destroy playing) –If the cross-musician feeling is good you play better (eg take up of others improvisation and rhythmic ideas) Does the tool (the piano) assist? –out of tune means you cant play –good key action leads to better playing –good sound quality leads to better playing –there is an intimate connection between tool and mind Do the tools interact? –bad matching (eg guitar and piano in same tonal register) can destroy playing Therefore playing jazz is definitely NEC! –The sum is significantly greater than the whole, and the network and tools are essential –(I am talking here about quality of playing - achieving a jazz feel, not basic note work)

12. Do jazz performers leave artefacts? Does one performer leave something for later use? –to another performer –to himself –to the environment e.g. start a song with a certain rhythm, tempo? –really setting up a pattern

13. Instruments as tools Does the particular layout of the instrument enhance the cognitive extension? Most jazz tools have a visual component (keyboard lays out chords spatially but different transpositions have different patterns)

14. Do performers play other instruments (use other tools) Does piano player play the drums? –piano player does play rhythmically –if the piano is heard as drums then this is like playing a drum? –eg the passing drum rhythm to drummer Does piano player play the bass? –e.g. if bass is lost then piano can play the bass line to resynch

15. Jazz performance - Tactical and Strategic Strategic –Book the hall and the group –Advertise Tactical –perform a piece

16. Can jazz be network extended?

17. Utility of the Network To achieve the quality of face to face? –when communication MUST be distributed, with reduced communication, can the network rise to the same quality as face to face? To improve over face-to-face? –Are there situations where the network can enhance performance eg training direction better awareness of what others are doing better awareness of holistic view of performance

18. What does NEC give you? People playing jazz together play better than on their own Other players contribute to creativity But group must also be disciplined for this to happen –some maverick behaviour (strange chords, rhythms, phrases) may add to creativity, but too much is disruptive

19. Confidence Confidence is key to quality of playing and integration Easy to lose confidence if group not working together, not synchronised Can the players help increase confidence Can the network help increase confidence Might a bad network reduce confidence? Rhythmic stability seems very important

20. Rhythmic collaboration (1) Ending a song Actions –looked at the drummer and played a rhythm –taken over by drummer –subsequently mutual ending Needed –visual communication –music communication –shared schema/pattern

21. Rhythmic collaboration (2) Guitar lost synchronisation –Save the Last Dance for Me has unusual 5-bar phrase lengths –Guitarist changed chord too early –Pianist emphasised new chord at the right place and guitar resynchronised Needed –music communication (both ends) –musical validation ability –shared memory of song

22. Schemas Schemas are typical patterns/stories we expect to happen –eating at restaurant –dodgy accountants –enemy movements? Musical Schemas –4 bar phrases –Surprise Symphony [play example] –jazz structure: tune, solo, tune

23. What is communicated? Organisational –who is to solo next –when a solo is finished –timing of piece –when to complete a piece Leadership –leadership passes between the players, semi ad hoc Musical Patterns –Tempo, eg change to double time –Style, eg bebop, latin –Phrases, specific phrases may be echoed, or modified –Harmony, eg player may lay down a chord substitution and this is taken up by others

24. Communication of Information Rhythmic Metre (beat), the gross timing of the piece sync the tapping of feet Rhythmic Style, the gross type of piece, eg Latin, Swing, Rock selection of accompaniment pattern Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to lead the rhythm? sparseness of accompaniment Harmonic Progression, roots sync the place in the sequence more space to play richer higher voicings Harmonic Progression where I am???

25. Communication of Information Rhythmic Metre (beat), the gross timing of the piece sync the tapping of feet Rhythmic Style, the gross type of piece, eg Latin, Swing, Rock selection of accompaniment pattern Rhythmic Confidence/Stability, whether the drummer is reliable personal confidence need to lead the rhythm? sparseness of accompaniment ?? easier to play if the piano sound is good ??

26. Others tools Is it possible to extend the mind into others instruments? In a genuinely collaborative situation can this achieve a deeper common understanding? Would need intimate feedback between other tool and ones brain, via network

27. Key collaborative performance aspects Synchronised rhythm –style (accents, patterns) varies (swing, latin) –key aspect for a good performance –key for confidence Harmonic structure –where are we in the chord sequence? Organisational structure –when do we end and change solos?

28. Creating jazz NEC

29. How do we create a Jazz NEC? MIDI instruments: –keyboard –midi drums –midi guitar (or use keyboard) MIDI is a stream of digitised events that can be communicated, instrumented and recorded Setup a MIDI group across the network with remote players, and play! –connect via sensor fabric –build OpenSim module to communicate MIDI?

30. NEC over the Sensor Fabric Fabric Node Sensor Client Sensor Client Sensor Client Synth Plugins Dashboard Control Data Control

31. Experiments

32. How do we evaluate a Jazz NEC? Evaluation is quality of jazz feel –must be done by other musicians (or keen listeners) subjective feel –BUT I believe you will get a good degree of agreement (you can tell when a band is together or tight) Could you analyse the MIDI stream mathematically? –Not sure, you might pick up gross defects (mismatch of rhythm, complete collapse of group), but for the jazz feeling I suspect this is elusive

33. What can be measured? Degree of music playing itself –subjective judgement by player post hoc listening and recall (there I was playing well, there I got lost) additional effector (foot button?) pressed to indicate im now in the flow Emotional State –Emotive-like devices, measuring emotion, mood, physiological factors Group review by players –post hoc listening and recall –discussion of what was going on, issues (I couldnt hear the guitar, the drums were lagging, I felt the group was not together) …

34. Experiments Effect of network quality –can artificially reduce MIDI signals between players reduce volume so cant hear drummer simulate bandwidth reduction by dropping packets Effect of face-to-face presence –playing over headphones vs being in visual contact Effect of injecting information –put drum double time rhythm onto the network to see if it is taken up by others If things go wrong, can the group recover?

35. QOI Quality of Information measures could be applied to performance?

36. Measuring vs changing Could just measure a performance Could introduce a change and see the effects It may be premature to experiment with the effects of introduced change

37. Simple intervention – play to click Rising to f-f: –prioritise click over full drums, given reduced bandwidth –agent to approximate drums and send to players Improving f-f –already used in real performances –would click give confidence to new players –intervene with double time click? –auto quantise drums – would this increase confidence?

38. BeatFinder Locate beats in music –1, 2, 3, 4, 1, 2, 3, 4 –a complex task, lets assume we can do it Show coloured screen for in time, early and late? Detect individual accuracy of beats Detect collaborative accuracy of beats Work out and display where in the music we are

39. Displaying place in tune use beat tracker to calculate place in tune –jazz navigational structure usually fairly simple –would need to visualise place in context? would this improve confidence in new player? or take away the skills learning?

40. Experiments with beat finder Surely groups already synchronise their playing? –NO, not if nervous or multiple versions of rhythm and not clear where beat is Possible experiments: –For individual practise –Dashboard view of collaborative performance –See other players beats (all, or selected) –One player (drummer, trainer) pushes a synchronisation pulse to other players What effects would this have?

41. Using network for training transcription/copying is very important for learning could a trainer play examples to copy? –in the context of a networked performance –note-for-note copy or approximate patterns could a trainer give real-time cues?

42. The relevance to ITA

43. What is the point of jazz in the ITA? An example of extended mind –well understood, easy to relate to (for some) –an articulation of the abstract cognitive concepts NATO uses it as an example of C2 –???? Realistic test bed of some complex experiments Stimulation of new ideas

44. Metaphors (Lakoff) Cognitive metaphors bridge from concrete to abstract –LIFE IS A JOURNEY –TIME FLOWS FORWARDS –EMOTIONS GO UP AND DOWN (Saslow) Musical Metaphors –MUSIC IS A PATH –MUSIC IS A FORCE

45. The FORCE Metaphor MUSICAL EXAMPLE Military TERMS DEFENCE, SEIZE

46. Jazz as extended mind coupling with external artefacts –finger positions on keyboard to calculate harmonic progressions socially extended –other people as memory of tunes, styles, rhythms, harmonic progressions –synchronisations of rhythms sense of self –jazz playing in social context affects sense of self