Painting an experience? How aesthetics might assist a neuroscience of sensory experience Ron Chrisley Centre for Research in Cognitive Science and School of Informatics, University of Sussex Neuroesthetics: Where Art and the Brain Collide ESF Workshop, IULM, Milan September 2009

5 relevant areas of my research 1.Embodied creativity 2.Enactive models of experience 3.Synthetic phenomenology 4.Interactive empiricism 5.Art works/installations (The title of my talk concerns area 3)

My research and neuroesthetics Not a neuroscientist Much of the work I am reporting on is trying to provide a bridge from art/aesthetics into the cognitive sciences, which then permits a connection with neuroscience Re: this workshop: –How can the models of aesthetic processes I am investigating be informed by neuroscience/aesthetics? –Can the models suggest future directions for neuroscience? Aesthetics/art?

1. Embodied creativity Goal: Design a robot/environment system likely to exhibit creative behaviour: –Novel (at least for the robot) –Of (aesthetic) value (for humans, if possible) Engineering approach: –No direct modelling of human creativity –But exploit what is known about creativity in humans (and animals?), when expedient –Allow for possibility that insights into the human case may accrue anyway Manifesto only: No implementation yet –Set of "axioms" –Assume case of musical output for examples

Principles of embodied (aesthetic) creativity 1. If you make your robot pleasure-seeking, and make creativity pleasurable, you'll make your robot creative 2. To be a good creator, it helps to be an appreciator 3. Let the robot experience output in the real world, as we do 4. We won’t like what the robot likes unless it likes what we like 5. An important motivator is the approval or attention of others 6. Novelty can be achieved by trying to produce outputs on the subjective edge of chaos (that lie just beyond the robot’s ability to explain/predict) 7. Let dynamics play a role in appreciation 8. Patterns in one's own states can be the objects of appreciation 9. The best way to make outputs in the real world is to be embodied in the real world

Underlying architecture CNM: –Recurrent neural network –Forward model of environment Learns to anticipate/predict the sensory input it will receive if it performs a given action in a given context In conjunction with motivators can enable the robot to select actions that carry an expectation of "pleasure"

Underlying architecture Key: Recurrent Connection (Copy) Full Inter-Connection Between Layers Of Units Action Expected Sensations Predicted State Previous Predicted State (Context Units) D-map T-map

Generalisation to aesthetic experience Perhaps the content of aesthetic experience can be understood using the same framework, with two extensions: 1.Affective expectations 2.Aesthetic expectations

Aesthetic experience: affect Just as a system can have expectations concerning its actions and resulting sensory inputs, it can also have expectations concerning its actions and resulting desirable or undesirable states Thus, affect, and in particular the affective character of aesthetic experience, may be accommodated in EBA

Aesthetic experience: artifice When we experience a visual work of art we do not just experience its visual properties (explicable in terms of expectations to receive input x if we move our eyes thus) We also experience it as a work of art: we possess expectations for how the work would change if we were to make (or the artist were to have made) this or that brushstroke Set of all such expectations is the content of the aesthetic experience of the work

3. Synthetic phenomenology A science of consciousness needs a way to refer to or specify the content of conscious experiences Standard means: e.g., "Mary is having a visual experience of a red bike leaning against a white fence" Problem: Can only specify experiences with linguistic, conceptual, content Yet several good reasons to believe that some of content of experience is non-conceptual

Do I have to draw you a picture? An obvious alternative is to use non-linguistic, non-symbolic specifications E.g., for the case of visual experiences, use images Can't just take a picture of the scene the subject is seeing (literalism) Even in the case of a robot model of experience, can't just use the raw video camera output –For example; the current "output" of a human retina contains gaps or blindspots that are not part of experience. –Furthermore, our visual experience, as opposed to our retinal output, at any given time is stable, encompassing more than the current region of foveation, and is coloured to the periphery But what alternatives are there?

Depictive specifications of the content of visual experience If the set of expectations determines experiential content, then displaying those expectations (in the right way) will count as a specification of that content "Filled-in" areas specify what input the robot would expect to receive if it moved its head so that it is looking in that location Grey areas do not indicate an expectation to receive grey input; they indicate to you the absence of any expectation for that location –"Absence of expectation is not an expectation of absence" –Alternative architectures (e.g., generalising neural networks) would have no such undefined regions of state space

Depictions Need not explicitly occur anywhere in the agent; are of experience corresponding to current expected inputs, not actual inputs Generated off-line by the theorist: –Sample the action space –Feed actions into forward model –Arrange the resulting expected sensory patterns in a spatial array according to the spatial relations of the sampled actions Theorist's experience of the depiction (plus interpretation/bracketing instructions) is the same as (or shares crucial properties with) the experience to be specified

Depiction of an expectational state

Depictive specification and art In a sense, content specification is what (at least some) artists have been trying to do for millenia Thus, a clear role for –artists –graphic designers –sound engineers –directors –etc.

Depictive specification and art E.g., creating a film of a car chase in the desert Not just a matter of "objectively recording" Requires artistic insight into how film will affect viewers, e.g.: –Pacing –timing, number, kinds of cuts and edits –what is in the frame –steadiness Cf Picasso's portraits

Problem: Subjectivity of Art? Science strives to be objective But content of a work of art is highly subjective Yes, but artisitic insights can be objectively investigated; e.g.: –Perspective –Moving images: “persistence of vision” –Effect of colour on mood\ –Et al

Problem: Subjectivity of Art? Is subjectivity always at odds with science? Objective science is not the elimination of the scientist's subjectivity Rather the negotiation of it Perhaps subjectivity of theorist can be exploited Lessons from interactive Art?

A two-way interaction Often, it has been thought that the interaction was one-way: –cogntive scientists informing the work of designers, artists, and other "creative" types But the converse interaction is needed as well: –Artistic input into radically different means of specifying the contents of experience

4. Interactive empiricism Some problems concerning (aesthetic) experience are philosophical; require conceptual breakthroughs/progress Such breakthroughs (e.g. new concepts) may not be achievable by reason alone, but require experiential activity –“…Modelling of consciousness… requires some clarifications and refinements of our concept of consciousness. Design of, construction of, and interaction with artificial systems can itself assist in this conceptual development.” (Sloman and Chrisley 2003) –Sensory augmentation (e.g., work with Froese and Spiers on how using the “Enactive Torch” might alter concepts of perception) –Art works are also “artificial systems”: creation of and interaction with

5. Art works/installations

Recap 1.Embodied (artistic) creativity 2.Enactive models of (aesthetic) experience 3.Synthetic phenomenology: specifying the content of the sensory, affective and aesthetic components of experiencing art works 4.Interactive empiricism: changing our concepts of experience through creating or interacting with art 5.Art works/installations based on the above

5. Art works/installations

Thank you. Comments welcome: