1. 1 CASPAR Contemporary Arts Testbed: CASPAR Contemporary Arts Testbed: Preservation of Interactive Multimedia Performances Kia Ng and Eleni Mikroyannidi ICSRiM - University of Leeds caspar@icsrim.org.ukcaspar@icsrim.org.uk www.icsrim.org.uk/casparwww.icsrim.org.uk/caspar

2. Overview Context and Motivation –CASPAR EC IST Project Contemporary Arts Testbed –Interactive Multimedia Performance (IMP) Music via Motion (MvM) i-Maestro 3D Augmented Mirror (AMIR) System ICSRiM Conducting Interface Research and Development –IMP dataset –CIDOC-CRM and FRBR ontologies Result and Conclusion –ICSRiM IMP Archival System Interface and functionalities Scenarios Validation

3. Context CASPAR EC IST Project –Build a framework to support end-to-end preservation lifecycle for scientific, artistic and cultural information Contemporary performing arts testbed –Interactive Multimedia Performance (IMP) Interactive Multimedia –Interactive multimedia performances –Performance capture/recording and analysis beyond normal AV

4. Music via Motion (MvM) MvM: a framework to integrate interactive multimedia, motion sensing, audio synthesis, VR and AR technologies –To explore virtual and augmented instruments –To provide users/performers with real-time control of multimedia events with their physical movements Application areas include stage performance and installation arts

5. IMP Elements and Processes Example of an IMP production process based on the Music via Motion (MvM) system, www.kcng.org/mvm Data Analysis, Classification and Recognition Mapping Multimedia Generation GUI (for monitor & control) Data Capture & Processing Motions multimodal data Multimedia output Mapping Strategy Parameters

6. i-Maestro 3D Augmented Mirror (AMIR) Technology-enhanced learning More in-depth recording and understanding –playing gesture –Intangible heritage –Individual performance style Data stream include: audio, video, 3D motion data, sensor data, analysis Multimodal feedbacks with graphical visualisation and sonification Max/MSP jitter, c/c++, Lua, SDIF Excerpt from Bach's Partita No. 3 in E major. BMW 1006 (Preludio)

7. AMIR (www.i-maestro.org) 7

8. ICSRiM Conducting Interface A system for tracking and analysing a conductors hand movements –Uses multiple wiimotes to capture a conductors movements –These movements are analysed –This information is then fed back to the user in an entertaining yet educational manner

9. System Setup

10. GUI 3D environment Intuitive UI Multiple modes Multimodal data capture and analysis

11. Complexity of the IMP Preservation IMP rely heavily on digital media A recording (e.g. video) of a performance is not sufficient for re-performance Preserving individual components of a performance is clearly a challenge However, putting these components together in one place does not make a performance They need to be assembled in a logical and temporal order with their inter-relationships Preserving this knowledge through time is even more challenging Preservation for –Re-performance at a later time –Historical study/analysis in performing arts in the future –Better preparation for the future needs Example IMP interface from the i-Maestro EC IST Project involving 3D motion data, sonification, and graphical feedbacks Example IMP - from a Music via Motion (MvM) performance involving motion generated music with dance

12. 12 IMP Dataset An Interactive Multimedia Performance (IMP) work typically consists of the following digital objects: –Audio –Video –2D/3D motion data –Sensor data –Max/MSP patches –Configuration files –and others For reconstruction of an IMP work –Representation Information –Inter-relationship Logical Temporal

13. CIDOC-CRM for IMPs The CIDOC Conceptual Reference Model (CIDOC-CRM) has been proposed as a standard ontology for enabling interoperability among digital libraries It defines a set concepts for physical and temporal entities Conceptual Objects Physical Objects Types Temporal Entities Appellations Actors PlacesTime Spans are referred/ refine participate in withinat affect/ refer to has location refer to/ iden tify

14. CIDOC-CRM for IMPs (2) CIDOC-CRM can be used to describe a performance –at a high level More specialised vocabularies are necessary for IMPs –e.g. how different types of equipment are connected together in the performance. CIDOC-CRM –lacks concepts for digital objects –is designed primarily for documentation of what has happened, whereas in digital preservation, it is also required to document the reconstruction of a past event from preserved components.

15. CIDOC-CRM FRBR Extension Two main objectives in extending CIDOC-CRM for the preservation of an IMP: –To provide a domain specific vocabulary for describing an IMP. e.g. details on how the archived performance was carried out and how it can be re-performed –To provide a vocabulary for describing digital objects, their interrelationships and operations performed on them in the digital preservation context FRBR (Functional Requirements for Bibliographic Records) –describes a high level conceptual model of creative works and how they are represented in the real world It is used as extension ontology of the CIDOC-CRM –for describing the conception process of the performance 15

16. 16 Representation Information The Representation Information for this performance is in the form of CIDOC-CRM and FRBR instantiation. F52.Performance IMP F8.Person Kia (Director) Frank (Performer) E53.Place Leeds - UK E52.Time-Span 2hours:5PM-12/02/07 E22.Man-Made Object Cello Sound Mixer Computer System E73.Information Object Music Music Score P16F – use specific object P7F – took place at P4F – has time span P14F – carried out by

17. 17 Mapping IMP details to CIDOC-CRM and FRBR The FRBR concept F52.Performance is used to model an IMP. The FRBR concept F8.Person represents the people involved in the performance, e.g. directors, performers, MAX/MSP programmers, etc. The digital objects used (software patches, 3D models, etc) are instances of the E73.Information Object CIDOC-CRM class, whilst any physical objects (instruments, computer equipment, etc) are instances of the E22.Man-Made Object class. The E52.Time-Span and E53.Place classes are used to represent the time-span and location of the performance respectively.

18. 18 Mapping IMP details to CIDOC-CRM and FRBR (contd.)

19. ICSRiM IMP Archival System The ICSRiM IMP Archival System –for the preservation of different IMPs –based on the CASPAR Framework –Integration of the selected CASPAR components Communication with the Repository containing the Knowledge base and the DataStore Archival System Knowledge Base RepositoryDataStore communicates RepInfo Performance Files CASPAR Web Services calls

20. Web Interface of the IMP Archival System

21. Cyclops tool A tool to capture appropriate Representation Information to enhance Virtualisation and future re-use of the digital object. Cyclops provides a usable interface to describe the Performances in the form of graphs The graph in the backend is saved as an RDF instantiation of the CIDOC and FRBR ontologies Defined terminologies according to various data types

22. Cyclops Interface

23. Ingestion and Retrieval of an IMP Scenario 1: Ingestion of an IMP –Description of the performance with the use of Cyclops Generation of the Representation Information –Both the metadata and the digital files are ingested and stored in the Repository Scenario 2: Access of an IMP –Queries are performed on the metadata –The related objects are returned to the user –Instructions extracted from the RepInfo provide details on how to assemble the objects

24. Scenario 1: Ingestion of an IMP Steps in Screenshots 24

25. Step 1: Describe the IMP Follow the Ingest wizard to complete the ingestion 25

26. Step 1: Describe the IMP (2) Create Representation Information with the use of Cyclops tool It is saved in RDF format. In this way, the graph was transformed into an RDF instantiation of the CIDOC-CRM and FRBR ontologies. 26

27. Step 2: RepInfo parsing The system detects and parses the RepInfo for defined data types 27

28. Step 3: Addition of the digital files Upload the related digital files of the performance The files are linked to the RepInfo The package is stored in the Repository 28

29. Scenario 2: Retrieval of an IMP Steps in Screenshots 29

30. Step1: Search for the IMP The search is done by the name of the performance 30

31. Step1: Search for the IMP (2) The system returns a table with the following selected information from the RepInfo –The name of the Performance –Details on the performance –The associated with the performance files –The links to download the files of the performance –Details on the files (e.g. which one is the data file, the patch file, how to use them etc.) –A link to the RDF description. 31

32. Step 2: Download Associated Performance files Select and download the components of the performance 32

33. Step 3: Reconstruct the performance The user has to understand the connection of the components Instructions on the connection is provided in the table By assembling the components the user is able to see the performance 33

34. Retrieval of an IMP

35. Validation and Feedback Participant –A small set of experts who work with IMP Validation –Ingestion of an IMP work (their own work) –Retrieval and reconstruction of an IMP they have no information about it Download and connect the components using the instructions –Fill in a questionnaire with their feedback Results –The system is useful to store and preserve the IMPs –The system provided all the needed instructions for the reconstruction of an IMP The appropriate RepInfo has been captured –Some difficulty in using Cyclops (complexity of the graphs) Demo videos and tutorials helped Cyclops graph templates to help to get started

36. CASPAR IMP References Selected publications include: –E. Mikroyannidi, B. Ong, D. Giaretta and K. Ng, Preservation of Interactive Multimedia Performance with the use of Ontology Models, in Proc. of the Digital Resources for the Humanities & Arts (DRHA) Conference, Belfast, September 2009 –K. Ng, E. Mikroyannidi, B. Ong, N. Esposito and D. Giaretta, "Interactive Multimedia Systems for Technology-Enhanced Learning and Preservation" in Proc. Of the 15 th International Conference on Distributed Multimedia Systems (DMS) 2009, San Francisco, September 2009. –K. Ng, T.V. Pham, B. Ong, A. Mikroyannidis, and D. Giaretta, "Preservation of Interactive Multimedia Performances", International Journal of Metadata, Semantics and Ontologies, 3(3), 28 February 2009, pp. 183-196, Inderscience Publishers. For latest publications on CASPAR IMP Preservation, see –www.icsrim.org.uk/casparwww.icsrim.org.uk/caspar –CASPAR project website 36

37. Conclusions Preserving knowledge is vital for future reconstruction of interactive multimedia performances Proposed IMP preservation process –Made use of standard ontology models, CIDOC-CRM and FRBR to create Representation Information Used to define interrelationships between the components Creation of a web-based archival system for the preservation of IMPs –Integrated selected components of the CASPAR Framework –Current prototype online: http://www.icsrim.org.uk/imp-preserve 37

38. Thank you Kia Ng and Eleni Mikroyannidi ICSRiM – University of Leeds www.icsrim.org.uk/casparwww.icsrim.org.uk/caspar caspar@icsrim.org.ukcaspar@icsrim.org.uk 38