Object Denotation Environment EuroCODE ~ Complete Object Denotation Environment A Component-based Semantic Knowledge Platform Erfahrungen mit einem EU-Proposal aus dem MultiMedia Lab Guerino Mazzola

Autumn 2002 The Topos of Music Geometric Logic of Concepts, Theory, and Performance In collaboration with Stefan Göller and Stefan Müller Contributions by Carlos Agon, Moreno Andreatta, Gérard Assayag, Jan Beran, Chantal Buteau, Roberto Ferretti, Anja Fleischer, Harald Fripertinger, Jörg Garbers,Werner Hemmert, Michael Leyton, Emilio Lluis Puebla, Mariana Montiel Hernandez, Thomas Noll, Joachim Stange-Elbe, Hans Straub, Oliver Zahorka 1368pp/CD-ROM ISBN 3-7643-5731-2 ToM

... to survive! Apply! in order to survive

EU 6th FP Project Category 6th Framework Program -> Priority 2 = IST = Information Society Technologies 1st Call (Dec.2002) Activity code = IST-2002-2.3.1.7 ~ Semantic-based knowledge systems multimedia content mining, distributed computing platforms, mastery of complex information spaces, visualisation of high-dimensional objects, new formal models, covering foundational component-level as well as system-level research -> Instrument = STREP = Special Targeted REsearch and innovation Project (new results/knowledge -> develop products) -> STREP Activity = RTD + innovation-related + project management activities + demonstration activities EU 6th FP Instrument = STRIP = Special Targeted Research and Innovation Project

1. Knowledge is ordered access to information 3 principles

2. Knowledge is controlled by multimedia objects 3 principles

3 principles 3. The body of knowledge evolves in a cyclic process Production 3 principles Reflection Dissemination Documentation

presto® presto Here, we illustrate the theory on the jazz CD „Synthesis“ which I recorded in 1990. Its entire structure, in harmony, in rhythmics, and in melodics, was deduced and constructed by use of the composition software presto® (written for Atari computers, but now also working on Atari emulations on Mac OS X), and starting from the 26 classes of three-element motives. This composition (a grant from the cultural department of the city of Zurich) was not recognized as a computer-generated music by the jazz critics. Only the piano part was played by myself, the entire bass and percussion part was playe by synthesizers, driven by the presto® application via MIDI messages. During the production of this composition (Synthesis is a four-part, 45-minute piece), I never felt inhibited in my piano playing, in the contrary, it was a great pleasure to collaborate with complex structures of rhythm or melody, objects of a complexity that human percussionists would never be able to play from a score. 1:26

RUBATO® RUBATO L L S Os X

concept modeling GeneralNote Pause Note Onset Duration Onset Loudness Pitch – – – Ÿ STRG –

concept modeling MakroNote Satellites GeneralNote MakroNote Ornaments Schenker Analysis Satellites GeneralNote – Onset Loudness Duration Pitch Note STRG Ÿ Pause concept modeling MakroNote

form semiotics ◊ Den Forms Types E R Mono(E ) Diagrams (Den /E ) F-Name Den ◊ Forms Form Identifier D-Name form semiotics Type Diagram Coordinate Types E Mono(E ) Diagrams (Den /E ) R

frame spaces representation conjunction disjunction collection FrameSpace for type: Simple √ = ◊ ® R simple(√) = R representation frame spaces Limit √ = form-name diagram ® E limit(√) = lim(f. diagram ® E ) conjunction Colimit √ = form-name diagram ® E colimit(√) = colim(f. diagram ® E ) disjunction Power √ = fn ® E power(√) = WSpace(fn) collection

Babouska Hjelmslev‘s Babouska Principle Expression Signification Content Babouska Expression Signification Content Expression Signification Content Expression Signification Content Connotation Motivation Meta system

form semiotics Denotator The semiotic structure (Hjelmslev stratification) of denotators: The triadic sign structure (signifier/signification/signified) reads from left to right Denotator form semiotics Name Form Coordinate Topos Name Type/Diagram Identifier

patent Denote Render MakroNote Satellites GeneralNote – Onset Loudness Duration Pitch Note STRG Ÿ Pause

multimedia science OBJECTS manipulate operate denote CONCEPTS render parametrize objectivize conceive fold multimedia science VECTORS transform

OBJECTS CONCEPTS render objectivize fold multimedia science VECTORS

CONCEPTS fold multimedia science VECTORS

folding

folding x x = (x1,x2,x3,...,xn) n-cube x* = 0.x1x2x3...xn lexicographic ordering quantize: xi = 0,1,2,3,...,d real axis x* common ordering

OBJECTS objectivize multimedia science VECTORS

pinocchio

books

Movie of Stefan Göller‘s PrimavistaBrowser movie Workflow 500 Pinocchios Animates

satellites Sat Satlist Sat Satlist Geo RTC —2 —*3 —2 —*3 —2 —*3 —2 —*3 1 satellites RTC URL ETC Terminator Time —2 —*3 Trans Time —2 —*3 Rot Time —2 —*3 Scale Time —2 —*3 Color Time —2 —*4 Sound Time —2 —* Tex

manipulate OBJECTS manipulate objects

cockpit - espresso

cockpit - espresso

cockpit - espresso

The EuroCODE Architecture Mandatory Tools Challenging Prima- Vista- Browser Info- Codette Codette- Factory Cockpit- Factory XML 2 CODE Mus-e-Lear-ning AMO-Codette Data-Scape Etc. Framework: Denotator, Form, Topoi (Ring, Module, etc.) Diagram, ... LoGeo class library: Logical and geometric operations for predicate construction and manipulation search algorithms Cockpit 1 Cockpit 2

red&t The EuroCODE R&D Strategy = RED&T Generic tools Deliver Challenging Components Test LoGeo Extend XML 2 CODE AMO Codette = PredicateBuilder (Processes) Codette- Factory Mus-e- Lear-ning Data- Scape = StructureBuilder (Architecture) Etc. Cockpit- Factory = InterfaceBuilder (Interaction) Request

workpackages 11 partners: 5 Universities and research centers 1 foundation 1 big company 4 SMEs 15 WP, 122 DEL 3 years, 5.8 M€ ~ 926 MM 0.3 M€ management management-oriented packages WORK 1 (project management) WORK 2 (communication management) WORK 3 (scientific coordination) WORK 15 (transfer management) generic workpackages WORK 4 (Framework) WORK 5 (PredicateBuilder) WORK 6 (PrimavistaBrowser) WORK 7 (CodetteFactory) WORK 8 (CockpitFactory) challenging component targeted packages WORK 9 (XML2CODE) WORK 10 (AMOCodette) WORK 11 (DataScape) WORK 12 (TerrainDataView) WORK 13 (KnowledgeGate) WORK 14 (Mus-e-Learning) workpackages

management General Public EU Commission Steering Board (with representa- tives from each partner) Project Manager Administration Comunication Scientific Coordination Transfer Management WORK 1 WORK 2 WORK 3 WORK 15

Statistik zur 1. IST-Ausschreibung Die erste IST-Ausschreibung im 6 Statistik zur 1. IST-Ausschreibung Die erste IST-Ausschreibung im 6. Forschungsrahmenprogramm endete am 24. April 2003. Insgesamt sind 1473 Projektanträge eingereicht worden, und zwar 77,6% als Papierversion und 22,4% per elektronischer Einreichung. Ein erster statistischer Überblick über die Anzahl der eingereichten Projektanträge - aufgeschlüsselt nach den einzelnen strategischen Zielen aus dem IST-Arbeitsprogramm und mit Verteilung auf die einzelnen Instrumente (IP, NoE, STREP und andere: CA, SSA): 2.3.1.1 CMOS und Post-CMOS: 53 gesamt (11 IPs, 8 NoEs, 29 STREPs, 5 andere) 2.3.1.2 Mikro-, Nanosysteme: 82 gesamt (25 IPs, 14 NoEs, 36 STREPs, 7 andere) 2.3.1.3 Breitband für alle: 80 gesamt (12 IPs, 8 NoEs, 45 STREPs, 15 andere) 2.3.1.4 Mobile drahtlose Systeme: 92 gesamt (15 IPs, 15 NoEs, 50 STREPs, 12 andere) 2.3.1.5 Zuverlässigkeit, Sicherheit: 93 gesamt (23 IPs, 8 NoEs, 55 STREPs, 7 andere) 2.3.1.6 Multimodale Schnittstellen: 74 gesamt (23 IPs, 10 NoEs, 36 STREPs, 5 andere) 2.3.1.7 Wissenssysteme: 129 gesamt (18 IPs, 21 NoEs, 80 STREPs, 10 andere) 2.3.1.8 audiov. Syst., Heimplattformen: 78 gesamt (18 IPs, 7 NoEs, 48 STREPs, 5 andere) 2.3.1.9 eBusiness, eGovernment: 248 gesamt (59 IPs, 19 NoEs, 116 STREPs, 54 andere) 2.3.1.10 eSafety: 53 gesamt (11 IPs, 3 NoEs, 35 STREPs, 4 andere) 2.3.1.11 eHealth: 170 gesamt (47 IPs, 20 NoEs, 89 STREPs, 14 andere) 2.3.1.12 eLearning, Kultur: 217 gesamt (54 IPs, 27 NoEs, 115 STREPs, 21 andere) 2.3.4.2 FET vorausschauend: 30 gesamt (25 IPs, 4 NoEs, 1 STREP) 2.3.6 Begleitmaßnahmen: 35 gesamt (4 CAs, 31 SSAs) Summe: 1473 gesamt (348 IPs, 169 NoEs, 755 STREPs, 58 CAs, 138 SSAs, 5?) Die budgetmäßige Überzeichnung (Verhältnis von beantragtem Budget zu verfügbarem Budget) variiert in den einzelnen strategischen Zielen und liegt zwischen 5:1 und 20:1. Gemeinsame Ausschreibung mit Priorität 3 2.3.3.1 Produkte und Dienste 2010: 92 gesamt (52 IPs, 18 NoEs, 9 CAs, 13 SSAs) Weiterer Zeitplan für die 1. IST-Ausschreibung: Die Europäische Kommission hat folgenden Zeitplan für die erste IST-Ausschreibung vorgelegt: 12. bis 23. Mai 2003: Begutachtung der Projektanträge 2. bis 13. Juni 2003: Anhörungen (hearings) zu den erfolgreichen (über Schwellwerten liegend) IPs und NoEs 26. Kalenderwoche (=nächste Woche): Projektkoordinator erhält Bewertungsbericht (Evaluation Summary Report) und Begutachtungsbericht geht an das IST Committee. Ab diesem Zeitpunkt können wir Ihnen nähere Auskünfte zu Ihren Projektanträgen geben (Anfragen bitte per eMail an eu-ncp@dlr.de mit Angaben zu Antrags-Nr., Akronym, angesprochener IST Bereich 'strategic objective', Name des Antragstellers) 9.-10. Juli 2003: Bilaterale Gespräche zwischen Vertretern der Mitgliedstaaten (IST Committee) und Europäischer Kommission 25. Juli 2003: Endgültige Liste der geförderten Projekte (implementation plan) statistics

Schedule I 15 March Consortium Meeting 17 March Request for login and password of coordinator; necessary for all electronic proposal submission acti- vities 18 March Pre-registration (electronic) Need: Acronym, title, abstract, consortium members, some obvious code and key words 21-23 March Travel Mazzola to IRCAM for negotiations End of March First Draft of proposal; submission to EU site: visibility for all consortium partners 14-17 April Travel to Bonn and Osnabrück

Schedule II 14 April Revised Draft; check with national contact point for formal correctness 15 April Final (electronic) submission of proposal 24 April, 17:00 End of submission period 8 May Acknowlegment of receipt June Evaluation report From July Negotiation meetings in Bruselles From October Approval of funding

Evaluation Criteria II Quality of consortium high quality committed and well-suited partners complementarity SMEs adequately addressed Quality of management Project management of high quality Management of knowledge, intellectual property, and innovative activities Mobilisation of resources Resourses foreseen Resources integrated -> coherence Financial plan adequate

Submission form Parts A & B Part A A1: Proposal Nr, acronym, title, duration, call identifier, activity codes, keyword codes, free keywords, abstract (max. 2000 char.) A2: Participant portraits (9): Name of organization, address person in charge, phone number, email, internet homepage A3: Financial information: cost model of participants, costs, requested grant to budget, split among RTD, demonstration, management activities

Submission form Parts A & B Part B Front page: basic coordinates of proposal Contents page: contents list Proposal summary page B.1: Scientific and technological objectives of the project and state-of-the-art (3pp) B.2: relevance to the objectives of the IST Priority (3pp) B.3: Potential Impact (3pp) B.3.1: Contributions to standards B.4: The consortium and project resources (6pp) B.5: Project management (3pp) B.6: Workplan (15pp) —>… B.7: Other issues (ethical and gender issues, other EU issues)

Overview of plan methodology Part B.6: Workplan Overview of plan methodology Gantt chart of timing of different WPs (=work packages) Pert (=Program Evaluation Review Technique) diagram of components Detailed work description broken down into WPs: WP list (with: Nr., title, person months, start, end) Deliverable list (with: Nr., title, date, nature, dissemination level: public, restricted, confidential) Description of each WP (objectives, description, deliverables, milestones)

^ * FMT U Osnabrück IRCAM Paris Kom&Alg U Bonn LIM U Milano MML U Zürich fabric Lausanne finebrain Basel ViewTec Zürich IGD FhG CAPcom INI-GraphicsNet Foundation Darmstadt ^ Partner Task > Module Diagram Functor Denotator Form Satellite Folding LoGeo Retrieval Cockpit Data-scapes DBMS XML etc. integration Knowledge transfer Project Manage-ment

Provisional Workpackages IRCAM Paris (IR) m/m IR1.1 programming issues 16 IR1.2 cognitive/logical models 16 IR1.3 algebraic models 16 IR1.4 integration of concepts 10 IR1.5 denotator import/export 6 IR2.1 authoring module 10 IR2.2 GUI module 8 IR2.3 sound tools 10 FABRIC|CH Lausanne (FA) FA1 project management 3 FA2 state-of-the-art 5 FA3 concepts + tech. choice framework development 8 FA4 tests of framework 4 FA5 application development 8 FA6 dissemination 6 FA7 results assessment & risk management 1 FMT Osnabrück (FM) FM1 project management 36 FM2.1 MusEducator concepts 12 FM2.2 MusEducator implementation 12 FM2.3 MusEducator product 12 FM3.1 KnowledgeGate FM3.2 KnowledgeGate FM3.3 KnowledgeGate FM4.1 XML2CODE design 12 FM4.2 XML2CODE LIM Milano (LM) m/m LM1.1 XML2CODE Design 12 LM1.2 XML2CODE implementation 12 LM2.1 SPINE (Performance) concepts 12 LM2.2 SPINE implementation 12 LM3.1 Symbolic & Audio design 12 LM3.2 Symbolic & Audio 12 implementation 12 MML Zürich (MM) MM1 PrimavistaBrowser & InfoCodette 10 MM2 denotator and topos framework 10 MM3 LoGeo library 40 MM4 CodetteFactory 20 MM5 CockpitFactory 20 MM6 scientific coordination 36 FINEBRAIN Basel (FI) FI1 requirement analysis 24 FI2 search space concepts 24 FI3 transfer to clients 24 VIEWTEC Zürich (VT) m/m VT1.1 satellites for GIS general concepts 10 VT1.2 satellites for GIS design 18 VT2.1 TerrainDataView implementation 18 VT2.2 TerrainDataView product 18 K&A Bonn (KA) KA1 algorithms for denotator search 20 KA2 DenoBMS concepts 20 KA3 implementation of KA1 20 KA4 performance tests 12 INI Darmstadt (IN) IN1 transfer management 18

&T Time LoGeo (ProcessBuilder) PRODUCT IMPLEMENTATION DESIGN Framework (Foundation Classes) LoGeo (ProcessBuilder) Prima- Vista- Browser Info- Codette Codette- Factory (Structure- Builder) Cockpit- Factory (Interface- XML 2 CODE AMO Engine Data-Scape Know- ledge-Gate &T TerrainData View MusEdu-cator IN1 FA1 FA7 FM1 FA6 PRODUCT FI3 KA4 IR2.3 FM2.3 MM1 MM2 IR2.2 VT2.2 FM3.3 IR2.1 MM6 VT2.1 IMPLEMENTATION LM3.2 LM2.2 KA3 FM4.2 IR1.1 FM2.2 MM3 FA4 LM1.2 FM3.2 IR1.4 FA3 FI2 LM2.1 FI1 MM4 MM5 FA5 LM3.1 FM4.1 DESIGN IR1.2 FM2.1 KA2 VT1.1 LM1.1 FM3.1 VT1.2 IR1.3 KA1 FA2 Time

&T Time LoGeo (ProcessBuilder) Framework (Foundation Classes) XML 2 CODE AMO Engine Data-Scape Know- ledge-Gate Framework (Foundation Classes) LoGeo (ProcessBuilder) Prima- Vista- Browser Info- Codette Codette- Factory (Structure- Builder) Cockpit- Factory (Interface- MusEdu-cator Time DESIGN IMPLEMENTATION PRODUCT &T IN1 FA1 FA7 FM1 FA6 FI3 KA4 IR2.3 FM2.3 MM1 MM2 IR2.2 VT2.2 FM3.3 IR2.1 MM6 VT2.1 LM3.2 LM2.2 KA3 FM4.2 IR1.1 FM2.2 MM3 FA4 LM1.2 FM3.2 IR1.4 FA3 FI2 LM2.1 FI1 FA5 MM4 MM5 LM3.1 FM4.1 IR1.2 FM2.1 KA2 VT1.1 LM1.1 VT1.2 FM3.1 IR1.3 KA1 FA2

Consortium Size Scale of activities Duration Participants at least 3 independent legal entities (9+2) In at least 3 different Member States or Associate States (D,F,I,CH) Scale of activities Up to several Mega-€ (?) from a total of 3.625 Mega-€ for IST (55 Mega-€ for semantic-based knowledge systems, estimated 4-5 STREPs for 2-3 Mega-€ each) Costs are calculated from local rates per partner, no global presets! Essentially 2 cost models: FC = Full costs and AC = Additional costs AC = option for universities and research institutes (100% of add. costs) FC = option for SMEs (50% of full costs) Duration 2-3 years, or more if duly justified (3) Participants Universities (4) Research institutes (1+IGD) SMEs (3+CAPcom AG) Knowledge transfer organizations (1) Potential users and stakeholders (0 +IGD +CAPcom AG)

Evaluation Criteria I Relevance to the objective of the program Scientific Technical Socio-economic (the EuroCODE platform is mandatory and essential for the construction of the kowledge society) Political (strategic fostering of open European systems & SW) Specified in the work packages (=WPs) Scientific and technological excellence Defined and well-focused objective Clear progress beyond state-of-the-art Proposed S&T approach adequate to target Potential impact Impact on reinforcing competitiveness Added value of European level (against national) Exploitation and dissemination ensure optimal use of results

Distributed Rubato Library at IRCAM Audio Studio Workstation MeloTop

XML EuroCODE xml document type definition (.dtd) xml document (.xml) stylesheet (.xsl) form (.dtx) @form denotator (.dtx) @substance satellite (.dtx) <?xml version="1.0"?> <!DOCTYPE BOOK [ <!ELEMENT BOOK (P*)> <!ELEMENT P (#PCDATA)> ]> <BOOK> <P>chapter 1 - Intro</P> <P>chapter 2 - Conclusion</P> <P>Index</P> </BOOK> Personal:.Limit[Leaders, Staff]; Leaders:.Power{ProjectPerson}; Staff:.Power{ProjectPerson}; Resources:.Limit[Budget, Financing, Hardware, Processes]; Budget:.Power{Document}; Financing:.Power{Document}; Hardware:.Power{Document}; Processes:.Power{Document};

EuroCODE XML vs. xml documents object configurations character data content model types ad hoc rendering via XSL object configurations topoi all categorical types universal recursive multi- media representation (PCT patent)