Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 co-funded by the European Commission Final review September 16th 2008.

Similar presentations


Presentation on theme: "1 co-funded by the European Commission Final review September 16th 2008."— Presentation transcript:

1 1 co-funded by the European Commission Final review September 16th 2008

2 2 Primary goal is the detection of audio treasures in distributed archives. Primary goal is the detection of audio treasures in distributed archives. DISMARC created the tools to achieve this goal. DISMARC created the tools to achieve this goal. Via the DISMARC portal European cultural music archives become content-partners Via the DISMARC portal European cultural music archives become content-partners DISMARC offers a means of integrating audio into EDL/EUROPEANA. DISMARC offers a means of integrating audio into EDL/EUROPEANA. OVERVIEW

3 3 How DISMARC opens European archives to the world Built on the concept of DISMARC as service provider Built on the concept of DISMARC as service provider No altering of characteristics, content and structures of individual archives No altering of characteristics, content and structures of individual archives Creating inexpensive solutions Creating inexpensive solutions Fulfilling (some of) archives most ambitious dreams… Fulfilling (some of) archives most ambitious dreams…

4 4 Wish-list features: To become owner of an online catalogue, which is simple, visible immediately, supported technically by third party, minimal impact on archives budget To become owner of an online catalogue, which is simple, visible immediately, supported technically by third party, minimal impact on archives budget To use an international data standard without being forced to change own standards and routines To use an international data standard without being forced to change own standards and routines To migrate legacy catalogues to current standards To migrate legacy catalogues to current standards To be able to search across archives and to be searched in context of other catalogues by marrying distributed catalogues electronically in one database To be able to search across archives and to be searched in context of other catalogues by marrying distributed catalogues electronically in one databaseand…

5 5 Features contd. To have the possibility of multilingual searches and even multilingual free-text-search in own catalogue and across distributed archives To have the possibility of multilingual searches and even multilingual free-text-search in own catalogue and across distributed archives To be part of a portal that is dedicated particularly to the content of cultural music-audio archives To be part of a portal that is dedicated particularly to the content of cultural music-audio archives To create online access to the music itself with a simple solution (an audio server and search & retrieval tools) To create online access to the music itself with a simple solution (an audio server and search & retrieval tools) Also: service is free and with no obligation. Also: service is free and with no obligation.

6 6 New individual possibilities for archives: Supply improved service to the public without extending office hours Supply improved service to the public without extending office hours Enable exchange of audio, generate income Enable exchange of audio, generate income Present own music holdings in the context of other catalogues Present own music holdings in the context of other catalogues Establish sub-networks (i.e. European broadcaster catalogue) Establish sub-networks (i.e. European broadcaster catalogue)...and more...and more

7 7 Finally: Music archives are offered the chance to migrate to the online world in a European-oriented framework Music archives are offered the chance to migrate to the online world in a European-oriented framework This electronic music archive is a new species of cultural archive, a knowledge base, built on distributed content This electronic music archive is a new species of cultural archive, a knowledge base, built on distributed content DISMARC creates a powerful partner for other online services, in particular EUROPEANA DISMARC creates a powerful partner for other online services, in particular EUROPEANA It is obvious that this European cooperation creates advantages for individual archives It is obvious that this European cooperation creates advantages for individual archives

8 8 Results achieved Metadata template D3b Metadata template D3b Controlled Vocabularies Controlled Vocabularies Multilinguality (portal) Multilinguality (portal) Multilingual searches Multilingual searches DISMARC Portal DISMARC Portal Current state of content collection Current state of content collection Worldmap – implementation in VLE Worldmap – implementation in VLE DISMARC on a Stick DISMARC on a Stick Broadcasts Broadcasts Seminars Seminars Virtual CDs Virtual CDs DVD / online DVD DVD / online DVD IPR tool IPR tool

9 9 Demo Included Portal – screenshots & live demonstration Portal – screenshots & live demonstration Front end VS back end Front end VS back end 2 templates 2 templates DISMARC Archive

10 10 Metadata template Mainly developed in first period in collaboration among the archives Mainly developed in first period in collaboration among the archives AP based on Dublin Core, compatability with Europeana AP based on Dublin Core, compatability with Europeana Metadata template is also implicitly in many portal functions, e.g. the search categories etc. Metadata template is also implicitly in many portal functions, e.g. the search categories etc. Metadata Structures Manual Metadata Structures Manual D3b (M18), D3b (M18), Constantely updated on web: now v63 Constantely updated on web: now v63 Access through portal Access through portal

11 11 Accessing the Manual Deliverable 3 bCurrent Web Version Included in the Portals Help Wiki

12 12 Controlled Vocabularies where possible: based on existing standards, e.g. DCMIType, AFSs Ethnographic Thesaurus, MARCREL etc. where possible: based on existing standards, e.g. DCMIType, AFSs Ethnographic Thesaurus, MARCREL etc. Simple vocabularies developed and maintained by DISMARC, e.g. dmFormats Simple vocabularies developed and maintained by DISMARC, e.g. dmFormats Accessible through manual, back end and partly on the front end Accessible through manual, back end and partly on the front end

13 13 Controlled Vocabularies For end users as search features in portal For end users as search features in portal Look up definitions in the manual Look up definitions in the manualmanual in the back end of the portal, usually for archives in the back end of the portal, usually for archives

14 14 Multilinguality (portal) Usual functionality on many multilingual websites today: Usual functionality on many multilingual websites today: translates portal, but not the data translates portal, but not the data

15 15 Multilingual searches Translations in 30+ languages Concept As simple as possible, as complex as necessary (not only words, but phrases; synonyms etc.) own translations Example: herehere

16 16 Multilingual searches: Behind the scenes Back end: Login/Admin area/Dictionary Back end: Login/Admin area/Dictionary Overview and single term as example: Overview and single term as example: Translation integrated in workflow Translation integrated in workflow

17 17 DISMARC Network DISMARC Network Harvester (Metastore) Provider (DISMARC node) Other partners (EUROPEANA) Subproviders User

18 18 The various DISMARC nodes provide the data for the DISMARC central node harvester The various DISMARC nodes provide the data for the DISMARC central node harvester The central node itself also functions as a provider and can be harvested by other data portals like EUROPEANA The central node itself also functions as a provider and can be harvested by other data portals like EUROPEANA The rights free sound samples may be downloaded via a link in the metadata and are stored on the DISMARC Audio Server The rights free sound samples may be downloaded via a link in the metadata and are stored on the DISMARC Audio Server DISMARC Network DISMARC Network

19 19 The DISMARC central metastore provides powerful search capabilities for searching the diverse data The DISMARC central metastore provides powerful search capabilities for searching the diverse data No need to know how to query various individual systems No need to know how to query various individual systems New metadata is harvested automatically in pre-defined intervals (each night) New metadata is harvested automatically in pre-defined intervals (each night) Advantages Advantages

20 20 Partners may either: Partners may either: Have their local OAI provider Have their local OAI provider own (indicate the URL for the DISMARC Harvester)own (indicate the URL for the DISMARC Harvester) DISMARC installation (stick)DISMARC installation (stick) OAI provider hosted at the DISMARC central site OAI provider hosted at the DISMARC central site Data import via FTPData import via FTP How to connect to DISMARC How to connect to DISMARC

21 21 The DISMARC local providers DISMARC node lifecycle Installation Task 2.3 Mapping Task 2.4 Import Task 2.3 OAI Task 2.5 Impr. Task 2.8

22 22 Installation DISMARC node lifecycle Mapping Task 2.4 Import Task 2.3 OAI Task 2.5 Impr. Task 2.8 Installation Task 2.3

23 23 Prerequisites (D2.1/2.2) Iconv is a unix tool to convert the encoded import files to MDMs internal utf-8 encoding. Every common *nix distribution ships with iconv, whereas the support for MS Windows is supplied by cygwin. Iconv Zebra Server supplies the core functionality for the collection and object metadata. The Zebra Server is used to index and retrieve the generated xml records. Zebra Server to the need of a relational database as user and setting storage and the native support of MySQL5 by PHP, MySQL5 was the choice for MDM. Nevertheless, also the use of a Postgres database is possible MySQL 5 MDM is written as object oriented PHP application. Therefore the use of the latest PHP version is recommended. PHP 5.2 Since the MDM is a web application, a webserver is needed. There are no special preferences for its implementation Apache/IIS

24 24 Download and Extract Extract Webserver directory

25 25 Mapping DISMARC node lifecycle Installation Task 2.3 Import Task 2.3 OAI Task 2.5 Impr. Task 2.8 Mapping Task 2.4

26 26 Mapping concept Native data Normalisation & Enrichment Harvest & Indexation DISMARC Metastore Zebra IDX Application Presentation MDM

27 27 Mapping tool Connected to the currently used DISMARC application profile Connected to the currently used DISMARC application profile Generates: Generates: A basic mapper written in PHP A basic mapper written in PHP A collection description A collection description A table for further usage A table for further usage Sends this to the responsible personnel Sends this to the responsible personnel

28 28 Mapping tool

29 29 Mapping tool

30 30 Mapping tool

31 31 Mapping - Structure List of Importer Reader (provided) Mapper (via mail)

32 32 Import DISMARC node lifecycle Installation Task 2.3 Mapping Task 2.4 OAI Task 2.5 Impr. Task 2.8 Import Task 2.3

33 33 Audio DB – Network Submit data Metastore Reference data Use data DISMARC partner DISMARC users

34 34 Audio database Audio is stored at external web space Audio is stored at external web space Partners upload their data themselves Partners upload their data themselves ftp://ftp.dismarc-audio.org ftp://ftp.dismarc-audio.org ftp://ftp.dismarc-audio.org Username and password are provided separately Username and password are provided separately Each partner has own partition for safety reasons Each partner has own partition for safety reasons

35 35 FTP administration Users have been created

36 36 Import / Technical DISMARC node lifecycle Mapping Task 2.4 OAI Task 2.5 Impr. Task 2.8 Import Task 2.3 Installation Task 2.3

37 37 Data transformation Two basic transformation options Option 1: Direct transformation Option 2: The substitution list

38 38 Data transformation Two basic transformation options Option 1: Direct transformation Option 2: The substitution list

39 39 Direct Transformation

40 40 Native datasets

41 41 Direct usage of values

42 42 The result

43 43 Data transformation Two basic transformation options Option 1: Direct transformation Option 2: The substitution list

44 44 The substitution list

45 45 Native data and list

46 46 File structure

47 47 Look up the values

48 48 The result

49 49 The substitution list Can also be used to ADD data Can also be used to ADD data Just add the required fields and values to the list Just add the required fields and values to the list Add these new data to the mapped document within the mapper Add these new data to the mapped document within the mapper You are ready to feed GMEs tool You are ready to feed GMEs tool

50 50 Automated import First, new native data has to be transferred to the MetaStore via FTP Archive

51 51 Automated import Periodically iterates through the configured importer Starts automated importjob periodically Iterates through the active importer

52 52 Automated import For each importer, the new files directory is checked and each file will be passed to the Importer's reader Importer's mapper moves the native values in place, adds thesaurus terms and replaces codes to create dismarc records Each new file will be passed to the reader Importer's reader iterates through each native record in the new file and generates the raw data as xml

53 53 OAI OAI DISMARC node lifecycle Mapping Task 2.4 Import Task 2.3 Impr. Task 2.8 OAI Task 2.5 Installation Task 2.3

54 54 The provider Each partner may have a DISMARC node Each partner may have a DISMARC node The providers are located at the oai subdirectory The providers are located at the oai subdirectory

55 55 The harvester The harvester runs periodically The harvester runs periodically Checks its list of known providers Checks its list of known providers Requests a list of new data Requests a list of new data Collects the data and stores it in its own database Collects the data and stores it in its own database New data is then used in the central provider for EUROPEANA New data is then used in the central provider for EUROPEANA

56 56 List of known providers

57 57 One harvested URL Shall it be harvested at the next possible turn From where Which importer shall be used For which archive is it Cron settings Status reports

58 58 The front end and back end tools and infrastructure of the DISMARC system are described in Deliverable 2.2 (Technical Report) The front end and back end tools and infrastructure of the DISMARC system are described in Deliverable 2.2 (Technical Report) All test results are described in a Validation report which is part of D5.1 All test results are described in a Validation report which is part of D5.1 Reporting

59 59 DISMARC - EUROPEANA DISMARC is perceived as the audio content aggregator for the European Digital Library – the EUROPEANA portal. DISMARC is perceived as the audio content aggregator for the European Digital Library – the EUROPEANA portal. First test harvests have been conducted in early summer First test harvests have been conducted in early summer 2008.

60 60 Portal Status Portal Status Data to date ( ) Data to date ( ) 1.3 Mio data objects (items) 1.3 Mio data objects (items) This number is constantly increasing as several auto import routines have been installed that allow partners to import data themselves. (new data import each night) This number is constantly increasing as several auto import routines have been installed that allow partners to import data themselves. (new data import each night) object types: sound, data set, moving image, still image, text object types: sound, data set, moving image, still image, text Audio Samples: Audio Samples: data records carry altogether audio samples (some records carry more than one sample – eg. several tracks for one album) data records carry altogether audio samples (some records carry more than one sample – eg. several tracks for one album)

61 61 Autoimport routines are active for: Portal Status Portal Status - Importer Default EMEM xml importer [EMEM_XML_TO_DMOAP]: - Importer Default RBB MUSAD xml importer [RBB_MUSAD_XML_TO_DMOAP]: - Importer Default RBB MultiKulti xml importer [RBB_MULTIKULTI_XML_TO_DMOAP]: - Importer Default YLE TEHO (sound effects) importer [YLE_TEHO_TO_DMOAP]: - Importer Default YLE FONO (sound recordings) importer [YLE_FONO_TO_DMOAP]: - Importer Default SVA Excel importer [SVA_CSV_TO_DMOAP]: - Importer OAI MDM Importer [OAI_MDM]: - Importer Default OEM importer [OEM_XML]: - Importer Default SOAS file importer [SOAS_PLAIN]: - Importer Default KKA DB importer [KKA_DB]: - Importer ISPAN default ExcelXML parser (Sygn.) [ISPAN_EXCELXML_TO_DMOAP]: - Importer GMC's MARC Reader [GMC_MARC]: - Importer WOMEX Database Mapper [WOMEX_DB]: - Importer GMC OpenOffice FODS XML importer [GMC_FODS]: - Importer MGR Default CVS importer [MGR_CSV_TO_DMOAP]: - Importer TOM Database to DMOAP [TOM_DBXML]: - Importer HMTH DublinCore XML [HMTH_XML_TO_DMOAP]:

62 62 Usage of the portal: Monthly Statistics for September 2008 ( )> Hits: /daily Visits: >300/daily Monthly Statistics for September 2008 ( )> Hits: /daily Visits: >300/daily One to two users register to the platform each day One to two users register to the platform each day To date we have around 90 registrants to the platform, from 12 countries (AT, DE, DK, ES, FI, IE, LT, NO, PL, SE, UK, US) To date we have around 90 registrants to the platform, from 12 countries (AT, DE, DK, ES, FI, IE, LT, NO, PL, SE, UK, US) Portal Status Portal Status

63 63 Participants to DISMARC: 1 Prototype: 12 partner archives with altogether 20 collections are included 1 Prototype: 12 partner archives with altogether 20 collections are included 2 Test portal: 7 partner archives with altogether 12 collections are included 2 Test portal: 7 partner archives with altogether 12 collections are included 3 Queuing (Mapping and test import in process): 14 partners with altogether 21 collections 3 Queuing (Mapping and test import in process): 14 partners with altogether 21 collections 4 Collections without object metadata: 42 collection descriptions included 4 Collections without object metadata: 42 collection descriptions included SUM of partners cooperating to date with DISMARC (points 1 to 4): SUM of partners cooperating to date with DISMARC (points 1 to 4): 75 partners from 18 countries 75 partners from 18 countries (AT, DE, DK, EE, FI, GR, HU, LI, LV, NO, PL, RU, SE, SK, SL, UA, UK, US) Portal Status Portal Status

64 64 Type of content organisations cooperating with DISMARC: Public organisations: Public organisations: Music Archives, Music Centres, Broadcasting Companies, Libraries, Museums, Universities, Academies of Sciences Music Archives, Music Centres, Broadcasting Companies, Libraries, Museums, Universities, Academies of Sciences Private organisations: Private organisations: Music Societies, Music Archives, Music Conferences, Folk Institutes Music Societies, Music Archives, Music Conferences, Folk Institutes Private collectors Private collectors Portal Status Portal Status

65 65 Searching the DISMARC world map is a visual way to find audios in the DISMARC database. Searching the DISMARC world map is a visual way to find audios in the DISMARC database. World Map World Map

66 66 Via a click on the areas/countries that provide sounds the user zooms in and gets to the audio files. Via a click on the areas/countries that provide sounds the user zooms in and gets to the audio files. World Map World Map

67 67 Further results: broadcasts Positive responses from EBU European Broadcasting Union) workshop in Seville, October 2007 Positive responses from EBU European Broadcasting Union) workshop in Seville, October 2007 Support from the EBU-radio-director in November 2007 Support from the EBU-radio-director in November 2007 Three broadcasts of DISMARC-audio in RBB (August 2008) Three broadcasts of DISMARC-audio in RBB (August 2008) Two broadcasts on Radio Flora, Hanover by HMTH (April & July 2008) Two broadcasts on Radio Flora, Hanover by HMTH (April & July 2008) Two broadcasts by YLE in winter 2008 Two broadcasts by YLE in winter 2008

68 68 Various seminars… 'DISMARC and the Swedish Music Community / Sweden /SVA DISMARC structure, Adam Mickiewicz University / Poland / ISPAN Musical Futures: Copyright and Archives / England / SOAS/RBB History of music archives, Adam Mickiewicz University / Poland / ISPAN Management of Cultural Heritage Assets / Germany / EMEM Archive seminar Norwegian Folk Music Archives / Norway / YLE Sound Archives: Yesterday, Today and Tomorrow conference / Poland / ISPAN Karls-University / Czech Republic / ISPAN Presentation to Wissenschaftskolleg Ways of Knowledge / Germany / EMEM

69 69 more seminars DISMARC overview in European Seminar in Ethnomusicology bulletin (print) / EMEM Historical Development of Ethnomusicology, presentation of DISMARC at Carl von Ossietzky University of Oldenburg / Germany / HTMH From the wax cylinder to the iPod: Forms, history, and meaning of musical archives /HMTH in coop. with EMEM, 2007 Teaching the music teachers began September semester 2007 /HMTH To whom does music belong? Music as intellectual property - GEMA, archives and internet downloads /HMTH, winter 2007 iPod&Co – Digital Archives and Access to Music /EMEM 2007 Cologne Music Cultures on Record / EMEM / Berlin 2007 Music Cultures on Record / EMEM / Berlin 2007

70 70 DVD Skip if LAN present

71 71 Further results jt

72 72

73 73

74 74 Virtual CDsCDs Skip if LAN present

75 75

76 76

77 77 IPR tools IPR basket as demonstrated in a screencast for direct communication between users and archives IPR basket as demonstrated in a screencast for direct communication between users and archives screencast IPR licensing tool IPR licensing tooltool Dependent upon user requirements Dependent upon user requirements

78 78

79 79 Euro-vision DISMARC supports i2010 / Digital Libraries by cooperating closely with the EDL/EUROPEANA, DISMARC supports i2010 / Digital Libraries by cooperating closely with the EDL/EUROPEANA, by joining the EDL thematic network and by joining the EDL thematic network and by being the audio partner in proposed EUROPEANAconnect by being the audio partner in proposed EUROPEANAconnect

80 80 Performance indicators ait Indicators Period 4 (expected) Period 4 (actual) Processed records on DISMARC server 200, mil. ( more than 3,000 audio) Increase in accessGeneral public Increase in portal visits 20,00018, 670 (public access May 08, official launch Sept 2008 – figures for 4 months; September 08 – 257,000 hits. Project web site visits1,5005,460

81 81 MonthHits Page viewsVisits 1Mar 086,1403, Apr 0825,8475, Mai 0817,2265, Jun 0822,0027,4041,234 5Jul 0823,3547,1371,128 6Aug 0832,8718,5341,019 Total127,44038,0015,460 Website statistics

82 82 Challenges encountered many archives are relatively isolated, old databases, lack of IT expertise – supply of technical support archives are wary of public scrutiny; but involvement with DISMARC means visibility of metadata standards, contents, work practices in archives – that has been met by creating a test server new archives took time in sending data, signatures on Starter Kit (MoU) also took time – in answer support routines have been developed partner SOAS personnel changes – reallocation of effort

83 83 Dissemination Conferences, trade fairs, broadcasts, seminars, workshops, magazine and book publications, web dissemination, etc. Over 100 events, as notified herehere DISMARC and its impact are known throughout the cultural music archive world DISMARC and its impact are known throughout the cultural music archive world

84 84 User involvement & feedback mg Constant feedback from partners, two user groups created in second period general users and archive partners results presented in D 5.1 and D 5.1.2

85 85 User involvement & feedback mg current archive total: 75 (as of September 12th 2008) and further archives: +/-200 current portal stats: 194,512 hits September 2008 current registered visitors: +/- 100

86 86 September 08

87 87 Project website jt Skip if LAN present

88 88 Overview of project

89 89 Dissemination

90 90 Virtual CD pilot

91 91 Multi-linguality

92 92 Impact: changes & benefits jt Users can search for and find previously undiscoverable music and information archives integrated into digital world integration with i2010 Digital Libraries – without DISMARC, no audio WP in EUROPEANAconnect DISMARC effect – funding (ISPAN), data- restructuring (EMEM, South Africa, Kyrgyzstan), clarification of rights (RBB Reichsrundfunk Archive) and more…

93 93 Impact: changes & benefits jt side effects for partners: all partners archives now on-line for first time, metadata improved; development of IPR expertise; OAI / interoperability ISPAN and HMTH archives documented for first time; YLE sound effects library online for free use GME World Map music browser integrated as browse- portal (on-line)on-line

94 94 Exploitation, sustainability DEMAND: Run portal technically Assist archives to participate Ensure technical and data compliance with EDL Update of infrastructure SOLUTIONS: RBB funding server for 5 years AIT giving a hand to archives, asking for a fee or seeking funds for this service DISMARC-consortium members create an advisory board Three curators looking after compliance and development: RBB (general representation, marketing etc.), AIT (technical), EMEM (data, mappings)

95 95 DISMARC on a stick XAMPP (Webserver, PHP, MySQL) Zebra Server Operating system MetaDataManager

96 96 AIT Forschungsgesellschaft mbH Klosterwiesgasse 32/1 A-8010 Graz DISMARC co-funded by the European Commission DISMARC - Stick DISMARC - Stick 1. Installation of the VMware – Player Stored on the USB-stick (can also be downloaded from: ) Stored on the USB-stick (can also be downloaded from: ) 2. Running the DISMARC-System on a stick (or any other writable storage medium) using the VMware-Player

97 97 co-funded by the European Commission


Download ppt "1 co-funded by the European Commission Final review September 16th 2008."

Similar presentations


Ads by Google