Download presentation
Presentation is loading. Please wait.
Published byJulianna Bates Modified over 8 years ago
1
1 David Nathan Endangered Languages Archive SOAS University of London LingDy Feb 15, 2013 ELAR and Digital Archiving for Documentation of Endangered Languages
2
2 What is a digital language archive? a trusted repository created and maintained by an institution with a commitment to the long-term preservation of archived material has policies and processes for acquiring, cataloguing, preserving, disseminating, and migrating (updating formats) a platform for building and supporting relationships between data providers and data users
3
3 General archiving functions advise acquire preserve add value provide access develop trust
4
4 Why is language archiving different? what is a language? unlike business data, it is not conventionalised (like $, age, year of publication etc) – what and how to code? varying and competing expectations
5
5 And endangered languages archiving? extremely diverse context – languages, cultures, communities, individuals, projects typical source - fieldworkers typical materials - documentation difficult for archive staff to manage sensitivities and restrictions
6
6 What can a language archive offer? Security - keep your electronic materials safe Preservation - store your materials for the long term Discovery - help others to find out about your materials, and you to find out about users Protocols - respect and implement sensitivities, restrictions Sharing - share results of your work, if appropriate Acknowledgement - create citable acknowledgement Mobilisation - create usable language materials Quality and standards - advice for assuring your materials are of the highest quality and robust standards
7
7 There are different kinds of language archives from local to global - different coverage, contexts, methods, collection policies consider placing your materials in more than one … there are also sites for aggregating different archives’ holdings, eg Virtual Language Observatory, OLACVirtual Language ObservatoryOLAC
8
8 Why digital? preservation: digitisation is the only way that audio and video (non-symbolic material) can be preserved for the future … because it can be copied and transmitted with zero loss also good for cataloguing, sharing, dissemination, repurposing
9
9 Digital disadvantages digital data is fragile and ephemeral cost (human, equipment, maintenance) requires strategy and luck to get right preservation depends on file and data formats depend on tools and software some formats require particular software (can we archive the software?) formats: prefer standard, stable, open, explicit, long-lasting some materials may have to be ‘migrated’
10
10 What do depositors have to do? select and contact an archive prepare materials select structure suitable encodings and formats complete metadata, metadocumentation, agreements send materials to archive(s) work with archive during curation etc ongoing management, updating, dissemination
11
11 OAIS model OAIS archives define three types of ‘packages’ ingestion, archive, dissemination: ArchiveDissemination afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds IngestionProducersDesignated communities
12
12 ELAR - architecture reduced boundaries between depositors, users and archive: users add, update content; negotiate access Archive afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds afd_34 dfa dfadf fds fdafds & UsersProducers request give access contribute edit
13
13 Redefining the digital EL archive a platform for developing and conducting relationships between knowledge producers and knowledge users – a social networking archive level the playing field between researchers and community members/other stakeholders encourage, recognise and cater for diversity
14
14 Data management and archiving use good data management practices whether or not you plan to archive materials document decisions, steps, conventions, structures, encodings appropriate and conventional data encoding methods (e.g. Unicode) be explicit and consistent plan for flowing data, working with others, across different systems (cf Bird and Simons, ‘Seven Dimensions of Portability’) good data management practices will make a future archiving process easier and better
15
15 Users and potential users depositors – deposit, access or update materials speakers and their descendants other researchers - comparative/historical linguists, typologists, theoreticians, anthropologists, historians, musicologists etc etc other “stakeholders”, eg educationalists, funders journalists and the wider public
16
16 ELAR facts and figures archived collections: ~200 online (published) collections: 150 average collection size about 80 GB online data bundles: ~25,000 online bundles access: unrestricted 10,000, restricted 15,000 total number of files held: around 200,000 total volume of files held: around 10 TB registered users: ~800 annual number of website "hits": 230,000
17
17 ELAR facts and figures – users increasing number of community members, including Aleut (Canada), Tai-Ahom, Wadar (India), Burushaski (Pakistan), Serrano, Cahuilla, Arapaho (USA), Iraqi Jewish (Iraq), Saami (Finland), Wabena (Tanzania), Torwali (Pakistan), Hani, Bai (China), Irish comments: “I found your site while looking up my grandmother, and i found her on your site speaking our language. and i would love for my children her great grandchildren to hear our language coming from her". many interdisciplinary researchers, particularly archivists and anthropologists
18
18 Our task … to preserve and disseminate documentation of endangered languages
19
19 Why is this important? over 50% of the world’s 7000 languages: are endangered likely to cease to be spoken this century little or nothing known about the majority of them language documentations and the archives that support, preserve, and disseminate them, will become the means of transmission of many languages
20
20 A perfect storm? documentation methods expose sensitivities & vulnerabilities documentation performed by and for linguists and “others” “big data” – resources channeled to analysis, broader audiences “open data” – push for unmoderated access
21
21 Protocol the sensitivities and access restrictions associated with EL resources need to be discussed, collected and recorded in the field
22
22 Protocol and access control principles: granularity – file, bundle or collection access is a relation between object and user protocol values can be changed over time ELAR’s URCS system User Researcher Community member Subscriber
23
23 ELAR’s protocol values U – resource available to all registered users R – resource available to users registered as researchers C – resource available to users endorsed as members of relevant language community S – resource available to users who have been given individual access rights for that resource
24
24
25
25
26
26
27
27 User xx has just applied for access to restricted material in the deposit solega-107128. The following message was attached to the application: "Hi [depositor], Please delegate me for access to the material on Solegas." Subscription application: formal
28
28 This email is to inform you that user xx's application for access to restricted material in the deposit musgrave2007tulehu has just been approved. The depositor included the following note to the user: "The researcher is known to me personally and I know that his interest is legitimate." Subscription response: formal
29
29 User xx has just applied for access to restricted material in the deposit budd2008beirebo. The following message was attached to the application: "I'm xx. I like to learn Bislama language, but never heard what it sounds like. Am very curious " Subscription application: “curious”
30
30 User xx has just applied for access to restricted material in the deposit verstraete2010paman. The following message was attached to the application: "I am currently doing my masters in Linguistics and I'm researching on an endangered language in Malaysia. I would like to see a sample of the data from the fieldwork since I'm not use to this yet. I hope that I can gain more understanding in carrying out the fieldwork." Subscription application: establish credentials and reason
31
31 This email is to inform you that user xx's application for access to restricted material in the deposit verstraete2010paman has just been rejected. The depositor included the following note: "Dear xx, I am sorry we cannot give you access to this deposit. The Lamalama community has asked us to restrict access to community members. With best wishes, [depositor]" Subscription response: rejected, with reason
32
32 This email is to inform you that user xx’s application for access to restricted material in the deposit caballero2009raramuri has just been approved. The depositor included the following note to the user: "Please let me know if you're looking for any specific materials or if you have any questions." Subscription response: offering further help
33
33 This email is to inform you that user xx's application for access to restricted material in the deposit kunbarlang-389 has just been approved. The depositor included the following note to the user: "Hi xx I've approved your access to this collection, but you should know that there is an update in the material I've just deposited, with much more information on both music and texts. I'd be happy to give you access to that when it is processed. Next time I come to London (October or November this year) I'd be happy to meet up if you would like to discuss." Response: further info and offer to meet
34
34 What can you archive (at ELAR)? media - audio, video graphics - images, scans texts - fieldnotes, grammars, description, analysis structured data - aligned and annotated transcriptions, databases, lexica metadata, metadocumentation - contextual information about the materials, both structured and unstructured
35
35 Archive objects an “object” could be a file, a set of files, a directory, or a set of files with their relationships explicitly defined like other archives, ELAR uses a set principle, we call “bundles” (like DoBeS’ sessions) See bundles at ELAR
36
36 Archive objects ELAR Collection Bundle File
37
37 resource(s) for an endangered language it could be just one file catalogue / metadata deposit form viewview existing deposits can also be updated, added to, and metadata added/modified What is required to make a deposit?
38
38 Archive material should be selected example: Depositor’s question: How much video can I archive? answer:...
39
39 How can I deliver data? hard disks we return them we also send them out flash cards and USB sticks email good for samples for evaluation OK for most text materials Dropbox etc a web upload facility may be provided one day we can download from your server
40
40 What about CDs and DVDs? we have found CDs, and especially DVDs, to be very unreliable DVD fail rate > 10% cause confusion as files are allocated to fit on disks, not according to corpus structure create a lot of work for depositors and for ELAR
41
41 Express yourself - Metadata metadata is data about data containers data about data its functions for identification, management, retrieval of data provides the context and understanding of that data carries those understandings into the future, and to others
42
42 Express yourself - Metadata metadata reflects the knowledge and practices of data providers … and therefore defines and constrains audiences and usages for the data all value-adding to recordings of events (annotations transcriptions, translations, glosses, comments, interpretations, part of speech tagging etc) can be considered metadata data and metadata lie on a spectrum and depend on how they are used rather than being absolutely different things
43
43 Express yourself - Metadata distinguish between metadata scheme (eg set of categories) and the way that scheme is expressed
44
IDaudiotranscription 1TRS00065.wavbjt_02.txt 2TRS00066.wavkrs_43.txt TRS00065.wav bjt_02.txt TRS00066.wav krs_43.txt tagged relational filename: sessions.xls filename: sessions.xmlsessions.xml
45
45 Express yourself - Metadata example you could choose categories from OLAC, IMDI etc schemes or formulate your own this would be a scheme of logical categories (speaker, location, date etc) you could express these in different language(s) you could structure the categories and values in different ways, eg as spreadsheet, database, XML
46
46 Express yourself - Metadata you need to choose a set of metadata categories applying across whole collection + metadata categories that apply to particular types of objects (eg transcriptions, video), or to individual objects + ways of expressing and encoding all that metadata
47
47
48
48
49
49 Example Ju|’hoan (Biesele) Ju|’hoan (Biesele)
50
50 Potential sources of metadata deposit form spreadsheets MS Word tables, CSV etc IMDI and OLAC XML files custom XML notes, correspondence and reports filenames direct input to ELAR interface audio files images (/captions) meta-metadata files
51
51 A survey we collected information from about 50 ELAR deposits
52
About 80% of most frequently occurring categories can be mapped to OLAC 20language Subject.language 17dateDate 17descriptionDescription 16idIdentifier 16speakerContributor 16titleTitle 15formatFormat 13typeType 12creatorCreator 12file nameIdentifier 12notes 11rightsRights 10durationCoverage 9contentDescription 9contributorContributor 9nameContributor 9relationRelation 8age 8comment 8genreType.linguistic 8subject.language Subject.language 7date recordedDate 7document 1 7gender 7placeCoverage 6directoryIdentifier 5locationCoverage 5rec_dateDate 5recorderContributor termOLAC
53
53 Depositors also add categories such as: detailed locations metadata in Spanish indigenous genres and titles (eg of songs) parents’ and spouse’s mother tongues, birthplaces number of children, their language competence L2, L3 and competencies languages heard clan/moiety occupation education level
54
54 … more metadata: date left home country photos (/captions) of consultants, field sessions etc equipment microphone workflow status naming and organisational codes and principles recorder/linguist experience level biography and project description (“meta- documentation”)
55
55 What is the distribution?
56
56 Term frequencyNumber of terms 201 172 163 151 131 123 111 101 94 84 74 61 53 45 317 251 1613
57
57
58
58 A visualisation
59
59
60
60
61
61
62
62 Discussion and conclusions for endangered language documentation, the metadata framework is to be discovered, not predefined (cf Jeff Wallman, TBRC)
63
63 MD and resource discovery “discovery” is not neutral: what is emphasized/distilled? who gains? who does the work? MD is also about the distribution of labor and resources
64
64 MD and users MD is more responsible for the form, presentation, and usage of documentation than generally acknowledged MD should be equally accessible to and relevant for community members – it may even be more relevant to them than any “linguistic” data
65
65 OLAC: Open Language Archives Community: IMDI: ISLE Metadata Initiative more categories, software specific ELAR: for endangered language documentation, metadata framework is to be discovered, not predefined Common metadata standards Title Identifier Creator Contributor Language Subject.language Date Description Format Type Rights Coverage Relation
66
66 Types of metadata people metadata – creator’s / participants’ details descriptive metadata – content of data administrative metadata – eg. who did what when, relationships between objects, IPR and permissions structural metadata – how collection and its objects are organised, associated, formatted preservation metadata – character encoding, file format access and usage protocols
67
67 Examples example - XLS example example - XML example example – key example example – key XML example example – summary and requests example example - notes example
68
68 Meta-documentation Nathan (2010): “think of metadata as meta- documentation, the documentation of your data itself, and the conditions (linguistic, social, physical, technical, historical, biographical) under which it was produced. Such meta-documentation should be as rich and appropriate as the documentary materials themselves.”
69
69 Meta-documentation identity of stakeholders involved, and their roles attitudes of language consultants, towards their languages and towards the documenter and documentation project relationships with consultants and community (Good 2010 mentions what he called ‘the 4 Cs’: ‘contact, consent, compensation, culture’); goals and methodology of researcher, including research methods and tools, corpus theorisation (Woodbury 2011), theoretical assumptions behind annotation, potential for revitalisation
70
70 Meta-documentation project and researcher biography: knowledge and experience of the researcher and consultants (eg. researcher’s knowledge at beginning of project, what training researcher and consultants received) for funded projects: grant application, reports, email communications agreements entered into – formal or informal (eg. Memorandum of Understanding, compensation arrangements), and promises made to stakeholders relationships between this and other projects
71
71 Formats/encoding format choices at these levels: representation of information representation of characters how characters are assembled into files (file formats)
72
72 Characters use UTF-8 (aka Unicode ISO 10646) be aware of using characters outside ASCII (common US keyboard characters) – these can break if UTF-8 is not used distinguish character encoding and fonts (a font is simply a set of images for a “character set”) something may be coded perfectly in UTF-8 but there is no suitable font applied some fonts may display special characters correctly but this does not mean that encoding is correct
73
73 File formats audio WAV (what if original is not WAV??) resolution: 16 bit, 44.1KHz, stereo or better video changing frequently MPEG4 or MTS/H264/AVCH aspect, resolution: depends on project get advice from achive before depositing
74
74 File formats images TIFF **OR** original from device resolution: archive quality is 300dpi or better
75
75 File formats text best is plain text PDF/A often acceptable, may pose problem if MS-Word or ODF, check with archive structured data (spreadsheets, databases original format should be supplied provide a preservable derivative as well (eg csv, PDF) common linguistic software (ELAN, Transcriber, Toolbox, Praat etc) their file formats are generally preservable
76
76 Can I still use MS Word? ELAR no longer accepts MS Word files but Word is still useful quicker to type up useful tables, functions, macros etc solutions think “text only” tables as spreadsheets (are they bad too?) (advanced) complex materials formatted as styles, then export as marked up PDF/A – but not a perfect solution
77
77 My cells have multiple values! example: keywords this is probably OK, as keywords are atomic just consistently use a suitable delimiter e.g. use comma - if data values cannot have commas ELAR recommends double pipe “||”
78
78 My cells have multiple values! example: speakers in a recording speakers are probably not atomic – they have other attributes create a separate “speakers” sheet give each speaker an ID (number or initials) use the IDs in the original sheet, with delimiter (implements one to many) (advanced) or make another sheet to associate recordings with speakers (implements many to many)
79
79 Standards we have already mentioned some standards – UTF-8, WAV etc there are other relevant standards, eg ISO 639-3 (language/dialect names) metadata systems you can also establish project-local standards, eg to handle special characters (eg \e = schwa) data field names document them! – for your usage and for correspondence to wider standards
80
80
81
81
82
82 THANK YOU! www.elar-archive.org David Nathandjn@soas.ac.uk
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.