Presentation is loading. Please wait.

Presentation is loading. Please wait.

Overview: Requirements for implementing the AARDVARC vision Gary Simons SIL International AARDVARC Workshop 9–11 May 2013, Ypsilanti, MI.

Similar presentations


Presentation on theme: "Overview: Requirements for implementing the AARDVARC vision Gary Simons SIL International AARDVARC Workshop 9–11 May 2013, Ypsilanti, MI."— Presentation transcript:

1

2 Overview: Requirements for implementing the AARDVARC vision Gary Simons SIL International AARDVARC Workshop 9–11 May 2013, Ypsilanti, MI

3 The context  A cross-cutting, NSF-wide initiative called  Cyberinfrastructure Framework for 21st Century Science and Engineering (CIF21) Cyberinfrastructure Framework for 21st Century Science and Engineering (CIF21)  Vision statement  “CIF21 will provide a comprehensive, integrated, sus- tainable, and secure cyberinfrastructure to accelerate research and education and new functional capabili- ties in computational and data-intensive science and engineering, thereby transforming our ability to effectively address and solve the many complex problems facing science and society.” 2

4 The funding program  AARDVARC grant was awarded by NSF’s program on Building Community and Capacity for Data- Intensive Research in the Social, Behavioral, and Economic Sciences and in Education and Human Resources (BCC-SBE/EHR)Building Community and Capacity for Data- Intensive Research in the Social, Behavioral, and Economic Sciences and in Education and Human Resources (BCC-SBE/EHR)  We “seek to enable research communities to de- velop visions, teams, and prototype capabilities dedicated to creating and utilizing innovative and large-scale data resources and relevant analytic techniques to advance fundamental research for the SBE and EHR areas of research.” 3

5 A three-stage program 1. Funded projects focus on bringing together cross-disciplinary communities to work on the design of cyberinfrastructure for data-intensive research. [2012 and 2013] 2. A selection (perhaps one-fourth) of these communities will be funded to develop prototypes of the facilities designed in Stage 1. [Beginning 2014, funding permitting] 3. An even smaller number of projects will be funded to develop the actual facility. 4

6 Roadmap for current project  The competition will be fierce across a wide range of disciplines.  In order to succeed in the second stage of the program, we must write a top-25% proposal.  Can we put ourselves in the shoes of potential re- viewers and anticipate what the likely critiques to an AARDVARC implementation proposal might be?  If so, that could help us set an agenda for the problems we should be working on during the course of the current project. 5

7 Fast forward to implementation  The current AARDVARC proposal is not an implementation proposal  However, reading it through that lens sheds light on what would need to be addressed if it were  Reading the proposal in this way,  I have imagined four show-stopping reviewer critiques that we want to be sure to avoid  This presentation discusses the requirements for an implementation proposal that would avoid these critiques 6

8 Critiques we want to avoid 1. The focus seems too narrow to be truly transformative. 2. The issues of sustainability are not adequately addressed. 3. It is not clear that automatic transcription of under-resourced languages is even possible. 4. There is not an adequate story about how the community will work on a large scale to fill the repository. 7

9 1. Find the right framing  Vision of CIF21: “transform our ability to effectively address and solve the many com- plex problems facing science and society”  Potential critique  The AARDVARC focus seems too narrow to be truly transformative.  Requirement  A successful proposal will need to frame the proposed cyberinfrastructure in terms that non- linguists will embrace as truly transformative. 8

10 Problem  The name AARDVARC frames the problem in terms of a repository for automatically annotated video and audio resources  Among non-linguists is a framing in terms of automatic annotation likely to rise to the top 25% of cross-cutting problems?  Probably not since solving the transcription bottleneck puts the focus on a means to the end, rather than the end itself  The true end is having a repository of data from every language 9

11 A more compelling framing  The AARDVARC name fails to name the main thing — language  The most fundamental problem for data- intensive research in the 21 st century is that we lack a repository of interoperable data from every human language  Among non-linguists, would a framing like that rise to the top 25% of cross-cutting problems?  This seems much more likely  And others have already laid some groundwork 10

12 Human Language Project  Building by analogy to the Human Genome Project, Abney and Bird have proposed a Human Language Project to the computational linguistics community:  “We present a grand challenge to build a corpus that will include all of the world’s languages, in a consistent structure that permits large-scale cross- linguistic processing, enabling the study of universal linguistics.” (Abney and Bird 2010)  In two conference papers, they have argued the motivation for the project and specified basic formats for data 11

13 Language Commons  Building on “the commons” tradition, Bice, Bird, and Welcher have spearheaded the Language Commons  “The Language Commons is an international consortium that is creating a large collection of written and spoken language material, made available under open licenses. The content includes text and speech corpora, along with translations, lexicons and other linguistic resources that support large-scale investigation of the world's languages.”  Currently an open collection in the Internet Archive  Browse:  Submit: 12

14 We need to join forces  AARDVARC, Human Language Project, and the Language Commons are variations on the same fundamental vision  A repository of interoperable data from every human language  Facing fierce competition with other disciplines  We are too small to have competing visions, we need a single vision that others will find compelling  For an implementation proposal, we should all join forces to create a grand vision of cyberinfrastructure for language-related research in the 21 st century that will embrace every language 13

15 References  The Human Language Project: Building a universal corpus of the World’s languages Steven Abney and Steven Bird Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 88-97, Uppsala, Sweden The Human Language Project: Building a universal corpus of the World’s languages  Towards a data model for the Universal Corpus Steven Abney and Steven Bird Proceedings of the 4th Workshop on Building and Using Comparable Corpora, , Portland, USA Towards a data model for the Universal Corpus  The Language Commons Wiki Ed Bice and others Presentation at Wikimania 2010, Gdańsk, Poland The Language Commons Wiki  The Rosetta Project and The Language Commons Laura Welcher Presentation posted on The Long Now Foundation blog. The Rosetta Project and The Language Commons 14

16 2. Ensure sustainability  Vision of CIF21:  “provide a … sustainable... cyberinfrastructure”  Potential critique  The issues of sustainability are not adequately addressed.  Requirement  A successful proposal will need to give a convincing plan for the sustainability of the infrastructure and the resources it houses. 15

17 16 A repository is not enough  Simply building a repository does not ensure sustainability  It must also function as an archive that guarantees access far into the future  A huge NSF investment in the repository we envision would go to waste if it could not  Continue operating after the grant money ran out  Survive the inevitable upgrades to hardware and system software at the host institution  Recover from a disaster (natural or institutional)

18 17 Non-use is also waste  Even deeper than the sustained functioning of a repository is the sustained use of the resources it houses  The huge investment would also go to waste if  Resources deteriorate or slip to obsolete formats  Potential users never discover relevant resources  Users are unable to access discovered resources  Users cannot make sense of resources they access  Accessed resources are not compatible with the computational working environments of users

19 Conditions of sustainable use  A complete proposal would addresses the condi- tions of sustainable use (Simons & Bird 2008, sec. 3)  Extant — Preserved through off-site backup, refreshing copies, format migration, fixity metadata  Discoverable — Adequate descriptive metadata accessed through open and easy-to-use search  Available — User has rights to access as well as a means of access  Interpretable — Markup, encoding, abbreviations, terminology, methodologies are well documented  Portable — File formats that are open (not proprietary) and work on all platforms 18

20 Checklist for responsible archiving  A good proposal would measure up against the criteria of the TAPS Checklist (Chang 2010, pp )  Based on a review of mainstream tools for assessing archival practices, TAPS is a checklist of 16 points to help linguists evaluate whether a prospective home for their data will be a responsible archive  Target — Are the mission and audience a good fit?  Access — Will your audiences have adequate access?  Preservation — Is the archive following best practices for ensuring long-term preservation?  Sustainability — Is the institution well situated for the long term? 19

21 A repository or an aggregator?  Or should the infrastructure have an aggregator at the center rather than a single repository?  In today’s web economy, being the aggregator (rather than a supplier) is the sweet spot (Simons 2007 paints a vision of such a cyberinfrastructure)  This would require community agreement on:  Metadata standards (content, format, protocol) — OLAC provides a starting point  Data standards (contents, formats, protocols) — Universal Corpus provides a starting point  Still needs a self-service default repository  e.g. Language Commons in Internet Archive 20

22 References  Toward a global infrastructure for the sustainability of language resources Gary Simons and Steven Bird Proceedings of the 22nd Pacific Asia Conference on Language, Information and Computation, 20– 22 November 2008, Cebu City, Philippines. Pages 87–100. Toward a global infrastructure for the sustainability of language resources  TAPS: Checklist for responsible archiving of digital language resources Debbie Chang MA thesis, Graduate Institute of Applied Linguistics. Dallas, TX. TAPS: Checklist for responsible archiving of digital language resources  Doing linguistics in the 21st century: Interoperation and the quest for the global riches of knowledge Gary Simons Proceedings of the E-MELD/DTS-L Workshop: Toward the Interoperability of Language Resources, 13–15 July 2007, Palo Alto, CA. Doing linguistics in the 21st century: Interoperation and the quest for the global riches of knowledge 21

23 3. Focus on achievable automation  Purpose of BCC-SBE/EHR:  “enable research communities to develop … prototype capabilities”  Potential critique  It is not clear that automatic transcription of under-resourced languages is even possible.  Requirement  A successful proposal will need a compelling description of automated helps for annotation that can be implemented today. 22

24 The BCC-SBE/EHR vision  Building Community and Capacity for Data-Intensive Research program is about activity in the present to support research in the future: Building Community and Capacity for Data-Intensive Research 23 Present activities We “seek to enable research communities to develop visions, teams, and prototype capabilities Present focus dedicated to creating and utilizing innovative and large-scale data resources and relevant analytic techniques Future result to advance fundamental research for the SBE and EHR areas of research.”

25 Setting the right target  Automated transcription of under-resourced languages is still in the future  It is an advance in fundamental research that can be furthered by a data-intensive cyberinfrastructure  The follow-up proposal in the BCC program is an implementation proposal, not a research proposal  It must focus on the automated helps for annotation that we can implement immediately  It is not meant to be a request to support research on annotation tasks we cannot currently automate  It should implement a framework into which we can plug the latter as that research comes to fruit 24

26 Sorting the tasks  During the AARDVARC project we should  Identify annotation tasks that we can automate now  Plan work modules for these in the proposed implementation grant  Identify annotation tasks that are clearly in the future  Pursue research grants on these through the normal research programs  Implementation proposal would mention supplying data to future research as within its broader impacts  Identify annotation tasks that are borderline  Conduct proof-of-concept testing now to determine whether it belongs in the first set or the second set

27 Breaking the bottleneck  The repository should embrace all strategies for breaking the transcription bottleneck  Focus on the end of data in every language, as opposed to a particular means for getting it  A promising new strategy is oral annotation  Woodbury (2003) proposed this to turn a huge collection of tapes from 15 years of Cup’ik radio broadcasts into usable data  Make running oral translations  Do careful respeaking of “hard-to-hear tapes”  This inspired the development of BOLD:  Basic Oral Language Documentation 26

28 References  Defining documentary linguistics Anthony Woodbury In Peter Austin (ed.), Language Documentation and Description 1: London: SOAS. Defining documentary linguistics  The rise of documentary linguistics and a new kind of corpus Gary Simons Presented at 5th National Natural Language Research Symposium, De La Salle University, Manila, 25 Nov The rise of documentary linguistics and a new kind of corpus  Basic Oral Language Documentation D. Will Reiman Language Documentation and Conservation, Vol. 4, pp Basic Oral Language Documentation  A scalable method for preserving oral literature from small languages Steven Bird Proceedings of the 12th International Conference on Asia-Pacific Digital Libraries, 5-14, Gold Coast, Australia A scalable method for preserving oral literature from small languages  To BOLDly go where no one has gone before Brenda Boerger Language Documentation and Conservation, Vol. 5, pp To BOLDly go where no one has gone before 27

29  Original recording on first recorder  Careful respeacking on second recorder  Original played back (with pauses) into left channel  Respoken on mike into right channel Example of respeaking 28 From fieldwork of Will Reiman on Kasanga [cji] language, Guinea-Bissau

30 A known best practice in field methods  Instructions for the Recording of Linguistic Data  In Bouquiaux and Thomas (1976), trans. Roberts (1992). Studying and Describing an Unwritten Language. Dallas: Summer Institute of Linguistics.  “Go over this spontaneous recording, either with the narrator himself or with a qualified speaker, in order to have it repeated sentence by sentence, in a careful, relatively slow, yet normal manner, and to have it whistled (tone languages).” (p. 180)  Goes on to describe method using 2 tape recorders  This method may be even more essential today as we prepare recordings for automatic transcription

31 BOLD:PNG  A project led by Steven Bird; see  Trained university students to use low-cost digital recorders to go back to their home villages to make recordings and to annotate them orally  Problems:  Managing all the files on all the recorders did not scale  Two recorder annotation was too complicated 30

32 Working on solutions  Language Preservation 2.0: Crowdsourcing Oral Language Documentation using Mobile Devices   They have developed an Android app, Aikuma  Files shared within community via Internet or local Wi-Fi hub; supports voting for what to release  Annotate on a single device with a simple two-button tool  Blog post containing two demo videos from Bird’s current field trip in the Amazon Blog post 31

33 4. Foster global collaboration  Purpose of BCC-SBE/EHR:  “enable research communities … to creat[e] new, large-scale, next-generation data resources”  Potential critique  There is not an adequate story about how the community will work on a large scale.  Requirement  A successful proposal will need a compelling account of how a global community of researchers, speakers, and citizen scientists will collaborate to fill the repository with annotated resources. 32

34 The real challenge  Building the repository is one thing, but filling it with resources from most languages will be quite another  Funded staff will be able to implement the repository, but it will take thousands of volunteers to really fill it  Realizing the vision will depend on  Mobilizing the research community to participate  Mobilizing speaker communities to participate  Mobilizing citizen scientists to participate  Building an infrastructure that supports collaboration among all these players on a global scale 33

35 Resources as open-ended  Repository must support open-ended annotation  After initial deposit, other players should be able to  Add careful respeaking  Add a translation (either oral or written)  Add a transcription (of text or of translation)  Add a translation of the translation  Invoke an automatic transcription or translation  Check and revise the automatic output  Each addition should be a separate deposit (with its own metadata) that links back to what it annotates (i.e., stand-off markup) 34

36 Resource workflow  The types and languages of the complete set of annotations associated with a resource comprise the state of that resource  The annotation tasks are operators on that state  Each annotation task has a prerequisite state  Performing the task changes the state of the resource  This defines an implicit workflow  For any resource, there is a set of possible next tasks  The infrastructure needs to manage that workflow 35

37 36 Supply and demand  We need to match up two things:  The huge demand for annotation tasks to be done — all of the possible next tasks for all resources  The supply of people worldwide who could do them  Our infrastructure needs to be a marketplace that matches supply with demand  E.g., eBay, eHarmony, mTurk.commTurk.com  Match a user’s language profile to find next tasks to do  E.g., TED’s Open Translation Project using AmaraOpen Translation ProjectAmara  Web tool to segment videos and add subtitles  140 languages, ~10,000 translators, >50,000 translations

38 If we build it …  They won’t necessarily come!  In addition to describing the infrastructure we would implement to match supply and demand, a compelling proposal would also:  Describe the plans for organizing the people who participate (including governance)  Describe plans for mobilizing the various target communities: researchers, speakers, citizens  Describe incentives for participation, especially ones that are built into the design of the infrastructure 37

39 Conclusion  The AARDVARC project gives us the opportunity to build the vision and plans for a sustainable cyberinfrastructure to  Collect and provide access to interoperable data resources from every human language  Harness automation wherever possible to add the needed transcriptions and translations  Create a marketplace that will permit thousands worldwide to collaborate in performing the annotation tasks that cannot be automated  Thus transforming our ability to address and solve language-related problems facing science and society


Download ppt "Overview: Requirements for implementing the AARDVARC vision Gary Simons SIL International AARDVARC Workshop 9–11 May 2013, Ypsilanti, MI."

Similar presentations


Ads by Google