Presentation is loading. Please wait.

Presentation is loading. Please wait.

Data Collection in Nespole! Goals, procedures and tools Susanne Burger (Carnegie Mellon University) Erica Costantini (University of Trieste) Recent Advances.

Similar presentations


Presentation on theme: "Data Collection in Nespole! Goals, procedures and tools Susanne Burger (Carnegie Mellon University) Erica Costantini (University of Trieste) Recent Advances."— Presentation transcript:

1 Data Collection in Nespole! Goals, procedures and tools Susanne Burger (Carnegie Mellon University) Erica Costantini (University of Trieste) Recent Advances in Speech Translation Systems

2 New Idea Information about Users: Acceptance Usage Behavior Wish-list Problem solving... System Information (Dry Run): Stability Speed Bugs... Speech Material: Domain, concept, vocabulary Style (Human machine conversation) Quality (Robustness)... Why data collection? Learning by Data J.T. Hackos, J.C. Redish, User and Task Analysis for interface design, J. Wiley & Sons, 1998.

3 Learning by Data 1 Mass-Data from the scratch Artificial Scenario/Environment/Set up Wizard of Oz Cooperative User/Actor Data collection through usage of beta-system with increasing reality 2 User-study Data Analysis Development Training Testing Evaluation Beta-System

4 Data Collection: Planning Who are the “Data Customers”? Nespole! : ASR MT Synthesis Interface Development... Type of Collection? Nespole! : Mass Data Collection Specific features User study Customer Needs? Nespole! : Audio / Video Transcription (levels of transcription) Segmentation Time and Budget Data Usage? Nespole! : Analysis Development Training Testing Evaluation

5 Mass-Data Collection: Showcase 1 Travel Scenario / H323 Set up Monolingual Cooperative Users Travel + Multimodality Beta System MT Unseen Users Multimodal Experiment IDEA: NEgotiation through SPOken Language in E-commerce NespoleShowcase1-System Nespole! Data Collection Analysis Development Training Testing Evaluation

6 Example: Mass-Data Collection (Showcase 1) Monolingual data collection for system development “Assembling Line” Data Collection Procedure Recording Scen./Topic Participants Environment Equipment Data

7 Recording Scen./Topic Participants Environment Equipment Data

8 Scenarios Scenario in Nespole! Detailed description of: –the customers’ features (age, marital status…); –the destination of the travel; –the objectives and preferences for the holiday (accommodation, sport activities, cultural events…) J. M. Carroll, Ed., Scenario-Based Design: Envisioning Work and Technology in System Development, New York, J. Wiley & Sons, Scenario: “story” about users, their work, their environment, how they do tasks, the task they need to do, and all combinations of these elements (*).

9 Scenarios in Nespole!

10 Scenario example Situation (Winter Holidays in Val di Fiemme): choose your vacation starting date after December 10th you want to stay there for (a weekend, 1 week, 2 weeks) you have 2 children (choose 2 ages between 2 and 11) and wife/husband you want to travel by car and park it at the hotel you already know the road to Val di Fiemme you want accommodation in ** or *** hotels in Val di Fiemme with bed & breakfast choose two hotels among: Latemar in Molina, Bellavista in Cavalese, Excelsior in Cavalese, Lagorai in Cavalese, Belvedere in Panchia, Bellaria in Predazzo, Cimon in Predazzo, Erica in Tesero, Lucia in Tesero, Montanara in Ziano, Zanon in Ziano you want to practice a winter sport (choose your favorite winter sport among the following: down hill skiing, cross-country skiing/snowshoeing, ice skating, snow- boarding)

11 Things to ask for : prices and how far in advance to book types of ski-lifts nearby and their distance from hotel existence of cross-country trails and ice skating areas details about favorite winter-sport (exact location, prices, possibility of renting equipment) type of parking facilities for the car possibility of eating in the hotel and prices of dinner and late supper daycare and activities for children in the hotel special prices for children Scenario example

12 Scenario definition in Nespole! Example: Showcase 1 analysis of messages (in four languages); clustering of the s on the base of the request type; selection s concerning requests which could be discussed through phone call; construction of 21 scenarios; selection of 5 scenarios* among the 21 (done by the APT tourist board office manager) *

13 Recording Scen./Topic Participants Environment Equipment Data

14 Participants CUSTOMERS: AGENTS: Italian professional agents working at Trentino tourist office APT

15 Recording Scen./Topic Participants Environment Equipment Data

16 APT (agent’s site, Italy) records the English client via H323 connection and the Italian agent via headset CMU (client’s site, USA) records the Italian agent via H323 connection and the English client via headset Environment File.wav (stereo) H323 Eng. customer Agent (local) File.wav (stereo) H323 Agent Eng. Customer (local)

17 Recording Scen./Topic Participants Environment Equipment Data

18 Hardware: PC Pentium 200 and up Software: Windows NT or Win 98 Total Recorder NetMeeting3.01 Microphone: Headset or close microphone Environment: Quiet office Equipment

19 Recording Scen./Topic Participants Environment Equipment Data

20 Recording Procedure (customer’s site)

21 Recording: LTI Data Collection Database Oracle database, accessible online, containing detailed information and descriptions about meetings recorded, demographics of the speakers, transcriptions and audio files (currently two separate interfaces to enter data into and retrieve data from the database)

22 Recording Scen./Topic Participants Environment Equipment Data

23 2 stereo wav files Spr protocol Rpr protocol video tapes (200 collected dialogues )

24 File naming conventions Confusion with parallel recordings; different types of files concerning the same recording; different languages, types of scenario, locations; stereo vs mono files, etc. Why? Example from Nespole! file naming conventions

25 Log data: recording protocol

26 Log data: speaker protocol

27 Audio Data Transcription Conventions Transcription Tool TRL Files MAR Files Voc Lists... m054_1_0575_QXE_00: if it was, I don't know, in the beginning of the century, I would think so, but. m054_5_0576_MTY_00: yeah, I mean, +/we d=/+ we don't know a lot about anything. m054_4_0577_ZMW_00: but +/even/+ I think even if they would have known a little bit more. think about all these chicken farms or things like all this +/k=/+ kind of really terrible behavior against animals, anyway. so, +/I/+ +/I don't think/+ +/I th=/+ I think as soon as some financial or land things or things like this came into the game, they don't think anymore about animal behavior. this is +/ku=/+ just secondary%. m054_3_0578_AAH_00: m054_5_0579_MTY_00: right. m054_4_0580_ZMW_00: so, this... Transcription process

28 Audio Data Transcription Conventions Transcription Tool TRL Files MAR Files Voc Lists... m054_1_0575_QXE_00: if it was, I don't know, in the beginning of the century, I would think so, but. m054_5_0576_MTY_00: yeah, I mean, +/we d=/+ we don't know a lot about anything. m054_4_0577_ZMW_00: but +/even/+ I think even if they would have known a little bit more. think about all these chicken farms or things like all this +/k=/+ kind of really terrible behavior against animals, anyway. so, +/I/+ +/I don't think/+ +/I th=/+ I think as soon as some financial or land things or things like this came into the game, they don't think anymore about animal behavior. this is +/ku=/+ just secondary%. m054_3_0578_AAH_00: m054_5_0579_MTY_00: right. m054_4_0580_ZMW_00: so, this... Transcription process

29 -Verbmobil II: - we are familiar with VMB and we have appropriate tools - BAS partitur format - finite/close system (parsing, filtering,converting) - line oriented, no formats (one line/turn) - turn oriented (turn-IDs contain full identification) - time stamps and trl are in different files linked by turn-ID (- Transcription (trl) Conventions S. Burger, L. Besacier, P. Coletti, F. Metze and C. Morel, “The NESPOLE! VoIP Dialogue Database”, in Proc. of Eurospeech Aalborg, Denmark.

30

31 -words -capitalization -punctuation -white space -turn-end -syntax -non-grammatical phrases -broken words -interrupted words -acoustically hard to understand -pauses and breathing -filled pauses -acoustically not understandable -human noise -word tags -elements -rules Orthography: - orthographic rules as long as they are non-ambiguous - no capitalization in case of initial sentence position - vocabulary lists to keep vocabulary spelled the same Content

32 Foreign Language Turn (JAP, GER,..) ;.. global Comment..'.. Apostrophe (reduced word)..-.. (--) Hyphen (compound word) $.. spelled Letter ~..Name #.. Number *.. Neologism/Mispronunciation <*XXX.. Foreign Word (FRA,ITA,..)..... /.... Lengthening..% Poor intelligible..= Articulated Break-off.._ Interruption of a Word, Left Fragment _.. Interruption of a Word, Right Fragment.. Technical Interruption of a Word, Beginning.. Technical Interruption of a Word, End Technical interruption of a Turn t Technical Break-off of a Turn Comment on Pronunciation. / ? /, Punctuation +/.. Beginning of a Repetition/Correction../+ End of a Repetition/Correction -/.. Beginning of a False Start../- End of a False Start / Respiration / Filled Pause (Hesitation) Filled Pause (Hesitation) / Filled Pause (Hesitation) Unidentifiable Sound Production / Nonverbal Artikulatory Sound (sound: smacking) / Nonverbal Artikulatory Sound (sound: swallowing) / Nonverbal Artikulatory Sound (sound: clearing one's throat) / Nonverbal Artikulatory Sound (sound: cough) / Nonverbal Artikulatory Sound (sound: laughing) / Nonverbal Artikulatory Sound (other sounds) / Technical Noise Technical Noise Pause during Active Interference by a Passively Interfered Speaker Active Interference by Acoustic Passive Interference of Acoustic Events.. Beginning of Noise Interference..:> End of Noise Interference Local Comment !KEY!.. Code Word Scenario Caused Pause

33 Audio Data Transcription Conventions Transcription Tool TRL Files MAR Files Voc Lists... m054_1_0575_QXE_00: if it was, I don't know, in the beginning of the century, I would think so, but. m054_5_0576_MTY_00: yeah, I mean, +/we d=/+ we don't know a lot about anything. m054_4_0577_ZMW_00: but +/even/+ I think even if they would have known a little bit more. think about all these chicken farms or things like all this +/k=/+ kind of really terrible behavior against animals, anyway. so, +/I/+ +/I don't think/+ +/I th=/+ I think as soon as some financial or land things or things like this came into the game, they don't think anymore about animal behavior. this is +/ku=/+ just secondary%. m054_3_0578_AAH_00: m054_5_0579_MTY_00: right. m054_4_0580_ZMW_00: so, this... Transcription process

34 Why another tool? Other requirements as before: - Windows instead of Linux - Meetings – multiparty transcription - Transcriber from different backgrounds At that time (over three years ago) there wasn’t a sufficient transcriber tool We did a study what would be the basic requirements. We asked transcribers what they would find convenient. We programmed a beta tool according to that. We are still using this tool (and so do different other places in the mean time) We call it TransEdit. Transcription Tools

35 MFC program Windows text editor click-able buttons for transcription elements automatic turn naming and counting label editor parallel display of multi audio signals easy turn segmentation lots of listen functions easy handling, no research functions “home work” but available for universities (write to: TransEdit: transcription tool just for transcribers

36

37 Audio Data Transcription Conventions Transcription Tool TRL Files MAR Files Voc Lists... m054_1_0575_QXE_00: if it was, I don't know, in the beginning of the century, I would think so, but. m054_5_0576_MTY_00: yeah, I mean, +/we d=/+ we don't know a lot about anything. m054_4_0577_ZMW_00: but +/even/+ I think even if they would have known a little bit more. think about all these chicken farms or things like all this +/k=/+ kind of really terrible behavior against animals, anyway. so, +/I/+ +/I don't think/+ +/I th=/+ I think as soon as some financial or land things or things like this came into the game, they don't think anymore about animal behavior. this is +/ku=/+ just secondary%. m054_3_0578_AAH_00: m054_5_0579_MTY_00: right. m054_4_0580_ZMW_00: so, this... Transcription process

38 ; CDR: ; TRV: ; File: e025at ; Last changes made on 09/29/2000 ; Transcriber: VLM ; Comments: ; e025_1_0000_ITL_00: hello ? can you hear me now ? e025_2_0001_XYZABC_00: hello. e025_1_0002_ITL_00: hello%. yeah%. e025_2_0003_ XYZABC _00: yes, I can. e025_1_0004_ITL_00: yes, okay. so ? e025_2_0005_ XYZABC _00: -/hi I would like/- yes ? e025_1_0006_ITL_00: yes, can you hear me now ? e025_2_0007_ XYZABC _00: yes, I can. e025_1_0008_ITL_00: okay. wonderful. so, can I help you ? e025_2_0009_ XYZABC _00: -/all right I would like/- yes, madam. I would like to schedule a winter vacation in the north of Italy. e025_1_0010_ITL_00: e025_1_0011_ITL_00: yes. would you like t= t e025_1_0012_ITL_00: yes. would you like to come here% in summer or during winter ? e025_2_0013_ XYZABC _00: in winter please.

39 automatic convention check close check and correction by another transcriber spell-checking marker file and trl file cross-check first pass transcription (but not rough..) Data transcription process

40 Audio Data Transcription Conventions Transcription Tool TRL Files MAR Files Voc Lists... m054_1_0575_QXE_00: if it was, I don't know, in the beginning of the century, I would think so, but. m054_5_0576_MTY_00: yeah, I mean, +/we d=/+ we don't know a lot about anything. m054_4_0577_ZMW_00: but +/even/+ I think even if they would have known a little bit more. think about all these chicken farms or things like all this +/k=/+ kind of really terrible behavior against animals, anyway. so, +/I/+ +/I don't think/+ +/I th=/+ I think as soon as some financial or land things or things like this came into the game, they don't think anymore about animal behavior. this is +/ku=/+ just secondary%. m054_3_0578_AAH_00: m054_5_0579_MTY_00: right. m054_4_0580_ZMW_00: so, this... Transcription process

41 Following mass-data collection Showcase 2a and 2b

42 Analysis of medical databases Definition of some scripts Pre-tests Scenarios Data collection Medical scenarios development Doctors


Download ppt "Data Collection in Nespole! Goals, procedures and tools Susanne Burger (Carnegie Mellon University) Erica Costantini (University of Trieste) Recent Advances."

Similar presentations


Ads by Google