Presentation is loading. Please wait.

Presentation is loading. Please wait.

Gile SURVEYS1 SURVEYS IN TS SOME PRINCIPLES AND A CASE STUDY

Similar presentations


Presentation on theme: "Gile SURVEYS1 SURVEYS IN TS SOME PRINCIPLES AND A CASE STUDY"— Presentation transcript:

1 Gile SURVEYS1 SURVEYS IN TS SOME PRINCIPLES AND A CASE STUDY

2 Gile SURVEYS2 WHY SURVEYS? IN EMPIRICAL RESEARCH OBJECTIVE DESCRIPTION AND MEASUREMENT CANNOT COVER EFFICIENTLY ALL RELEVANT PHENOMENA IN PARTICULAR: THOUGHTS, MOTIVATIONS, OPINIONS, INTENTIONS CANNOT BE OBSERVED OR MEASURED DIRECTLY INDIRECT EXPERIMENTAL PROCEDURES ON INDICATORS ARE COMPLEX AND SOMETIMES TRICKY ASKING PEOPLE IS ANOTHER OPTION

3 Gile SURVEYS3 CAUTION PEOPLES UTTERANCES ARE NOT ALWAYS TRUTHFUL OR ACCURATE THEY ARE NOT NECESSARILY EASY TO INTERPRET BUT THEY ARE OFTEN A USEFUL COMPONENT OF RESEARCH FOR: -PRELIMINARY EXPLORATION -COMPLEMENTARY COLLECTION OF INFORMATION -VALIDATION OF INFORMATION OBTAINED THROUGH OTHER METHODS RELEVANCE TO RESEARCH INTO TRANSLATION IS OBVIOUS BECAUSE OF: COMPLEX PROCESSING IS PART OF TRANSLATION, ONLY PRODUCT IS VISIBLE, CORPORA ARE DIFFICULT TO OBTAIN, INTERPRETATION OF INDICATORS IS DIFFICULT

4 Gile SURVEYS4 THERE ARE MANY TYPES OF SURVEYS LARGE SCALE/SMALL SCALE SPOKEN (INTERVIEWS) VS. QUESTIONNAIRES COLLECT OBJECTIVE INFO VS. SUBJECTIVE INFO FOR QUALITATIVE PURPOSES / SIMPLE QUANTITATIVE PURPOSES / ADVANCED QUANTITATIVE PROCESSING INDEPENDENT / PART OF A LARGE PROJECT WITH OTHER COMPONENTS

5 Gile SURVEYS5 (SOME) OBSTACLES TO COLLECTION OF TRUTHFUL, ACCURATE INFO IN SURVEYS LACK OF WILLINGNESS OF PEOPLE TO INVEST EFFORTS IN SURVEYS LOW RESPONSE RATES, INCOMPLETE ANSWERS - RESPONSES VARY AS A FUNCTION OF SELF-IMAGE AND IMAGE SUPPOSEDLY GIVEN TO INTERVIEWER (SO LESS THAN TRUTHFUL RESPONSE POSSIBLE) - RESPONSES VARY AS A FUNCTION OF ULTERIOR INTERESTS - QUESTIONS (AND ANSWERS) MAY BE INTERPRETED DIFFERENTLY BY AUTHORS AND READERS (RESPONDENTS AND ANALYSTS)

6 Gile SURVEYS6 IMPORTANT STRATEGIES -MINIMIZE EFFORTS FOR RESPONDENTS (CLEAR QUESTIONS, EASY TO ANSWER, LENGTH OF QUESTIONNAIRES) -PILOT THE QUESTIONNAIRES TO MAKE SURE PEOPLE UNDERSTAND PILOTING IS ALSO IMPORTANT TO DETECT MISSING ITEMS, AMBIGUITY AND OTHER WEAKNESSES -TRY TO ELIMINATE OR REDUCE EFFECT OF SELF-IMAGE (QUESTIONING STRATEGIES, TRAINING OF INTERVIEWERS) IF NOT POSSIBLE, USE RESULTS CONSERVATIVELY

7 Gile SURVEYS7 SAMPLING AND REPRESENTATIVITY SIZE IS NOT THE ONLY CONCERN REPRESENTATIVITY IS PARAMOUNT POPULATION SUB-POPULATION BIAS (SYSTEMATIC ERROR – versus SAMPLING ERROR) AVOIDING BIAS THROUGH RANDOM SAMPLING MAY WANT STRATIFIED RANDOM SAMPLING (RANDOM SAMPLING WITHIN SUB-POPULATIONS THAT YOU WANT TO INCLUDE)

8 Gile SURVEYS8 SAMPLING ISSUES SMALL SAMPLE SIZE DOES NOT IMPLY BIAS THROUGH IT MAY NOT HAVE FEATURES CLOSE ENOUGH TO THOSE OF THE POPULATION DUE TO SAMPLING ERROR SAMPLING ERROR DECREASES WITH SAMPLE SIZE (BUT PROPORTIONALLY TO SQUARE ROOT OF SAMPLE SIZE – MUTLIPLY SAMPLE SIZE BY 4 NAND YOU WILL ONLY REDUCE SAMPLING ERROR BY 2) SO INCREASING SAMPLE SIZE IS ONLY EFFICIENT TO A MODERATE EXTENT ESPECIALLY AS PATTERNS STABILIZE

9 Gile SURVEYS9 CASE STUDY: PUBLISHING IN TS EST SEMINAR IN GHENT IN 2005 EDITORS ASKED TO TALK ABOUT: - CRITERIA FOR PUBLICATION - DIFFICULTIES IN PUBLISHING ACTIVITY -REFEREEING DURING SEMINAR: JOURNAL EDITORS DID NOT SEEM AWARE OF THE FACT THAT THEY ARE IN COMPETITION DID NOT MENTION PROBLEMS WITH REFEREEING SO SURVEY TO CHECK BOTH

10 Gile SURVEYS10 PURPOSE OF THE SURVEY IDENTIFY FACTORS WHICH MAY BE RELEVANT/IMPORTANT (NOT QUANTIFY THEM) DO SOME EXPLORATION IN PREPARATION FOR FURTHER STUDIES IF INTERESTING ISSUES COME UP SURVEY DESIGN TAKES THESE OBJECTIVES ON BOARD (LESS STRINGENT REQUIREMENTS THAN IF HYPOTHESES WERE TO BE TESTED OR IF INFLUENCES WERE TO BE QUANTIFIED)

11 Gile SURVEYS11 QUESTIONNAIRE OR INTERVIEWS? QUESTIONNAIRES - INTERNET WHAT QUESTIONS? ONE SET ON CRITERIA FOR SELECTION ONE SET ON REFEREEING PROBLEMS (Not standard procedure, but for efficiency and higher response rate) -THINK OF CERTAIN CRITERIA FOR SELECTION -BUT LET RESPONDENTS ADD -ON REFEREEING, ASK ABOUT POSITIVE EXPERIENCE AND NEGATIVE EXPERIENCE -ASK ABOUT REFEREES EXPERIENCE -PLUS QUESTIONS ON RESPONDENTS TO ANALYZE BY EXPERIENCE, COUNTRY, LANGUAGES IF POSSIBLE (MINIMAX)

12 Gile SURVEYS12 CLASSIFY RESPONDENTS AS BEGINNERS / EXPERIENCED / IN THE MIDDLE BECAUSE THIS MAY BE RELEVANT TENTATIVE DUAL CRITERION WITH ARBITRARY THRESHOLDS (TURNED OUT TO BE ALL RIGHT – PERHAPS BECAUSE DESIGNER OF SURVEY KNEW THE FIELD WELL ENOUGH) - How long have you been in the field of TS? - Less than 5 years - 5 to 10 years - More than 10 years -How many papers have you published so far in the field, either in journals or in collective volumes? - Less than to 10 - More than 10

13 Gile SURVEYS13 SELECTION CRITERIA WHEN CHOOSING JOURNALS Reputation/Rating - ATTENTION this is subjective Speed of publication Choosing the language of publication Why these criteria? What scale? - Not at all important - Not very important - Quite important - I dont know Why not numerical? Why not more?

14 Gile SURVEYS14 REFEREEING - How often have you felt that the referees comments were useful to you? - ATT - How often have you felt that referees misunderstood your statements? - ATT - If you have acted as a referee yourself, how often have you felt that you had to perform this function because nobody else with the right qualification was available or willing to do the job, but were not fully qualified because of language problems or because you were not familiar enough with the field, topic or research methods concerned? – ATT Often/Sometimes / Rarely / I dont know NOTE: BOTH POSITIVE AND NEGATIVE Why? AUTHORS AND REFEREES VIEWPOINT Why?

15 Gile SURVEYS15 PILOTING DISTRIBUTION MODE : INTERNET ADVANTAGES AND DISADVANTAGES? SENT TO COLLEAGUES EXPERIENCED IN SURVEYS FOR INPUT (CHIARO AND NOCELLA) Why? SOME CHANGES ON BASIS OF THEIR COMMENTS AND SUGGESTIONS - PILOTING 1: SENT TO ABOUT 15 SCHOLARS (AFTERWARDS, THEY ARE NO LONGER AVAILABLE AS RESPONDENTS) - PILOTING 2: INITIAL PRODUCTION: FIRST BATCH – POSSIBILITY OF FINDING FURTHER PROBLEMS AND CORRECTING - THEN FINAL PRODUCTION

16 Gile SURVEYS16 PREPARATION OF DATA FOR ANALYSIS EXCEL FOR SIMPLICITY, PORTABILITY (DATA BASE MORE POWERFUL, BUT MORE DIFFICULT TO EXPORT TO OTHER COLLEAGUES/SYSTEMS) NAMES, MEANINGFUL (FOR EASIER READING AND INTERPRETATION THAN CODES) W/ DIFFERENT INITIALS (FOR AUTOMATIC COPYING BY EXCEL – SAVES TIME) COLUMN FOR FURTHER COMMENTS IDENTIFIERS (NOT NAMES OF RESPONDENTS, UNLESS EXPLICIT PERMISSION) INITIAL INFO ON RESPONDENTS TO HELP FIND OUT ABOUT INFLUENCE OF POTENTIALLY RELEVANT FACTORS (COUNTRY, EXPERIENCE)

17 Gile SURVEYS17 IdentCountryExperien ce PublicsReputati on of journal Speed of publicatio n Languag e UsefulMisunder stings Not qualified Other criteria 1AUSB 5-10C 10+QUITE 3NOT VYZEROOFTENSMTMSNEVERINTERA CTN W/ REFERE ES 2BELGBCQNZOSRAREPOLITIC S 3CANN/A 4CHINA 5CZ 6DK 7FINL

18 Gile SURVEYS18 COUNTRIES n = 85 AUSTR 2OEST 3 BELG 4 GERM 4 GR 1 IRL 1 ITAL 4 NL 1 PORT 1 SPAIN 14 UK 4 FRANCE 1 CAN 11 US 1 CHINA 2 IL 3 JAP 1 KOR 6 MALAY 1 CZ 2 POL 3 SLOV 1 DK 5 FINL 3 SWED 1 SOUTHAF 1 N/A

19 Gile SURVEYS19 WHY RELEVANT AND WHAT CAN WE SAY ABOUT THIS SAMPLE? FROM ALL OVER THE WORLD BUT STRONG WEST EUROPEAN BIAS POSSIBILITY OF CHECKING BY REGIONS POSSIBILITY OF CHECKING BY COUNTRIES FOR SOME FACTORS (IF HYPOTHESES ON LANGUAGE, REFEREEING W/O FULL QUALIFICATIONS)

20 Gile SURVEYS20 ExperPublsReputSpeedLangUsefulMisunde rst Not qualOther crit A:15/18% B:16/18% C:54/64% A:14/17% B:13/15% C:58/68% Q:75/88% NV:10/12 % Q:50/60% NV:33/37 % Z:3/4% Q:39/45% NV:32/38 % Z:13/15% NA:1 O:42/50% S:34/40% R:5/5% NA:4 O:3/4% S:35/42% R:42/49% NA:5 O:2/3% S:25/37% R:24/37% Z:15/23% NA:19 INTL 8 SPCLZD 4 READERS HIP 4 POLITICS, OPEN MIND, INTEGRIT Y OF PAPER/E DITORS N = 85 A 0-5 yrs B 5-9 yrs C > 10 yrs A > 5 B 5-9 C > 10 Q quite impt NV not very impt NA: not available O often S stimes R rarely Z never Z: zero imptance

21 Gile SURVEYS21 INTERPRETING THE RESULTS CRITERIA FOR CHOOSING JOURNAL REPUTATION: Q 88% CLEARLY IMPORTANT SPEED: Q 60% - NV 37% OFTEN IMPORTANT – When? LANGUAGE: Q 45% - NV 38% SOMETIMES NOT CLEAR BY COUNTRY IN SPITE OF HYPOTHESIS – FURTHER EXPLORATION POSSIBLE REFEREEING USEFUL: O 50% - S 40% - R 5% POSITIVE ON THE WHOLE – OVERASSESSED BECAUSE POLITICALLY CORRECT? MISUNDERSTANDINGS: O: 4% - S: 42% - R: 49% OK ON THE WHOLE I NOT COMPETENT: O: 3% - S: 37% NOT INSIGNIFICANT – UNDERASSESSED? – NOT CLEAR BY COUNTRY DESPITE SPECIFIC SITUATIONS – FURTHER EXPLORATION

22 Gile SURVEYS22 SOME LIMITATIONS AND SHORTCOMINGS OF THE STUDY -NOT RANDOM SAMPLE, DIFFICULT TO ASSESS BIAS - QUESTIONNAIRE SENT IN ENGLISH ONLY POSSIBLE LANGUAGE BIAS, PARTICULARLY IMPORTANT FOR QUESTION ON RELEVANCE ON LANGUAGE - IN LAST QUESTION, POSSIBLE AMBIGUITY BETWEEN NON APPLICABLE AND NEVER. SHOULD HAVE ADDED NON APPLICABLE CATEGORY

23 Gile SURVEYS23 WAS THE EXERCISE USEFUL? QUANTITATIVE RESULTS CONFIRM INTUITIVE KNOWLEDGE, GIVE GENERAL ORDER OF MAGNITUDE NO BIG SURPRISES PERHAPS USEFUL TO KNOW THAT MORE THAN A THIRD OF THE RESPONDENTS SAY THAT SOMETIMES THEY ARE NOT FULLY QUALIFIED QUALITATIVE RESULT: MANY RESPONDENTS WANT TO INTERACT WITH REFEREES – PERHAPS USEFUL FOR EDITORS POLICY DECISIONS


Download ppt "Gile SURVEYS1 SURVEYS IN TS SOME PRINCIPLES AND A CASE STUDY"

Similar presentations


Ads by Google