1. D.Gile citations as indicators1 CITATIONS AS INDICATORS IN TS Daniel Gile daniel.gile@yahoo.com www.cirinandgile.com

2. D.Gile citations as indicators2 AIMS OF THIS PRESENTATION THIS PRESENTATION IS OFFERED IN THE FRAMEWORK OF RESEARCHER TRAINING RESEARCH CONTEXT IS: RESEARCH ON TS AS AN OBJECT OF STUDY TS IS A COMPLEX ENTITY, HENCE THE USE OF INDICATORS IN ITS STUDY THE CONCEPT OF INDICATORS IS EXPLAINED USES OF CITATIONS AS AN INDICATOR IN THE STUDY OF TS ARE EXPLAINED METHODOLOGICAL ISSUES WHICH ARE RELEVANT TO OTHER EMPIRICAL STUDIES ARE DISCUSSED

3. D.Gile citations as indicators3 INDICATORS INDICATORS ARE MEASURABLE UNITS (or at least countable units) WHICH ARE USED TO MEASURE (or at least detect/identify) ASPECTS OF ENTITIES WHICH ARE DIFFICULT TO MEASURE DIRECTLY BECAUSE THEY ARE COMPLEX AND/OR DIFFICULT TO DEFINE OPERATIONALLY

4. D.Gile citations as indicators4 EXAMPLES OF INDICATORS - FOR HEALTH: BODY TEMPERATURE, BLOOD PRESSURE… - FOR NATIONAL WEALTH: GNP - FOR LEVEL OF CRIMINAL ACTIVITY: NUMBER OF CRIMINAL TRIALS, NUMBER OF COMPLAINTS TO THE POLICE… - FOR TRANSLATION QUALITY: NUMBER OF ERRORS AND OMISSIONS, NUMBER OF COMPLAINTS FROM USERS - FOR QUALITY OF TRANSLATOR TRAINING PROGRAM: PROPORTION OF GRADUATES WHO FIND WORK AS TRANSLATORS

5. D.Gile citations as indicators5 IMPORTANT FEATURES OF INDICATORS (1) - VALIDITY (DO THEY MEASURE WHAT THEY ARE SUPPOSED TO MEASURE?) - CONVENIENCE AND COST OF ACCESS (HOW LONG DOES IT TAKE TO MEASURE THEM? HOW MUCH MONEY/MANPOWER etc. DOES IT COST?) - EASE/COST OF PROCESSING (ONCE YOU HAVE THEM, IN ORDER TO DO SOMETHING USEFUL WITH THEM, HOW MUCH EFFORT AND HOW MANY RESOURCES ARE REQUIRED?)

6. D.Gile citations as indicators6 IMPORTANT FEATURES OF INDICATORS (2) - COVERAGE (WHAT PROPORTION OF THE RELEVANT PHENOMENON IN THE ENTITY TO BE STUDIED DO THEY COVER?) - SENSITIVITY (HOW MUCH OF THE SIGNAL OR PHENOMENON THEY ARE SUPPOSED TO PICK UP THEY ACTUALLY DO PICK UP?) - SELECTIVITY (DO THEY PICK UP THE TARGET SIGNAL OR PHENOMENON ONLY, OR DO THEY ALSO PICK UP OTHER SIGNALS WHICH INTERFERE WITH THE EXPLORATION PROCESS?) COMPLAINTS FOR CRIMINAL ACTIVITY? VAL/COV/SENS/SEL COMPLAINTS OF TRANSLATION USERS? VAL/COV/SENS/SEL PROPORTION OF GRADUATES WHO FIND EMPLOYMENT? VAL/COV/SENS/SEL

7. D.Gile citations as indicators7 CITATIONS WHEN INVESTIGATING TS AS AN OBJECT OF STUDY: WRITING NORMS IN SCIENCE IMPOSE REFERENCES TO AUTHORS AND WORKS USED THESE TAKE THE FORM OF CITATIONS Operational definition: EXPLICIT REFERENCE IN A RESEARCH TEXT TO ANOTHER AUTHORS PUBLISHED TEXT i.e. NAME OF AUTHOR AND PARTICULAR TEXT WITH REFERENCES

8. D.Gile citations as indicators8 CITATIONS AS AN INDICATOR OF INFLUENCE VALID? EASE AND COST OF ACCESS? EASE AND COST OF PROCESSING? COVERAGE? SENSITIVITY? SELECTIVITY?

9. D.Gile citations as indicators9 IN THE FIELD CONSIDERED VALID WIDELY USED By GOVERNMENTS, SOCIOLOGISTS OF SCIENCE… IN TS: COVERAGE INSUFFICIENT SENSITIVITY PERHAPS INSUFFICIENT CORRELATION BETWEEN NUMBER OF CITATIONS AND ACTUAL INFLUENCE PERHAPS NOT STRONG ENOUGH

10. D.Gile citations as indicators10 TENTATIVE METHODOLOGICAL IMPROVEMENT - 3 LEVELS OF INFLUENCE OF A CITED TEXT? L1: In passing, as illustration, historical reference, NO VISIBLE IMPACT on citing authors thought or work, judging by the citing text L2: SOME IMPACT: Clear signs in citing text that citing author has thought about the cited text significantly or is building on it L3: CENTRAL in citing text EVALUATE INFLUENCE OF CITED TEXTS IN CITING TEXT

11. D.Gile citations as indicators11 3 LEVELS OF INFLUENCE: IS IT A PROGRESS? PERHAPS TOO MUCH VARIABILITY? PERHAPS DIFFICULT TO CATEGORIZE? HAVE TO CHECK IF NOT GOOD ENOUGH, IMPROVE DEFINITION IF NOT GOOD ENOUGH, GIVE IT UP CHECK IN CLASS WITH PAPERS SEE % OR AGREEMENT, DISAGREEMENT IF OK, COMPARE RESULTS WITH 3 LEVELS AND WITHOUT

12. D.Gile citations as indicators12 3 LEVELS OF INFLUENCE: TESTS? A NUMBER OF TESTS WITH COLLEAGUES TAKE PUBLISHED TEXT ASK PANEL OF RESEARCHERS TO ASSESS LEVEL CHECK CONVERGENCE IF CONVERGENCE, OK, IMPLEMENT (IF COLLECTION COST NOT TOO HIGH) IF NOT ENOUGH CONVERGENCE - TRY TO IMPROVE OPERATIONAL DEFINITION - OR INFER THAT THIS IMPROVEMENT, HOWEVER ATTRACTIVE CONCEPTUALLY, DOES NOT REALLY WORK

13. D.Gile citations as indicators13 OPERATIONALIZING CITATION ANALYSIS TO MEASURE AUTHOR INFLUENCE - DETERMINE SAMPLING METHOD AND SAMPLE SIZE (TYPE OF CITING TEXTS, PERIOD, LANGUAGES, GEOGRAPHIC AND SOCIOLOGICAL IDENTITY…) IF THESE CONSIDERATIONS ARE NOT TAKEN UP, PROBLEM OF REPRESENTATIVITY -PROCESSING (SIMPLE COUNTS? PROPORTIONS? NUMBER OF TIMES CITED OR NUMBER OF PAPERS IN WHICH CITED?...)

14. D.Gile citations as indicators14 WHAT QUESTIONS CAN CITANAL HELP WITH? INITIAL CONTRIBUTIONS - WHO ARE THE MOST INFLUENTIAL AUTHORS AT TIME t? IN TS? IN A PARTICULAR BRANCH OF TS? ON A PARTICULAR TOPIC? WITH A PARTICULAR METHODOLOGY? - SAME QUESTION ABOUT GROUPS, SCHOOLS - EVOLUTION OF INFLUENCE OVER TIME - INFLUENCE OF PUBLICATION MEDIUM (PAPERS IN JOURNALS, PAPERS IN COLLECTIVE VOLUMES, MONOGRAPHS, THESES AND DISSERTATIONS) - INFLUENCE OF LANGUAGE OF PUBLICATION …

15. D.Gile citations as indicators15 WHAT QUESTIONS CAN CITANAL HELP WITH? ASSESSING QUALITY? ARE CITATIONS A VALID INDICATOR FOR RESEARCH QUALITY? QUESTION: ARE THE MOST OFTEN CITED AUTHORS IN TS/IN OTHER DISCIPLINES THE ONES WHO DO THE BEST RESEARCH?

16. D.Gile citations as indicators16 WHAT QUESTIONS CAN CITANAL HELP WITH? DEVELOPING THEIR USE - INTERACTIONS BETWEEN GROUPS (INCLUDING INTERDISCIPLINARITY) COLLECT INFORMATION ON CITED TEXT - FROM TS OR OTHER DISCIPLINES? Gile 2002/2006 TYPES OF CITATIONS - THEORIES - FINDINGS - RESEARCH METHODS - OPINIONS - TERMS… INDICATIONS ON QUALITATIVE ASPECTS OF TS RESEARCH EMPIRICAL/NON EMPIRICAL

17. D.Gile citations as indicators17 GENERAL METHODOLOGICAL NOTE:THRESHOLDS ASSUME EMPIRICAL: FINDINGS, METHODOLOGY, REFERENCES TO EMPIRICAL STUDIES NON EMPIRICAL: TERMS, OPINIONS, IF 100% EMPIRICAL NO PROBLEM IF 1% EMPIRICAL NO PROBLEM IF 50% EMPIRICAL? IF 30% EMPIRICAL? NECESSARILY SUBJECTIVE/CONVENTIONAL THRESHOLDS THIS IS COMMON TO MANY EMPIRICAL PROJECTS WITH METRICS SUCH AS PAUSE LENGTHS, NUMBER OF E/OS, etc. AT SOME TIME, SUBJECTIVE DECISIONS COME IN

18. D.Gile citations as indicators18 GENERAL METHODOLOGICAL NOTE: CLASSIFICATIONS FOR MANY USES OF INDICATORS IN EMPIRICAL RESEARCH, CLASSIFICATION IS CENTRAL PRACTICAL ISSUE: NOT EVERY PHENOMENON ENOUNTERED IS CLASSIFIABLE: - MISSING INFORMATION -BORDERLINE CASES WHAT DOES ONE DO? CONVENTIONS, PERSONAL DECISIONS, INFORM READER PRACTICAL ASSIGNMENT TO GAIN FAMILIARITY WITH PROBLEMS ?

19. D.Gile citations as indicators19 ASSIGNMENT (1) CHOOSE AT LEAST 3 SPECIFIC RESEARCH QUESTIONS FOR WHICH YOU BELIEVE CITATIONS ARE A GOOD INDICATOR Or THE EXPLORATORY QUESTION: WHAT CAN I FIND OUT ABOUT MY SAMPLE FROM CITATIONS? CHOOSE APPROPRIATE SAMPLE OF TEXTS TO TRY TO ANSWER THE QUESTIONS - At least 20 papers with a total of at least 200 citations if specific research questions - At least 5 papers with a total of at least 50 citations if your question is exploratory

20. D.Gile citations as indicators20 ASSIGNMENT (2) -STUDY THE SAMPLE REPORT ALL THE CITATION DATA ON EXCEL FILE (SEE SUGGESTIONS LATER IN THIS PRESENTATION) -REPORT RESULTS AND ANSWER TO QUESTION(S) ON SEPARATE WORD FILE - REPORT CAREFULLY ALL METHODOLOGICAL PROBLEMS YOU ENCOUNTERED AND ALL METHODOLOGICAL LESSONS YOU LEARNED FROM THE EXERCISE

21. D.Gile citations as indicators21 SUGGESTIONS FOR COLUMNS OF INFO ON DATA EXCEL FILE (1) 1. Citing author 2. (Affiliation) 3. Year of citing paper 4. References of citing paper 5. Page in which citation found 6. Cited author 7. (Affiliation of cited author) 8. (Discipline of cited author) 9. Year of cited reference 10.Cited reference

22. D.Gile citations as indicators22 SUGGESTIONS FOR COLUMNS OF INFO ON DATA EXCEL FILE (2) 11. Type of medium (thesis, dissertation, article, article in collective volume, monograph, internet document…) 12. Language 13. Year of cited reference 14. Cited reference 15. Type of reference (opinion, definition of concept, term, finding, methodology, reference to empirical study, plain reference to publication, theory,…) 16. Other comments or key words