On Standards in Science Metrics and Classifications Henk F. Moed Elsevier, Amsterdam, Netherlands Workshop on Science Metrics, Classifications, and Mapping Standards, August 11-12, 2011 School of Library and Information Science, Indiana University, Bloomington, USA

Workshop on "Bibliometric Standards", River Forest, Illinois, USA (11/06/1995) SubjectTopic Metrics (mathematical- statistical aspects) Macro indicators; journal impact factors; relative citation rates; ClassificationsDocument type; subject classifications; years; author and institutional names Terminology, concepts (theoretical aspects) Quality; impact; visibility; productivity; activity index; Research methodologyModels; normative principles; ‘Good’ analysis practices

Workshop on "Bibliometric Standards", River Forest, Illinois, USA (11/06/1995) 1.Standards can not be set by committee but must evolve through an on-going debate 2.Perhaps, the Scientometric community needs a refereed forum more dedicated to methodological issues 3.Often technical controversies reflect theoretical issues

Standardisation is (also) Providing frameworks in which various approaches can be positioned and compared with one another Example 1: Research assessment indicators Example 2: Subject classifications

Example 1: The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi ‐ dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentIncrease multi-discipl. research Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com- pletion rates

Multi ‐ dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentIncrease multi-discipl. research Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com- pletion rates Read column- wise

Multi ‐ dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators IndividualAllocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentIncrease multi-discipl. research Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com- pletion rates

Multi ‐ dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentIncrease multi-discipl. research Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion & hiring Research infrastruct. citation ‘prestige’ PhD com- pletion rates

MD-RAM: Example 1  Individual  Hiring/promotion  Productivity & impact  Individual  Hiring/promotion  Productivity & impact  PhD date, place, supervisor;  Invitations for conferences  PhD date, place, supervisor;  Invitations for conferences  Publications in international jrnls;  Actual citation impact  Publications in international jrnls;  Actual citation impact

Multi ‐ dimensional Research Assessment Matrix (Part) Unit of assessment PurposeOutput dimensions Bibliometric indicators Other indicators Individual Allocate resources Research productivity PublicationsPeer review Research group Improve performance Quality, scholarly impact Journal citation impact Patents, licences, spin offs DepartmentIncrease multi-discipl. research Innovation and social benefit Actual citation impact Invitations for conferences InstitutionIncrease regional engagement Sustainabi- lity & Scale Internat. co- authorship External research income Research field Promotion, hiring Research infrastruct. citation ‘prestige’ PhD com- pletion rates

MD-RAM: Example 2  Research group  Allocate resources  Res. productivity & impact  Research group  Allocate resources  Res. productivity & impact  Competitive research income;  Ratio research active/total staff  Competitive research income;  Ratio research active/total staff  Publications in international jrnls;  Actual citation impact  Publications in international jrnls;  Actual citation impact

Positioning bibliometric products Unit of assessment PurposeAspect Product Research group Improve performance; Allocate resources Productivity/ impact A Department; institution Allocate resources; Stimulate multi- disciplinary research Research infrastructure and potential B InstitutionStimulate multi- disciplinary research; Improve performance; Social benefits; sustainability; Prod./impact C Group, Dept, Institution Improve performance & regional engagement; Allocate resources; Productivity/ impact; social impact; infrastr. D

Example 2: Journal subject classification systems

Journal subject classifications based on..... Semantic words in journal titles Journal-to-journal citations Journal co-citation analysis Journal bibliographical coupling Journal co-usage analysis Semantic words from article titles and abstracts Thesaurus subject terms in disciplinary databases

Three citation relationships A B A C A B B C A cites B B is a cited reference in A B and C are co-cited by A A and B are bibliographically coupled via C

Analogy Model CITATIONSUSAGE (Collections of) publishing authors (Collections of) users Citing a documentRetrieving the full text of a document ArticleUser session Author’s institutional affiliation User’s account name Number of times citedNumber of times retrieved as full text

Forming subject categories from journal titles Journal titleSubject category Journal of Modern Optics Optics Journal of Orthopaedic Science Orthopedics Legal and Criminological Psychology Law; Criminology; Psychology Journal of Social Work Social work

Subject categories: Examples “Linguistics” Nr Jrnls “Agriculture” Nr Jrnls BI- & MULTILINGUALISM12AGRICULTURAL ECONOMICS18 DIALECTOLOGY5AGRICULTURE138 DISCOURSE STUDIES10AGRONOMY26 LANGUAGE226AQUACULTURE12 LINGUISTICS217CROP SCIENCE18 LINGUISTICS, APPLIED17DAIRY SCIENCE10 PHILOLOGY38FISHERIES23 PHONETICS13FOOD S&T147 PRAGMATICS7HORTICULTURE19 SEMANTICS7PEST SCIENCE & PESTICIDE14 SEMIOTICS3POULTRY SCIENCE9

Proposal Use the classification based on journal title words as a benchmark as it represents a classification that everyone can grasp intuitively and illustrate the special features and added value of the new approach

Thank you for your attention

What is measured by citations and references? - 1 Principal author(s) References conceived asCitations measure Garfield Salton Descriptors of document content GarfieldManifestations of scholarly information flows Utility (quantity of information use) Small Garfield Elements in a symbol making process Highly cited items as concept symbols Merton Registrations of intellectual property and peer recognition of the knowledge clain Intellectual influence Cole & Cole Zuckerman Socially defined quality GilbertPersuasion toolsAuthoritativeness ZuckermanReferencing motives and their consequences are analytically distinct Citations are proxies of more direct measurements

What is measured by citations and references? - 2 Principal author(s) References conceived asCitations measure Martin & Irvine Manifestations of peer recognition, information flows and political pressures Impact (actual influence) CozzensReferences are at the intersect of the reward, rhetorical and communication system but rhetorics comes first Recognition, persuasiveness and awareness each generate a certain portion of variation in citation counts WhiteInter-textual relationships mainly reflect straightforward acknowledgement of related documents Co-citation maps provide an aerial view measuring a historical consensus as to impor- tant authors and works

What is measured by citations and references? - 3 Principal author(s) References conceived asCitations measure van RaanReferences are partly particularistic but in large ensembles biases cancel out The upper part of the distribution of a ‘thermodynamic’ ensemble of many citers measures ‘top’ research WoutersThe reference is the product of the scientist The citation is the product of the indexer. Validity of citations cannot be grounded merely in reference behaviour