Presentation on theme: "18 April 2008Copyright Ganesha Associates 2008 1 Basic reading, writing and informatics skills for biomedical research Segment 9. Research assessment trends."— Presentation transcript:
18 April 2008Copyright Ganesha Associates 2008 1 Basic reading, writing and informatics skills for biomedical research Segment 9. Research assessment trends
18 April 2008Copyright Ganesha Associates 2008 2 The need for assessment in science “Afinal das contas é tudo uma questão de transformar conhecimento em riqueza e bem- estar para a população.” “When I was in graduate school I thought I would go into a room and people would slide money through a slot in the door, I would do science, and then slide the results back out. But as my career has gone on, they are increasingly opening the door and asking me what I am doing with their money.”
18 April 2008Copyright Ganesha Associates 2008 3 O Cientista Brasileiro é avaliado ?! –À primeira vista, o papel do cientista parece ser tão-somente expandir as fronteiras do conhecimento humano. Porém, o contrato social da ciência prevê que a informação gerada pelo cientista deva ter por fim modificar a realidade para a qual ele produziu a informação. Mas isso nem sempre ocorre. E a avaliação é um processo que possibilita à sociedade e aos gestores de pesquisa meios de aferir a real efetividade do trabalho do cientista. –O cientista, também, está sempre solicitando mais verbas ao governo para publicar cada vez mais informação. E a avaliação, novamente, é uma importante ferramenta para que se concedam verbas aos mais produtivos e inventivos. –Assim, a avaliação do cientista deve considerar o cotidiano do pesquisador, respeitando as demandas da sociedade, que nem sempre são apenas publicar artigos e livros que só interessarão a outros cientistas. –Atualmente, o desempenho dos cientistas é avaliado através das citações por eles obtidas, por os artigos publicados só nas revistas indexadas no Science Citation Index (SCI), ou Medline ou Scielo. –Taken from Pedrini, Alexandre de Gusmão, 2005
18 April 2008Copyright Ganesha Associates 2008 4 Como o/a cientista é avaliado ?! –National programs for scientific research have targets –National wealth and well-being –‘Basic science with a purpose’ –Getting funding is a highly competitive process –The scientist has an obligation to deliver results –We need management tools in science. –The quality of scientific output is one measure –Methods of scientific assessment include the analysis of citation statistics
18 April 2008Copyright Ganesha Associates 2008 5 Brazilian scientific output – molecular biology and biochemistry
18 April 2008Copyright Ganesha Associates 2008 6
18 April 2008Copyright Ganesha Associates 2008 7 The quality of (inter)national scientific research is at the heart of the problem Data from a CAPES report 2005
18 April 2008Copyright Ganesha Associates 2008 8
18 April 2008Copyright Ganesha Associates 2008 9
18 April 2008Copyright Ganesha Associates 2008 10 Qualis e a classificação de periódicos, anais, revistas e journais Qualis é o resultado do processo de classificação dos veículos utilizados pelos programas de pós-graduação para a divulgação da produção intelectual de seus docentes e alunos Tal processo foi concebido pela CAPES para atender as necessidades específicas do sistema de avaliação e baseia-se nas informações fornecidas pelos programas e pelo Coleta de Dados A classificação é feita ou coordenada pelo representante de cada área e passa por processo anual de atualização. Os veículos de divulgação citados pelos programas de pós- graduação são enquadrados em categorias indicativas da qualidade - A, B ou C e do âmbito de circulação dos mesmos - local, nacional ou internacional As combinações dessas categorias compõem nove alternativas indicativas da importância do veículo utilizado, e, por inferência, do próprio trabalho divulgado.
18 April 2008Copyright Ganesha Associates 2008 11
18 April 2008Copyright Ganesha Associates 2008 12 CAPES – scientific subdivisions –Ciências Biológicas I: Biologia Geral, Botânica, Genética, Oceanografia Biológica e Zoologia –Ciências Biológicas II: Biofísica, Biologia Molecular, Bioquímica, Farmacologia, Fisiologia, Morfologia, Anatomia, Histologia e Neurociências –Ciências Biológicas III: Microbiologia, Imunologia e Parasitologia –Saúde Coletiva –Ecologia e Meio Ambiente
18 April 2008Copyright Ganesha Associates 2008 13 Critérios de Classificação do Qualis Ciências Biológicas III: Microbiologia, Imunologia e Parasitologia QUALIS A - Trabalhos publicados em revista com fator de impacto (F1) igual ou maior que 1,945 QUALIS B - Trabalhos publicados em manuscrito com F1 entre 1 e 1,945 trabalhos publicados nas Memórias do Instituto Oswaldo Cruz e no Brasilian Journal of Medical and Biological Research QUALIS C - trabalhos publicados em revistas nacionais ou internacionais com F1 menor que 1.
18 April 2008Copyright Ganesha Associates 2008 14 Define: indexação A indexação de uma revista é o processo que confere o indicador de qualidade necessário para que esta possa pertencer a um ou mais bancos de dados de renome internacional, em uma área específica do conhecimento, como por exemplo, MEDLINE, ISI, LILACS, etc., segundo critérios preestabelecidos por essas instituições. É um fator importante para que a produção científica nela apresentada se torne conhecida e reconhecida internacionalmente. O periódico necessita, para isto, estar de acordo com os padrões básicos de apresentação formal e excelência de conteúdo científico, determinados internacionalmente.
18 April 2008Copyright Ganesha Associates 2008 15 As fontes de ‘indexação’ em ordem de ‘qualidade’ Google Scirus LILACS ScieLo PubMed/Medline Scopus Thomson Web of Science E muitos outros mais especializados
18 April 2008Copyright Ganesha Associates 2008 16
18 April 2008Copyright Ganesha Associates 2008 17 Journal selection process - ISI To properly reflect the global context in which scientific research takes place, and to provide balanced coverage in each category, ISI seeks to cover the best regional journals as well. However, rather than compare a regional journal with all journals in a particular subject category, the ISI editor considers it in terms of those journals in the category from the same geographic area. High journal publishing standards, especially timeliness, and English language bibliographic elements remain essential.
18 April 2008Copyright Ganesha Associates 2008 18 Journal selection process - Medline Scope and coverage: The journals brought to the Committee for review will contain articles predominantly on core biomedical subjects. –Journals whose content is predominantly a subject peripheral or related to biomedicine are occasionally brought to the Committee when they have some biomedical content. – In these cases, the Committee's advice is sought not only on the quality of the content but also on the contribution it makes to the coverage of the subjects in question. Generally, such journals will not be indexed if their biomedical content is already adequately covered. Quality of content: Scientific merit of a journal's content is the primary consideration in selecting journals for indexing. The validity, importance, originality, and contribution to the coverage of the field of the overall contents of each title are the key factors considered in recommending a title for indexing, whatever the intended purpose and audience.
18 April 2008Copyright Ganesha Associates 2008 19 Journal selection process - Medline Quality of editorial work: The journal should demonstrate features that contribute to the objectivity, credibility, and quality of its contents. These features may include information about the methods of selecting articles, especially on the explicit process of external peer review; statements indicating adherence to ethical guidelines; evidence that authors have disclosed financial conflicts of interest; timely correction of errata; explicit responsible retractions as appropriate; and opportunity for comments and dissenting opinion. Neither the advertising content nor commercial sponsorship should raise questions about the objectivity of the published material. Sponsorship by national or international professional societies may be considered. Production quality: Quality of the layout, printing, graphics, and illustrations are all considered in assessing a journal. Though not a requirement for selection, journals of archival importance should be printed on acid-free paper. For detailed information concerning the use of acid-free paper please refer to the Acid-Free Paper for Biomedical Literature Fact Sheet.Acid-Free Paper for Biomedical Literature
18 April 2008Copyright Ganesha Associates 2008 20 Journal selection process - Medline Foreign language journals: Criteria for selection are the same as for those written in English. In order to extend the accessibility of the journal’s content to a wider potential readership, the majority of published articles in the review issues must contain an English-language abstract before the title will be considered for possible indexing.
18 April 2008Copyright Ganesha Associates 2008 21 Journal selection process - SciELO Originality. Journals should publish the results of original scientific research which contributes significantly to knowledge in the area covered by the journal Peer review. The journal editor should specify how its peer review process will be applied for the approval of contributions Editorial board. The constitution of a journal's editorial board must be public. Its members should be renowned specialists from the country and abroad. Frequency. In the life sciences, a journal should be published at least four times a year and contain a minimum of 32 articles Lifetime. A journal must have published at least four issues. Timeliness of publication. A journal must be publishing on time according to its stated frequency. English. Journal articles must provide English translations of titles, abstracts, and keywords. Standardization. The journal must specify which standards it follows for editorial presentation of articles, e.g. bibliographical references style
18 April 2008Copyright Ganesha Associates 2008 22
18 April 2008Copyright Ganesha Associates 2008 23
18 April 2008Copyright Ganesha Associates 2008 24 The importance of ISI Impact Factors –What is a citation ? A citation for an article in a journal includes the author, title of article, title of periodical (also referred to as the source), volume and issue number, page number(s), and date of publication. A single citation is called a reference; a group of citations is a bibliography. –How are Impact Factors defined ? It is calculated each year by the commercial publisher Thomson for those journals which it tracks, and are published in the Journal Citation Report.Impact Impact Factors have a huge, but controversial, influence on the way published scientific research is perceived and evaluated. Originally intended to provide librarians with a relative index of importance for journals in a particular field in order to assist collection management
18 April 2008Copyright Ganesha Associates 2008 25 Impact Factor defined “The Impact Factor of a journal is probably the only quantitative way of assessing its worth and relevance to the academic community it serves.” Editorial, Journal of Computer-assisted Learning, 2001 Impact Factor 2008 = # citations received in 2008 to articles published in 2007/2006 ______________________________________________________ # number of source items published 2007 and 2006
18 April 2008Copyright Ganesha Associates 2008 26 Alternatives ? The Hirsch Index is rapidly gaining favor as a performance measurement tool for scientific authors. A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np - h) papers have no more than h citations Thus, the h-index reflects both the number of publications and the number of citations per publication. The index is designed to improve upon simpler measures such as the total number of citations or publications. Like the IF, the index works properly only for comparing scientists working in the same field. For more information see Scopus or Wikipedia
18 April 2008Copyright Ganesha Associates 2008 27 H-index - weaknesses The h-index is bounded by the total number of publications. This means that scientists with a short career are at an inherent disadvantage, regardless of the importance of their discoveries. For example, had Albert Einstein died in early 1906, his h-index would be stuck at 4 or 5, despite his being widely acknowledged as one of the most important physicists, even considering only his publications to that date.
18 April 2008Copyright Ganesha Associates 2008 28 Impact Factors and Authors Correlation between article citation rate and journal impact for four authors
18 April 2008Copyright Ganesha Associates 2008 29 Impact Factors and Journals Citation rates in 1986 or 1987 of articles published in three biochemical journals in 1983 or 1984, respectively. From Seglen, BMJ 1997;314:497
18 April 2008Copyright Ganesha Associates 2008 30 Impact Factors over time
18 April 2008Copyright Ganesha Associates 2008 31 Impact Factor and area
18 April 2008Copyright Ganesha Associates 2008 32 Impact Factors for Microbiology
18 April 2008Copyright Ganesha Associates 2008 33 Impact Factors for Neuroscience
18 April 2008Copyright Ganesha Associates 2008 34 Impact Factors and article type
18 April 2008Copyright Ganesha Associates 2008 35 Impact Factor - Summary points Use of journal impact factors conceals the difference in article citation rates Journals' impact factors are determined by technicalities unrelated to the scientific quality of their articles Journal impact factors depend on the research field: high impact factors are likely in journals covering large areas of basic research with a rapidly expanding but short lived literature that use many references per article Article citation rates determine the journal impact factor, not vice versa
18 April 2008Copyright Ganesha Associates 2008 36 The emergence of Open Access Clockwise: Harold Varmus, Michael Eisen, Pat Brown and David Lipman
18 April 2008Copyright Ganesha Associates 2008 37 Scientific publishing is a very profitable commercial business
18 April 2008Copyright Ganesha Associates 2008 38 Even when it is not-for-profit
18 April 2008Copyright Ganesha Associates 2008 39 Define: business model A business model (also called a business design) is the mechanism by which a business intends to generate revenue and profits. It is a summary of how a company plans to serve its customers. It involves both strategy and implementation Wikipedia
18 April 2008Copyright Ganesha Associates 2008 40 Exponential growth in the amount of data being published Subscription charge increases: 200% in last ten years - pressure on library budgets, reduction in number of subscriptions Publisher retention of copyright and therefore control “ Excessive” profits: 35%+ margins Restrictive online access - the bundle deal Some fundamental problems with the ‘print’ business model
18 April 2008Copyright Ganesha Associates 2008 41 Key features of the ‘open access’ business model Free, immediate access online Unrestricted use Author retains copyright Papers are deposited in a public online archive Author pays for publication
18 April 2008Copyright Ganesha Associates 2008 42 Advantages of Open Access Maximum impact for authors access to the largest possible audience New ways to access and use literature full-text searching and mining Greatly expanded access to research for scientists, educators, physicians, the public
18 April 2008Copyright Ganesha Associates 2008 43 Old ‘print’ subscription business model Researcher Publisher Reader Pay-per-viewSubscription Library Gov Funders Institutions Industry € €€€ €
18 April 2008Copyright Ganesha Associates 2008 44 New open access business model Researcher Publisher Reader € Public Digital Library Gov Funders Institutions Industry €
18 April 2008Copyright Ganesha Associates 2008 45 Open access a major force in Brasil
18 April 2008Copyright Ganesha Associates 2008 46 SciELO’s role in supporting OA
18 April 2008Copyright Ganesha Associates 2008 47 Consequences of Open Access A fundamental change in the business model underlying the scientific publishing industry Revenue: from subscription to fee-for-service Payer: was ‘the librarian’, now the research funder, usually the government, ultimately the tax-payer Marketing focus: was the librarian, now the researcher (author/reader) Future business driver: accountability to the tax payer
18 April 2008Copyright Ganesha Associates 2008 48 Some myths about Open Access Urban myth 1: “ISI doesn’t cover Open Access journals” ISI (Thomson Scientific) indexes around 200 Open Access journals (OAJs) for its Web of Science service. It applies the same criteria for inclusion in that index to Open Access journals as it does to subscription-based titles. Urban myth 2: “Open Access journals are not good quality” By dint of being included in the Web of Science index, around 200 Open Access journals enjoy the prestige of having an official Impact Factor, thus denoting some measure of quality to these titles. Moreover, some OAJs enjoy particularly high impact factors: the most notable in this respect, PLoS Biology, romped straight to the top position in the General Biology category the first year it qualified for such a calculation (journals must have been published for at least two years before an impact factor can be calculated). Urban myth 3: “Paying a publication fee to Open Access journals corrupts peer review” This would be very significant and deeply damaging to the integrity of the scholarly communication system were it true. It is not. Open Access journals that charge an article-processing fee have a firewall between the peer review process and the charge-imposition system, ensuring that authors cannot pay to subvert the peer review procedure in a manner akin to vanity publishing. Urban myth 4: “Open Access journal business models are unsustainable” Well, they may yet prove to be, but time is passing and we are seeing a move towards Open Access publishing not away from it, suggesting that some publishers at least are finding ways to sustain their publications even while making their content free online.. Alma Swan, American Scientist, 2006
18 April 2008Copyright Ganesha Associates 2008 49 What is Open Access not ? A movement – then who is the leader ? Free – electronic publishing, whatever the business model, is expensive Robust – benefits are being mis-sold to the main payer, and there is a real risk of loss of momentum A big issue for the majority of researchers - “…researchers do not consider that problems in accessing the journal literature are a significant barrier to their work…” Publishing Research Consortium 2006 Good news for national publishers in countries such as Brasil Just about documents and open archiving
18 April 2008Copyright Ganesha Associates 2008 50 Benefits of Open Access Access to full text. Better indexation and retrieval. Better integration between full text OA resources and other public archives of scientific data The potential for the emergence of new services based upon open access content or overlaying open access content (e.g. peer review services, text- and data-mining tools, bibliometrics and webometrics for better research evaluation, etc The potential for much wider access than the subscription publishing system gives for doctors/nurses, teachers/students, small biotech companies, etc Redistribution of publishing costs to richer nations Adapted from The Economic Impact of Enhanced Access to Research Findings by John Houghton & Peter Sheehan, Centre for Strategic Economic Studies, Victoria University, Melbourne, 2006
18 April 2008Copyright Ganesha Associates 2008 51 Big publishers fight back New Scientist recently published a plan by PR firm Dezenhall Resources detailing how traditional science publishers could turn government opinion against open-access journals.a plan There’s nothing wrong with employing a famously tough PR firm, but the memo reveals that Dezenhall recommended sidestepping the real issues around peer review and the business model that powers it, in favor of “rhetorical campaign points” about Big Government and scientists’ hypocrisy.famously toughpeer reviewthe business model The memo notes that the key issue is that “publishers are trying to protect their businesses and the integrity of the research they publish.” Yet, two of Dezenhall’s example messages focused on government “censorship” and “nationalization” of science. No one really believes, on either side, that government is taking over science publishing. The catalytic bill, Federal Research Public Access Act (S.2695), would only require that articles based on federally funded research be opened to public inspection within 6 months of publication.Federal Research Public Access Act
18 April 2008Copyright Ganesha Associates 2008 52 Impact factors drive submissions, (PLOS) First impact factor announced
18 April 2008Copyright Ganesha Associates 2008 53 Web access drives impact factor
18 April 2008Copyright Ganesha Associates 2008 54 Open Access – beyond the document Synthesize evidence systematically from many studies and teams of investigators and anticipate this integration from the design phase of research. –John Ioannides, NIH 2006 “Upload and share your raw data, and have a high impact factor for your blog — or perish? That day has not yet come, but web technologies, from personal publishing tools such as blogs to electronic laboratory notebooks, are pushing the character of the web from that of a large library towards providing a user-driven collaborative workspace”, –Nature editorial, Dec 2005 “…the 48% of trials which made their data available received 85% of the total citations… –Heather Piowar, March 2007
18 April 2008Copyright Ganesha Associates 2008 55 Open Archiving ACS has re-announced its Author Choice model to draw attention to the fact that authors can now automatically self-archive through payment of the Author Choice fee. Elsevier has announced a deal with the Howard Hughes Medical Institute (HHMI) which allows HHMI to pay a fee to have all of its funded research results which happen to be published in Elsevier journals automatically archived in PubMed six months after publication. In both instances the publisher has taken the lead, as arguably publishers historically always have, to ensure accessibility. In both instances, the idea of fee for service is maintained in the relation - the deals reduce bureaucratic administration costs and increases efficiency: Elsevier got $1,000 for every Cell Press article, and $1,500 for each article from other Elsevier journals. In general terms, the service encourages researchers with HHMI grants to dispatch their "obligations? to Open Access in this painless way, distracts them from other thoughts about institutional repositories, and encourages the use of Elsevier journals for publication. Presumably therefore other publishers will follow suit with HHMI. David Worlock, EPS, April 2007
18 April 2008Copyright Ganesha Associates 2008 56 Performance of OA medical journals (ISI)
18 April 2008Copyright Ganesha Associates 2008 57 Practical activity 9a - Research assessment Total duration ca. 30 minutes. Obtain Impact Factors for your top 5 journals and add them to your Excel spreadsheet. Send them as an attachment to the course organiser. Practical activity 9b - Why research is often wrong. Total duration ca. 2 hours. Read the New York Times article about the problems of making strong associations between diet and disease.New York Times Now read the Lancet Editorial about the drug Aprotinin. You may need to read some of the more recent literature on aprotinin to understand the full relevance of this second paper.Aprotinin