2. Datum Identifying high impact scientific work using NLP and Psychology. Brett Buttliere discussing Whalen et al., (2015) Email: b.buttliere(aτ)iwm-tuebingen.de Twitter: @BrettButtliere : 30 June 2015 - ACM 15

3. OUTLINE Thank you, great stuff. Whalen et al., 2015: Responding to: –Keyword network –Characterizing single papers –Metrics of ‚good science‘ Two further thoughts for discussion: –Utilizing social media data –Incorporating psychology and the sci lit: Discussion! :D LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

4. WHALEN ET AL., 2015 Predicted citations outside of journal based upon citations and keyword distance inside the journal. Steps: –Created keyword network –Characterized keywords in papers –Examined distance of citations –Regressed on citations outside j. Find that especially distant citation are most predictive of impact outside the journal. –Because… LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

5. RESPONSE: KEYWORD NETWORK: Really excellent potential, can be used to: –Make better predictions on what to read. –Examine similar hypotheses at other levels (e.g., keyword, author). Should probably examine the temporal aspect. –Older papers cited more in and out of journal. –Papers with popular keywords more cited? –Beginning of keyword (pairing) starts new. Some clever metrics might help. –One for popularity? –One for where it is in that list? %? Not clear for me. LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

6. CHARACTERIZING SINGLE PAPERS Keywords are a strong start. What else is possible? –Page views –Page rank –Download to read to citation ratios –Types of citations in the paper –Sentiment of the paper –Year seems important. Also take into account: –Author –Keywords –Journal Which is best? –Need to compare –It is an open question, and as more come, need to examine relationships. LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

7. WHAT IS GOOD SCIENCE & ‚IMPACT‘? Especially distant citations within-journal were most predictive of impact outside the journal. Best impact metric citations outside of journal? –Inside journal citations may be better, no? –Especially for leading journals. Need more than one metric. ? –** Examine relationships between them. ** –Decisions made on what I value. What do i (we) care about…? From Christobal Cobo‘s blog LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

8. 2 FURTHER THOUGHTS USING OTHER TYPES OF DATA (e.g., SOCIAL MEDIA) USING THE SCIENTIFIC LITERATURE –ESPECIALLY PSYCHOLOGY LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

9. SOCIAL MEDIA MENTIONS Available data: –Page views –Mentions –Comments –Discussion on mentions –Favorites Like Whalen et al., Use words. –Keywords –Sentiment of comments Advantages: –Data available from release –Easily available At minimum, can be combined with other metrics. LEIBNIZ FUR WISSENSHAFT KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

10. Scientists are human. –Biases (egocentrism) –Meaning making (Festinger) What type of science is best? –Structures of scientific revolution (Kuhn) –Falsificationism (Popper) –Strong Inference(Platt) –Progress made through conflict? We are examining social media for evidence of this. USING MORE LIT., ESP. PSYCH! :D LEIBNIZ FUR WISSENSHAFT KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

11. SUMMARY Whalen et al. (2015) make an excellent start. –Would like to take temporality into account. –Nice metrics might help the endeavor. Popularity of keywords. Where it is in the series. Tried to take it further by: –Using more of the data available. –Applying Psychological theory. LEIBNIZ FUR WISSENMEDIEN KNOWLEDGE MEDIA RESEARCH CENTER - TÜBINGEN

12. Datum Questions- discussion Thank you! Email: b.buttliere(aτ)iwm-tuebingen.de Twitter: @BrettButtliere : 30 June 2015 - ACM 15