1. Evaluating (Scientific) Knowledge for people, documents, organizations/activities/communities ICiS Workshop: Integrating, Representing and Reasoning over Human Knowledge Snowbird August 9 2010 Geoffrey Fox gcf@indiana.edu http://www.infomall.org http://www.futuregrid.org http://pti.iu.edu/http://www.infomall.orghttp://www.futuregrid.orghttp://pti.iu.edu/ Director, Digital Science Center, Pervasive Technology Institute Associate Dean for Research and Graduate Studies, School of Informatics and Computing Indiana University Bloomington

2. My Role My research would be on building Cyberinfrastructure (MapReduce/Bigtable/Clouds/Information Visualization) for “Integrating, Representing and Reasoning over Human Knowledge” – Use FutureGrid to prototype Cloud/Grid environments Here I talk in role as frustrated journal editor and School bureaucrat responsible for advising faculty how to get NSF grants and tenure.

3. Knowledge Evaluation is Important? Review of journal or conference paper. – Several conference management systems but don’t offer reviewing tools Supporting choice of panel reviewing proposals – And proposal review itself Supporting choice of Program Committee for conference Supporting promotion and tenure process – h-index appears in several referee reports Supporting ranking of organizations such as Journals, Universities and (Computer Science) departments. Deciding if some activity is useful such as TeraGrid; particular agency or agency program; a particular evaluation process (panel v individual reviews) Deciding if some concept useful such as multidisciplinary research, theory, computing ……. Evaluation of Knowledge evaluation methodologies

4. “Policy Informatics” aka “Command & Control” (military knowledge) In Data-Information-Knowledge-Wisdom-Decision(Evaluation) pipeline, some steps are “dynamic” (can be redone if you save raw data) but decisions are often “final” or “irreversible” We could (and do as preprints) publish everything as “disks free” and change our evaluations But Finite amount of research funding and finite number of tenure positions

5. SS

6. Citation Analysis Use of Google Scholar (Publish or Perish) to analyze contribution of individuals is well established #papers, # citations, h-index, hc-index (contemporary), g- index (square)….. There is ambiguity as to “best metric” and if such metrics are sound at all but in some cases, perhaps most serious problem is calculating them in unbiased fashion One can probably find metrics for “Geoffrey Fox” but it’s hard for more common names and for example most Asian names are hard – Google Scholar has crude approach to refine by including and excluding names e.g. include “Indiana University” or exclude “GQ Fox” (not clear where words are?) “Automating” hard unless analysis for each name done by hand – Even the name nontrivial – need “GC Fox” and “Geoffrey Fox”

7. Evaluating Documents As journal editor, I find choosing referees (and persuading them to write report) as hardest problem – Especially with increasing number of non traditional authors Need to identify related work and find authors or previous referees of these related papers – Currently ScholarOne uses largely useless keyword system – Can also look at originality of articles examined from overlap in text between a given article and some corpus (typically resubmitting conference paper unchanged) If unfamiliar with authors need to identify which of a multi-author paper is appropriate choice, where they are now and contact information – Current services DBLP, ACM Portal, LinkedIn, Facebook don’t tell you necessary information Need tools to quantify reliability of referees

8. Is High Performance Computing Useful for improving Knowledge Are papers that use TeraGrid “better” than those that don’t? – Does TeraGrid help enhance Knowledge? Correlate quality and type of papers with “use of TeraGrid” Possibly can be done by text analysis (does paper acknowledge TeraGrid) Here use indirect map TeraGrid  Projects/People  Papers

9. TeraGrid Analysis I: Bollen

10. TeraGrid Analysis II: Bollen

11. TeraGrid Web of Science

12. Need a Freely Available Toolkit Firstly current tools as in Google Scholar and CiteSeer have insufficient scope – Google Scholar stuck in early stage of “perpetual beta (?alpha)” after killing Windows Academic Live Secondly need to enable customization so that can explore evaluation choices – Current CS department rankings put Indiana in dungeon – partly because Fox/Gannon papers are not counted as not in approved journals Don’t want to let Thomson to control Impact Factors (relevant for tenure especially in Asia?) without scientific scrutiny As discussed ScholarOne (also Thomson) is dreadful but seems to have growing adoption Want to explore new ideas such as evaluating TeraGrid

13. Tools Needed More accurate scientific profiles; ACM Portal says I have 3 publications; DBLP 250; Google Scholar 1000 – Neither tells you my contact and professional information Unbundled CiteSeer/Google Scholar allowing more accurate document analysis – e.g. Analyze document on hand (as in conference submission) – Open decomposition into Authors, Title, Institution, emails, Abstract, Paper, citations Analyzers of Citations and/or text to suggest referees Analysis of novelty of document Tool to produce accurate h-index (etc.)

14. Some Research Needed Open analysis of concepts like Impact Factor, h-index, Indexing services – Look at definitions and – Possibilities of making valid deductions How do we evaluate “groups” (research, departments) as opposed to individuals Can one automate the current time consuming manual steps – Identity confusion in Google Scholar – Research profiles Compare traditional ethnography approach to evaluation (do a bunch of interviews) versus data deluge enabled version Why are Web 2.0 tools like Delicious, Facebook etc. little used in science