1. Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal
Asha Balakrishnan
Vanessa Peña
Bhavya Lal Task Leader
November 5, 2011
American Evaluation Association, 2011

2. Relevant Task Goal
Federal agency wished to know if their set-aside potential transformative research (PTR) program was funding interdisciplinary research (IDR)
11/5/2011

3. Associated Study Questions
How interdisciplinary is the set-aside program’s research [as compared with traditional programs]?
How interdisciplinary is the body of knowledge on which the awarded proposal draws?
How interdisciplinary is the funded PI [as compared with PIs of traditional programs]?
How interdisciplinary is the PI’s prior publication history?
11/5/2011

4. Definition of Interdisciplinarity
Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem.
The critical indicators of interdisciplinarity in research include evidence that the integrative synthesis is different from, and greater than, the sum of its parts.
Process of knowledge integration is important
micro-combinations of models or global schemes that unify disparate approaches
Consulting and partnering modes, not multidisciplinary contracting of services
Coordinated and collaborative inputs and organizational framework
Formation of a new community of knowers with a hybrid interlanguage
Generation of new insights and disciplinary relationships
A larger, more holistic understanding of the core problem or question
Altered perspectives and revised hypotheses.
C.S. Wagner et al. / Journal of Informetrics 165 (2011) 14–26
11/5/2011

5. Operationalizing Interdisciplinarity – Concept of Diversity Score
What is the metric?
The “I-score” measures the number of disparate scientific fields connected by a researcher’s body of work. (Porter, 2007)
How is it measured?
By using the subject categories of the cited references. A co-citation analysis of all journals has been measured empirically.
What does it intend to represent?
The I-score intends to measure the interdisciplinarity of the researcher’s work.
Low I-scores indicate that the researcher draws primarily from a specialized body of work.
What do we do with this measure? Is a low I-score good or bad? It seems that at STPI, we are mostly interested in innovation and research that crosses different fields. Yet the i-score does not adequately capture either. What really seems to matter is how the knowledge produced in the paper is used—not necessarily the bodies of literature that the author draws from. For example, John Nash published (alone) mostly in mathematics and economics; yet his work is used by evolutionary biologists, political scientists, and researchers in marketing. While the literature he draws from isn’t necessarily interdisciplinary, his work does connect disparate fields if we examined who eventually used his research. This also might be an N of 1 observation.
11/5/2011

6. Evaluative Approach
I-Score metric was employed to test if differences existed between test program awards and comparison awards to determine if:
Test program is attracting more interdisciplinary PIs
Test program is soliciting more interdisciplinary proposals
11/5/2011

7. Comparison Group Selection
10/4/2010
Selection of Comparison Awards: To assess the difference between the proposals funded by test program and proposals funded through traditional mechanisms, we considered many options and then selected a set of funded proposals from programs as the comparison group. Selection Methodology:
Identified a set of comparison awards from funded active awards that were:
On the order of total funding of test program
On the order of total duration of test program
Randomly selected 44 “comparison” awards
44 “test program” awards
11/5/2011

8. proposal interdisciplinarity
11/5/2011

9. Measuring Interdisciplinarity of Proposals
10/4/2010
Measuring Interdisciplinarity of Proposals
Activity:
Gathered the cited references from the Treatment and Comparison Group proposals
Extracted journal titles from references and matched to Thomson ISI subject categories
Used techniques and tools described in Porter et.al., 2010* to compute integration scores
*Porter, et al. Practical research proposal and publication profiling. Research Evaluation 19(1): (2010).
11/5/2011

10. Caveats
We assume that the references cited in the awarded proposal represent the interdisciplinarity of the proposed ideas
Not all references cited in a proposal were indexed in ISI’s Web of Knowledge database (ranged from 25% to 90% coverage)
Subject category assignment of journals is a proxy for “discipline”
11/5/2011

11. Treatment Proposal Integration Scores are Higher than the Comparison Group Proposal Integration Scores
10/4/2010
Tests done, however error in correlation matrx unquantified, and therfore statistical tests are less meaningful
11/5/2011

12. Comparison Group FY2007-FY2009
10/4/2010
Visual Representation of Average Test and Comparison Group Proposal Integration Score
Test Program FY2007-FY2009
Average I-Score = 0.64
Comparison Group FY2007-FY2009
Average I-Score = 0.55
Clin Med
Agri Sci
Env Sci & Tech
Clin Med
Chem
Matl Sci
Matl Sci
Geo
Geo
Biomed
Biomed
Inf Dis
Inf Dis
Ecol Sci
Ecol Sci
Cog Sci
Psych
Psych
Hlth & Soc
Phys
Phys
Bus & Mngt
Eng
Bus & Mngt
Eng
Econ. Polit. & Geo
Comp Sci
Econ. Polit. & Geo
Comp Sci
11/5/2011

13. Distribution of Disciplines – Project Level
10/4/2010
I-Score = 0.64
I-Score = 0.68
I-Score = 0.65
11/5/2011

14. Researcher interdisciplinarity
11/5/2011

15. Measuring Principal Investigator Interdisciplinarity
10/4/2010
Measuring Principal Investigator Interdisciplinarity
Activity:
Searched and obtain each of the Treatment and Comparison Group Principal Investigator’s publication histories (from Jan 1980 to June 2010) in the ISI Web of KnowledgeSM database
Employed integration score technique based on Porter et.al* to determine integration score of each PI (44 test PIs and 44 Comparison group PIs)
For PIs with a low number of publications, sensitivity analyses were conducted to ensure conclusions still held
No known methods for measuring cognitive integration
*Porter, et al. Measuring researcher interdisciplinarity. Scientometrics 72(1): (2007).
11/5/2011

16. 10/4/2010
Caveats
Only journal publications available in ISI’s Web of Knowledge database included
Subject category assignment of journals is a proxy for “discipline”
Only measured the PI’s interdisciplinarity, not those of the Co-PI’s or the team
No known methods for measuring cognitive integration
11/5/2011

17. No Difference Between Treatment PI Integration Scores and Comparison Group PI Integration Scores
10/4/2010
Tests done, however error in correlation matrx unquantified, and therfore statistical tests are less meaningful
11/5/2011

18. Average Test and Comparison Group PI Integration Score
10/4/2010
Test Program FY2007-FY2009
Average PI I-Score = 0.50
Comparison Group FY2007-FY2009
Average PI I-Score = 0.48
Clin Med
Env Sci & Tech
Clin Med
Env Sci & Tech
Biomed
Matl Sci
Biomed
Matl Sci
Inf Dis
Chem
Inf Dis
Chem
Cog Sci
Geo
Cog Sci
Geo
Hlth & Soc
Phys
Bus & Mngt
Eng
Eng
Econ. Polit. & Geo
Comp Sci
Econ. Polit. & Geo
Comp Sci
11/5/2011

19. Summary of Findings
I-scores for test group PIs and comparison group PIs are not different
I-scores from test groups proposal cited references are higher than I-scores from comparison group proposal cited references
Set-aside PTR program is not attracting PIs with a more interdisciplinary background than other programs. However, program is successful attracting researchers to submit more interdisciplinary ideas!
11/5/2011

20. Methodological Finding: I-score does not tell the complete story
10/4/2010
Example
PI Maps
Proposal Maps
I-Score = 0.72
I-Score = 0.54
I-Score = 0.42
Who are these?
Backman, Wang, and Nurmikko
11/5/2011
I-Score = 0.64
I-Score = 0.73
I-Score = 0.41

21. Caveats: Study Based on Many Assumptions
I-score is a good measure of interdisciplinarity
Little is known about how subject categories are generated and how static they are over time
Journal coverage in databases is incomplete
Proposal cited references reflect knowledge integration
Rhetorical vs. reward citations
Hypotheses
PTR tends to be interdisciplinary
Researchers with a history of interdisciplinary work tend to conduct PTR research
(beginning to test) Younger PIs tend to be more interdisciplinary
11/5/2011

22. Future Questions of Interest
How interdisciplinary was the team (integrate all PI publications into one set and determine i-score)
Are younger PIs more interdisciplinary?
Testing on larger population of awards
What is the relationship between PI and proposal I-scores?
Exploratory look found none in the test group, and significantly positive for comparison group
What is the relationship between Proposal Integration Score and PTR Score?
Exploratory look found none
No Relationship between treatment group program PI and Proposal I-Scores
Significant Positive Relationship for Comparison Group PI and Proposal I-Scores
No Correlation between Proposal Integration Score and PTR Score
Possible that there is no linkage between the degree of PTR and IDR
IDR may be a poor proxy for cognitive integration
11/5/2011