1. S T P I Challenges and Emerging Tools for Research and Technology Portfolio Analysis Christina Viola Srivastava, Brian Zuckerman, Bhavya Lal Science and Technology Policy Institute November 2006

2. S T P I 2 What is Portfolio Analysis? Portfolio analysis refers to analysis and evaluative activities at an organizational level higher or broader than the traditionally defined “program” or “initiative” Portfolios can be defined by virtually any theme of interest to an organization, including administrative needs, organizational structure, funding streams, goals, and results

3. S T P I 3 Examples of Portfolios Potentially of Interest to NIH Theme CategoryPortfolio Examples Disease category Cancer, AIDS, Depression, Obesity Area of Science Virology, Nanomedicine, Genomics, Epidemiology Character of Research Basic, Applied, Pioneering, Transdisciplinary Use of Technology Cancer imaging, DNA microarrarys Relevance of Legal, Ethical, of Administrative Issues Human Subjects, Animal Research, Clinical Trials Purpose of Award Research, Training, Capacity Building Administrative Type/Mechanism/Size of Award K-Series Awards, R01s, Centers Grants, Multi-PI Awards, Awards Under $50K Organizational Entity Generating Award NDPA, NIAID, EID Program Geographic Location of Awardee(s) or Research Activities Africa, California, Western US, Boston Institution of Awardee(s) Large Research Universites, Historically Minority Institutions, Public Institutions, Harvard University, Harvard-Affiliated Hospitals Demographics of Awardee(s) Women, Minorities, Young Investigators, Clinician- Researchers, Students

4. S T P I 4 “Roles” for Portfolio Analysis Retrospective Evaluation Prospective Reporting –External –Internal Strategic Planning –Identify areas of overlap in existing programs –Identify gaps in existing portfolios relative to strategic goals or normative expectations –Understand the role of research portfolios relative to broader context

5. S T P I 5 Portfolio Analysis: Challenges 1.Conceptually integrate goals, activities, and outcomes across programs to ask the right questions 2.Integrate and analyze datastreams across multiple programs/initiatives in order to answer them

6. S T P I 6 Examples of Datastream Types for Research Portfolios Awardee Datastreams. Data related to awardees, institutions, and research programs prior to the award. Award Datastreams. Data related to award proposals and funding. Activity Datastreams. Data related to activities and expenditures supported by an award. Output/Outcome Datastreams. These data describe proximate and ultimate outputs, outcomes, and impacts of supported activites. Planning and Administrative Datastreams. Data related to program planning, changes over time, grant review, procedural and administrative processes, etc. Environmental Datastreams. Data related to fields of research and policy beyond direct agency control.

7. S T P I 7 Hypotheses about Datastreams and “Roles” for Portfolio Analysis Prospective reporting, which generally seeks to respond to a specific information request, will likely draw on relatively few datastream types and will often be limited to a single category. Restrospective evaluation questions will focus on cross-correlation of input datastreams (e.g. awards, awardees) with activity, output and outcome datastreams. Strategic planning will likely draw on all categories of datastreams in complex ways.

8. S T P I 8 Visualization as a Tool for Portfolio Analysis Visualization helps to integrate data, facilitating analysis of patterns that can lead to new insights We’re used to visualizing quantitative data, but new tools allow visualization of text-based data as well Requirements: –Large quantities of text-based data, multiple sources and formats ok –Appropriate visualization software –Skilled analyst with broad understanding of programs and field

9. S T P I 9 Example: South Africa’s Biomedical R&D Portfolio South Africa of interest in Sub-Saharan region because: –Research strengths in a variety of domains –Broad range of institutions involved in world- class research –Significant NIH investment What can we learn from existing data about South Africa’s research strengths the role NIH plays there?

10. S T P I 10 Methods and Data Sources IN-SPIRE™ (developed by Pacific Northwest National Labs) Visualization of text documents based on word distribution, frequency and proximity to other key words Data sources: –titles and abstracts of PubMed publications w/South Africa affiliation, 2003-05 –titles and abstracts of CRISP entries for South Africa, 1999-2005

11. S T P I 11 Visualizing Publications Shows Multiple Areas of Research Strength Data Source: Titles and abstracts of PubMed publications affiliated with institutions in South Africa, 2003-2005 (n=3952) heart disease virology HIV epidemiology, treatment, prevention, maternal and child health TB antibiotic resistance HPV/cervical cancer non-ID ob/gyn, women’s health human and animal physiology cellular and molecular biology organic and analytical chemistry obesity neurology, mental health vaccines soil and water quality, treatment veterinary pathogens nutrition

12. S T P I 12 Analyzing Peaks Reveals “Meaning” Data Source: Titles and abstracts of PubMed publications affiliated with institutions in South Africa, 2003-2005 (n=3952) heart disease virology HIV epidemiology, treatment, prevention, maternal and child health TB antibiotic resistance HPV/cervical cancer non-ID ob/gyn, women’s health human and animal physiology cellular and molecular biology organic and analytical chemistry obesity neurology, mental health vaccines soil and water quality, treatment veterinary pathogens nutrition

13. S T P I 13 Repeating Process for Uganda Illustrates Differences Data Source: Titles and abstracts of PubMed publications affiliated with institutions in Uganda, 2003-2005 (N=292) malaria HIV epidemiology healthcare service delivery malaria, epidemiology ID-related healthcare services delivery HIV+comorbid epidemiology malaria, treatment HIV, maternal and child health HIV, treatment and prevention zoonoses and veterinary epidemiology

14. S T P I 14 Several South African Institutions Publish Research Data Source: PubMed citations with South Africa affiliation, 2003-05 >500 publications University of Cape Town Stellenbosch University University of the Witwatersrand 100 to 500 publications University of Pretoria Nelson R. Mandela School of Medicine University of KwaZulu-Natal Medical Research Council University of the Free State 50 to 100 publications University of the Western Cape Rhodes University North-West University Potchefstroom University for Christian Higher Education 20 to 50 publications Onderstepoort Veterinary Institute University of Durban Rand Afrikaans University National Institute for Communicable Diseases Medical University of Southern Africa University of the North University of Transkei-- Groote Schuur Hospital Harvard:19000+ Boston University: 5076 MIT: 1564 Carnegie Mellon: 591

15. S T P I 15 Coding Publications by Institution Shows Diversification Data Source: PubMed publications affiliated with institutions in South Africa, 2003-2005 University of Cape Town Stellenbosch University University of Witwatersrand

16. S T P I 16 While 18 NIH Institutes and Centers Are Involved With Research…. NIH InstituteUSSATotalNIH InstituteUSSATotal NIAID32840NIDA314 FIC27936NIDCR3 3 NIMH21122NIDDK3 3 NICHD15419NCRR2 2 NCI7310NIDCD112 NIAAA7 7NINDS2 2 NHLBI5 5NCCAM1 1 NLM5 5NHGRI1 1 NIA4 4NIEHS1 1 Data Source: CRISP abstracts mentioning “South Africa”, 1999-2005 NIH InstituteUS SA Total Grand Total14027167

17. S T P I 17 NIH Funding is Primarily HIV/AIDS- Related Data Source: Abstracts and titles of PubMed publications affiliated with South Africa, 2003-05; CRISP titles and absracts associated with South Africa, 1999-2005 CRISP abstracts

18. 18 S T P I

19. 19 About INSPIRE™ Visual Document Analysis IN-SPIRE™ creates mathematical representations of documents, which are organized into clusters and visualized into "maps" that can be interrogated for analysis. More specifically, IN-SPIRE™ performs the following steps: –The text engine scans through the document collection and automatically determines the distinguishing words or "topics" within the collection, based upon statistical measurements of word distribution, frequency, and co-occurrence with other words. –The text engine uses these distinguishing words to create a mathematical signature for each document in the collection. Then it does a rough similarity comparison of all the signatures to create cluster groupings. –IN-SPIRE™ compares the clusters against each other for similarity, and arranges them in high-dimensional space (about 200 axes) so that similar clusters are located close together. –That high-dimensional arrangement of clusters is then flattened down to a comprehensible 2- dimensions—trying to preserve a picture where similar clusters are located close to each other, and dissimilar clusters are located far apart. –Finally, the documents are added to the picture by arranging each within the invisible “bubble” of their respective cluster. All of this information is then mapped onto the Galaxy and ThemeView visualizations that convey the document and topical relationships of your information. Developed by Pacific Northwest National Laboratory (http://in-spire.pnl.gov/)