1. NRC Assessment of Doctoral Programs Charlotte Kuh (ckuh@nas.edu)

2. Study Goals Help universities improve their doctoral programs through benchmarking. Expand the talent pool through accessible and relevant information about doctoral programs. Benefit the nation’s research capacity by improving the quality of doctoral students.

3. Background NRC conducted assessments in 1982, 1993 –The “gold standard” of ranking studies In 2000, formed a committee, chaired by Jeremiah Ostriker, to study the methodology of assessment –What can be done with modern technology and improved university data systems? –How can multiple dimensions of doctoral programs be presented more accurately?

4. Findings (November 2003) An assessment was worth doing More emphasis and broader coverage needed for the quantitative measures: a benchmarking study Present qualitative data more accurately: “rankings should be presented as ranges of ratings” Study should be made more useful to students Analytic uses of data should be stressed On-going updates of quantitative variables should continue after the study was completed.

5. Committee Jeremiah Ostriker, Princeton, chair (astrophysics) Virginia Hinshaw, UC-Davis, vice-chair (bioscience) Elton Aberle, Wisconsin- Madison (agriculture) Norman Bradburn, Chicago (statistics) John Brauman, Stanford (chemistry) Jonathan Cole, Columbia (social sciences) Eric Kaler, Delaware (engineering) Earl Lewis, Emory (history) Joan Lorden, UNC-Charlotte (bioscience) Carol Lynch, Colorado (bioscience) Robert Nerem, Georgia Tech (bioengineering) Suzanne Ortega, Washington (sociology) Robert Spinrad, Xerox PARC (computer science) Catharine Stimpson, NYU, (humanities) Richard Wheeler, Illinois- Urbana (English)

6. Panel on Data Collection Norman Bradburn, Chicago, chair Richard Attiyeh, UC-San Diego Scott Bass, UMd- Baltimore County Julie Carpenter-Hubin, Ohio State Janet L. Greger, Connecticut Dianne Horgan, Arizona Marsha Kelman, Texas Karen Klomparens, Michigan State Bernard Lentz, Pennsylvania Harvey Waterman, Rutgers Ami Zusman, UC System

7. Agricultural Fields are Included for the First Time Fields and Sub-fields (1) Agricultural Economics Animal Sciences –Aquaculture and Fisheries –Domestic Animal Sciences –Wildlife Science Entomology Food Science and Engineering –Food Engineering and Processing (sub-fields are not data collection units) –Food Microbiology –Food Chemistry –Food Biotechnology

8. Agricultural fields and sub-fields (2) Nutrition –Animal and comparative nutrition –Human and Clinical Nutrition –International and Community Nutrition –Molecular, Genetic, and Biochemical Nutrition –Nutritional Epidemiology Plant Sciences –Agronomy and Crop Sciences –Forestry and Forest Sciences –Horticulture –Plant Pathology –Plant Breeding and Genetics Emerging Fields: Biotechnology Systems Biology

9. Next steps Process has been widely consultative. Work began in fall, 2005. July 2006-May 2007: Fielding questionnaires, follow-up, quality review and validation. Competition for research papers. December 2007-Data base and NRC analytic essay released. December 2007-March 2008: Data analyses performed by commissioned researchers April 2008-August 2008: Report review and publication September 2008: Report and website release. Release conference

10. A New Approach to Assessment of Doctoral Programs A unique resource for information about doctoral programs that will be easily accessible Comparative data about: –Doctoral education outcomes Time-to-degree, completion rates –Doctoral education practices Funding, review of progress, student workload, student services –Student characteristics –Linkage to research Citations and publications Research funding Research resources

11. No pure reputational ratings Why not? Rater knowledge – Fields have become both more interdisciplinary and more specialized Why not? The US News effect—rankings without understanding what was behind them. What to substitute? Weighted quantitative measures. Possibly along different dimensions.

12. How will it work? Collect data from institutions, doctoral programs, faculty, and students –Uniform definitions will yield comparable data in a number of dimensions Examples of data –Students: demographic characteristics, completion rates, time to degree –Faculty: interdisciplinary involvement, postdoc experience, citations and publications –Programs: Funding policies, enrollments, faculty size and characteristics, research funding of faculty, whether they track outcomes

13. Program Measures and a Student Questionnaire Questions to programs –Faculty names and characteristics –Numbers of students –Student characteristics and financing –Attrition and time to degree –Whether they collect and disseminate outcomes data

14. Examples of Indicators Publications per faculty member Citations per faculty member Grant support and distribution Library resources (separating out electronic media) Interdisciplinary Centers Faculty/student ratios

15. Some Problems Encountered What is a faculty member? –3 kinds: Core, Associated, New –Primarily faculty involved in dissertation research –Faculty can be involved with more than one doctoral program Multidisciplinarity can result in problems due to need to allocate faculty among programs

16. Rating Exercise: Implicit A sample of faculty will be asked to rate a sample of programs. Provided names of program faculty and some program data Ratings will be regressed on other program data Coefficients will be used with data from each program to obtain a range of ratings

17. Rating Exercise: Explicit Faculty will be asked importance to program quality of program, educational, and faculty characteristics. Weights on variables will be calculated from their answers. Weights can be applied to program data to produce range of ratings Rankings can be along different dimensions –Examples: research productivity, education effectiveness, interdisciplinarity, resources Users may access and interpret the data in ways that depend on their needs. Database will be updateable

18. Project Product A database containing data for each program arrayed by field and university. Software to permit comparison among user selected programs In 2008—papers reporting on analyses conducted with the data

19. Uses by Universities High level administrators –Understanding variation across programs –Ability to analyze multiple dimensions of doctoral program quality –Enabling comparison with programs in peer institutions Program administrators, Department chairs –An opportunity to identify areas of specialization –Encourages competition to improve educational practice

20. Uses by prospective students Students can identify what’s important to them and create their own rankings Analytic essay will assist students on using the data Updating will mean the data will be current Better matching of student preferences and program characteristics may lower attrition rates.

21. Project Website http://www7.nationalacademies.org/resdoc/index.html