Presentation is loading. Please wait.

Presentation is loading. Please wait.

Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH.

Similar presentations


Presentation on theme: "Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH."— Presentation transcript:

1 Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH

2 Outline 1.Hardcore philosophical underpinnings 2.Evaluating peer review 3.Alternatives to peer review 4.Justifying decisions about quality

3 Hardcore Underpinnings There is a dialectic between ontology and epistemology. John Burke Grammar of Motives

4 Ontology, Epistemology, Dialectic Ontology – what is taken to exist (what needs a name) Epistemology – what do you know about it Dialectic – thesis, antithesis, synthesis

5 Dialectic between Ontology and Epistemology What counts as data when you form your beliefs The relation between evidence and justice. Are your beliefs “sensible” and “reasonable” The relation between metrics and evaluation

6 Outline 1.Hardcore philosophical underpinnings 2.Evaluating peer review 3.Alternatives to peer review 4.Justifying decisions about quality

7 Evaluating Peer Review Selecting reviewers; providing incentives Informed objectivity; conflict of interest Review criteria; comparing across panels Ronald N. Kostoff US Office of Naval Research

8 Selecting Reviewers: Peer vs Non-Peer Balancing interests: Economic - government, academia, industry, society Intellectual - engineers, scientists, policy agencies Practical - curiosity, problem based, market driven

9 Informed objectivity “Good old boy” consensus Natural bias often subtle, needs diversity With paradigm shifts, experience becomes a liability

10 US National Science Foundation Review Criteria Intellectual merit –creative and original concepts Broader impacts –benefits to society

11 Outline 1.Hardcore philosophical underpinnings 2.Evaluating peer review 3.Alternatives to peer review 4.Justifying decisions about quality

12 The Age of Unreason “Changes are not what they used to be.” from book “The Age of Unreason” Charles Handy, 1989 a.Change by discontinuous leaps b.Learning from the past dangerous c.Evolution yes, but allow for revolution

13 NASA’s Earth Science Information Partners Courtesy Don Collins of the DAAC Alliance

14 Federalism Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly) Heterogeneous, diverse communities Data centers, academics, government, and industry Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

15 Outline 1.Hardcore philosophical underpinnings 2.Evaluating peer review 3.Alternatives to peer review 4.Justifying decisions about quality

16 Justifying Decisions about Quality Concept spaces and mapping Fact-value continuum? Fate of Knowledge (social construction)

17 Concept spaces and mapping

18 Fact-value continuum? Can we distinguish fact claims from value judgments? Are there really objective as opposed to subjective distinctions? Amartya Sen introduces value judgments in economics

19 Fate of knowledge Knowledge is social Cognitive processes are social (reasonable) Actions based on knowledge are justified through social processes.

20 Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

21 Promising developments Semantic web not words; rather “meaningful” data, concepts, and ideas. Science draws meaning from data; and has changed the way it justifies this. The semantic web offers to improve or supplant “peer” review (and education). The semantic web provides a “marketplace” for learning.

22 Profiles for discussion People Content Technology Values and priorities Economic Environment

23 Evolution to a Semantic Web HTML – Generic display of hypertext XML – Generic display of data RDF – Triples (subject verb object) OWL – Taxonomy, inference rules, and proofs

24 Changes in Science 1930 –Reason and Sense (logical positivism) 1960 – Beyond Reason (linguistic turn) 1990 – Social Construction (peer review) 2020 – Back to Reason and Sense

25 “Peers” and “Educators” Ontology – Locally coherent naming structures Realism – “Reifying” the structures Under standing – Logical foundations Knowledge – Justifying decisions (personal beliefs)

26 Business Plan and “Marketplace” Sales agents – Know-bots negotiate deals Value chains – Multiple entities in assembly line Public choice – Market forces and “fair” voting rules Learning Economy – From food chain to ecosystem

27 Back to the Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

28 Problems in Public Funding National Archives - GPRA - Curriculum - 2% GNP - What data is valuable? Is science making progress? What is “educated”? How much to spend?

29 The Age of Unreason “Changes are not what they used to be.” from book “The Age of Unreason” Charles Handy, 1989 a.Change by discontinuous leaps b.Learning from the past dangerous c.Evolution yes, but allow for revolution

30 Category I Earth Science Information Partners Courtesy Don Collins of the DAAC Alliance

31 MPEG-7: Metadata for Content Description Data Signal FeaturesModelSemantics Objects Events Actions People Labels Relationship Clusters Classes Collections Probabilities Confidences Color Texture Shape Motion Camera motion Regions Segments Mosaics Relationship (Spatio- temporal) Images Video Audio Multimedia Formats Layout SemanticModelsFeaturesStructureData Courtesy John Smith, IBM

32 Data, Community, Public Use FarmersUSDA $$$ on pest control Yes, treat your crops today Johns Hopkins $$ on fire ants studies NCDC GHRC USGS $ on real time data IBM Users Application solution provider Science data provider Backbone data center Technology provider Real-time riskmap Historical data Software Real-time data Real-time data Courtesy Yuan-Chi Chang, IBM

33 Other Cross Discipline ESIP’s Science Oceanography (2) Terrestrial Studies (4) Climate (3) Technology (3) Public Use Education (3) Regional Policy (4) Public Health Media Legal

34

35 Community I “A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.” E. O. Wilson Consilience: The Unity of Knowledge

36 Community II “It is the disorder of the scientific community— the laminated, finite, partially independent strata supporting one another; it is the disunification of science—the intercalation of different patterns of argument—that is responsible for its strength and coherence..” Peter Galison Image and Logic, 1997

37 Federalism Central coordination, local autonomy Tiered governance (US Federal vs States) Yield power to center (only reluctantly) Heterogeneous, diverse communities Data centers, academics, government, and industry Interdependence & minority interests Match and balance different values Take into account intensity of interest The whole is greater than the parts.

38 Federations Preserve Heterogeneity Economic - government, academia, industry Intellectual - academic disciplines, policy agencies Practical - problem based, market driven

39 Economic Model Public funding caps out at less than 3% of the GNP Dynamic “pricing” is needed to demonstrate value to end-users “Market forces” or other methods of public choice provide robust mechanism to rank options

40 Trading Zones Economic - government, academia, industry Intellectual - engineers, scientists, policy agencies Practical - curiosity, problem based, market driven

41 Peer Review Nurturing cooperation vs competition Economic - government, academia, industry Intellectual - engineers, scientists, policy agencies Practical - curiosity, problem based, market driven

42 Back to the Vision Concept-based science communication Personalized Learning Environment Economic Environment supports learning

43 Business Plan and “Marketplace” Sales agents – Know-bots negotiate deals Value chains – Multiple entities in assembly line Public choice – Market forces and “fair” voting rules Learning Economy – From food chain to ecosystem


Download ppt "Concrete Approaches to Quality Assessment: Moving Beyond Peer Review August 6, 2003 Howard Burrows Autonomous Systems Institute Lee, NH."

Similar presentations


Ads by Google