Towards Objective Ranking of Project Proposals Miroslav Kárný Department of Adaptive Systems Institute of Information Theory and Automation Academy of Sciences of the Czech Republic

… speaker’s home institute … speaker’s home institute Cybernetics  Communication & Control in Machines & Animals Cybernetics is speaker’s research domain and led to applications in: Cybernetics is speaker’s research domain and led to applications in: Adaptive control of paper machines, rolling mills, drum boilers,… Adaptive control of paper machines, rolling mills, drum boilers,… Nuclear medicine modeling & DM, dynamic image studies … Nuclear medicine modeling & DM, dynamic image studies … Support of operators of complex systems (FET) Support of operators of complex systems (FET) Traffic control in cities, optimization of financial strategies Traffic control in cities, optimization of financial strategies Multiple participants’ DM and E-democracy … Multiple participants’ DM and E-democracy … …? …! …? …! Bayesian DM: single-horse on decades-lasting trip with a good team Bayesian DM: single-horse on decades-lasting trip with a good team … nickname for Institute of Information Theory and Automation

FET organizes a review process … … to select the best proposals p among all submitted proposals An expert e assigns marks e m p  {0,…,M} to several proposals within a small group e p of proposals An expert e assigns marks e m p  {0,…,M} to several proposals within a small group e p of proposals A small group of experts p e, reviewing the proposal p, harmonizes the final mark m p via discussion A small group of experts p e, reviewing the proposal p, harmonizes the final mark m p via discussion Assembly of all experts completely ranks all proposals Assembly of all experts completely ranks all proposals EC supports top proposals up to a budget-implied border-line

Addressed problem Procedure is good & fair Each expert e has studied a tiny portion of all proposals Each expert e has studied a tiny portion of all proposals Experts’ marks e m p are subjectively scaled Experts’ marks e m p are subjectively scaled Discrete-valued marks cause many coincidences Discrete-valued marks cause many coincidences Time slot of the assembly is strongly limited Time slot of the assembly is strongly limited errorsmanipulationexpenses … up to the extremely disturbing step  An expert e assigns marks e m p  {0,…,M} to several proposals An expert e assigns marks e m p  {0,…,M} to several proposals within a small group e p of proposals within a small group e p of proposals A small group of experts p e, reviewing the proposal p, harmonizes the final mark m p via discussion A small group of experts p e, reviewing the proposal p, harmonizes the final mark m p via discussion Assembly of all experts completely ranks all proposals Assembly of all experts completely ranks all proposals

Aims to test belief that Bayesian DM is (almost) universal tool relying on the proper modeling only to test belief that Bayesian DM is (almost) universal tool relying on the proper modeling only to test a promising negotiation methodology needed in other contexts, too to test a promising negotiation methodology needed in other contexts, too to help FET to be fair and cost-efficient to help FET to be fair and cost-efficient to help proposing researchers to be treated fairly to help proposing researchers to be treated fairly to share fun (?) from the conclusions to share fun (?) from the conclusions … of the research … of the talk

Basic idea Experts serve as rank-measuring devices Ranking  estimation of rank r p from marks e m p, which are noise-corrupted observations of the objective rank Project proposal p has its objective rank r p !

Guide Experts as measuring devices Experts as measuring devices Prior knowledge Prior knowledge MAP estimate MAP estimate Experimental results Experimental results Discussion Discussion

Experts as measuring devices e m p … mark of proposal p by the expert e = r p … objective rank of proposal p r p … objective rank of proposal p + e  … personal error e  … personal error e  … personal error = e b … bias e b … bias + e  … personal fluctuations with variance e v e  … personal fluctuations with variance e v Simplicity & maximum entropy  e  assumed to be Gaussian experts try to be fair  mark e m p proportional to r p e  independent of p interpretation of marks top M  Nobel Prize top M  flawless

Prior knowledge e m p = r p + e b + e  = (r p – C) + ( e b + C) + e , for any C e m p = r p + e b + e  = (r p – C) + ( e b + C) + e , for any C Needed Available bias e b  [-M, M ], noise variance e v  [0, M 2 ] noise variance e v  [0, M 2 ] rank  [0, largest mark]  r p  [0, M ]  1 – 2  1 – 2 number of data number of data number of unknowns number of unknowns

MAP estimate Posterior log-likelihood function smoothly dependent on the estimated r, b, v smoothly dependent on the estimated r, b, v concave in the estimated r, b, v concave in the estimated r, b, v defined on a convex domain defined on a convex domain harmonised domain and data range harmonised domain and data range Evaluation Conditions for extreme are solved by successive approximations Conditions for extreme are solved by successive approximations  unique maximum  maximum in interior … fast, simple and reliable … can be used “on-line” … fast, simple and reliable … can be used “on-line”

Experiments - proposals’ viewpoint Processed marks m  {0, 0.5,…,30}; Assembly ranking available #Proposal #Proposal #Experts #Experts acceptance Threshold acceptance Threshold #proposals above T by A #proposals above T by A #proposals above T by us #proposals above T by us #proposals chosen by A and us #proposals chosen by A and us #common acceptance / A-one [%] #common acceptance / A-one [%] Extreme cases: typical numbers typical numbers prior does not spoil results with a few data prior does not spoil results with a few data

Histogram of rank estimates … box width about 2% of the mark range ! … box width about 2% of the mark range ! #(r >T  22) = 11 #(r >T  22) = 11 #(r>T  25) = 57

Experiments - experts’ viewpoint mean (bias) / Threshold [%] 6 4 mean (bias) / Threshold [%] 6 4 minimum (bias) / T minimum (bias) / T maximum (bias) / T maximum (bias) / T mean (std. dev.) / T mean (std. dev.) / T minimum (std. dev.) / T 10 7 minimum (std. dev.) / T 10 7 maximum (std. dev.) / T maximum (std. dev.) / T Box width containing significant number of proposals  3 % of T ! Box width containing significant number of proposals  3 % of T !

Individual results – small file

Individual top results – extensive file

Discussion it works it works it exhibits fast and reliable convergence it exhibits fast and reliable convergence it is reasonably robust to variations of prior statistics it is reasonably robust to variations of prior statistics Operational aspects Evaluation aspects it can substitute or at least support assembly ranking it can substitute or at least support assembly ranking it allows continuous-valued marking it allows continuous-valued marking it avoids the need to harmonize marks within p e it avoids the need to harmonize marks within p e it makes ranking less sensitive to experts’ biases & variations it makes ranking less sensitive to experts’ biases & variations it suppresses lottery-type results for gray-zone-ranked it suppresses lottery-type results for gray-zone-ranked proposals (those with the rank around threshold) proposals (those with the rank around threshold) it makes evaluation more objective it makes evaluation more objective

Discussion it checks reliability of experts, using their biases & variances: it checks reliability of experts, using their biases & variances: Methodological aspects [%] experts o.k. but unreliable or cheating rest still forms a significant portion Quality assurance aspects it allows tracking of “bad” experts it allows tracking of “bad” experts it opens a way to relate prior & posterior ranking, i.e., the achieved results of supported projects it opens a way to relate prior & posterior ranking, i.e., the achieved results of supported projects it can be tailored to other problems it can be tailored to other problems it can serve as a tool supporting negotiation it can serve as a tool supporting negotiation

Future alternative models of experts, e.g., non-normal, Markov-chain type alternative models of experts, e.g., non-normal, Markov-chain type comparison of prior and posterior ranking comparison of prior and posterior ranking application to other negotiation-type processes application to other negotiation-type processes application to individual marks & thresholds application to individual marks & thresholds quality assurance of the evaluation including experts’ competence ! quality assurance of the evaluation including experts’ competence !