1. Optimal revision of uncertain estimates in project portfolio selection Eeva Vilkkumaa, Juuso Liesiö, Ahti Salo Department of Mathematics and Systems Analysis, Aalto University School of Science and Technology

2. Contents Project portfolio selection Optimizer’s curse Revised estimates Discussion

3. Project portfolio selection Select a subset of projects within a budget, e.g., k out of n projects with the aim of maximizing the sum of the projects’ values μ i, i=1,...,n The values μ i are generally unknown, whereby decisions about which projects to select are made based on estimates V i about μ i. EstimatesPortfolio selectionValues t

4. Optimizer’s curse in portfolio selection Assume that the estimates are unbiased Portfolio maximization selects, on average, overestimated projects → the value of the portfolio is less than expected based on the estimation information (optimizer’s curse; cf. Smith and Winkler, 2006): where is the index set of the selected projects.

5. Optimizer’s curse in portfolio selection Choosing 10 projects out of 100 Values i.i.d with Unbiased estimates The larger the estimation error variance, the harder it is to identify the best projects, and the larger the difference between the estimated and realized portfolio value µ i ~ N(0,1 2 ) V i = µ i + ε i, ε i ~ N(0,σ 2 ) Standard deviation of estimation error Portfolio value

6. Optimal revision of the estimates Estimates do not account for the uncertainties Use Bayesian revised estimates instead as a basis for project selection For instance, with µ i ~ N(m i,σ i 2 ), V i ~ N(µ i,τ i 2 ): The estimate V and the prior information m are weighted according to their uncertainty. where

7. Optimal revision of the estimates With revised estimates the optimizers’ curse is eliminated, that is where is the index set of the projects selected using revised estimates Previous example –Choosing 10 projects out of 100 –True values i.i.d. with –Unbiased estimates µ i ~ N(0,1 2 ) V i = µ i + ε i, ε i ~ N(0,σ 2 ) Portfolio value Standard deviation of estimation error

8. Revised estimates and portfolio composition In the previous example, the projects’ values were identically distributed, and the estimation errors had equal variances Then, prioritization among the projects remains unchanged when the estimates are revised, because In general, using revised estimates may result in a different project prioritization than estimates

9. Revised estimates and portfolio composition Project value EstimateRevised estimate EstimateRevised estimate Same error variancesDifferent error variances Project value Choosing 3 projects out of 8 True values i.i.d. With µ i ~ N(0,1 2 ) On the left, estimates with equal error variance for all projects On the right, four projects (dashed) more difficult to estimate V i = µ i + ε i, ε i ~ N(0,0.5 2 ) V i = µ i + ε i, ε i ~ N(0,1 2 )

10. Revised estimates and portfolio composition On the left, equal error variances → estimates are shifted towards the common prior mean (zero) in the same proportion On the right: the revised estimates of the ”dashed” projects are more drawn towards zero, because the estimation information is less reliable Selection of 3 projects leads to different portfolios depending on whether the estimates are revised or not Project value EstimateRevised estimate EstimateRevised estimate Same error variancesDifferent error variances Project value

11. Revised estimates and portfolio value The use of revised estimates yields at least as high overall portfolio value as the use of initial estimates, i.e. Example: –Selection of 10 out of 100 projects with values µ i ~ N(3,1 2 ) –Population contains two types of projects: –Revised estimates yield higher portfolio value for any non-trivial division between projects with small and large estimation error variances 1) ε i ~ N(0,0.1 2 ) - small error variance 2) ε i ~ N(0,1 2 ) - large error variance Share of projects with large error variance [%] Portfolio value Optimal Estimates Revised estimates

12. Revised estimates and correct choices The share of correctly selected projects increases with revised estimates in the normally distributed case, i.e., where K is the index set of the projects in the optimal portfolio In the previous example, the difference between the two portfolios is statistically significant (α=0.05), when the share of projects with large error variance is between 25-55% Share of correct choices [%] Share of projects with large error variance [%]

13. Discussion Selection based on project prioritization resulting from estimates –The value of the portfolio will, on average, be lower than expected –If there are differences in the projects’ estimation error variances, too many projects with large error variance will be selected Suggestions for improving the selection process –Accounting for the uncertainties by using revised estimates –Sorting the projects in terms of estimation error variances by, e.g., budget division