1. Asking the Oracle: Introducing Forecasting Principles into Agent-Based Modelling GRASIA, Universidad Complutense de Madrid INSISOC, Universidad de Burgos GUESS, Universidade de Lisboa Samer Hassan Javier Arroyo Jos é M. Gal á n Luis Antunes Juan Pav ó n

2. Samer Hassan WCSS 2010 2 Contents A recurrent issue The Field of Forecasting Forecasting Principles into ABM Conclusions

3. Photo

4. Samer Hassan WCSS 2010 4 A recurrent issue… in ABM How many times have we heard… Are accurate predictions possible within ABM? Should predictions be the main aim of ABM? Is ABM mature enough to make proper predictions? Stakeholders want predictions: is ABM an answer?

5. Samer Hassan WCSS 2010 5 A recurrent issue… in SIMSOC “Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact forecasts?”

6. Samer Hassan WCSS 2010 6 A recurrent issue… in SIMSOC “Does anyone know of a correct, model-based forecast of the impact of any social policy?” Scott Moss, SIMSOC list, April 2009, “any correct policy impact forecasts?” “The response, once misunderstandings were sorted out, was several accounts of reasons why policy impacts could not be forecast. The criteria I suggested for deeming a forecast to be correct was the correct forecast of the timing and direction of change of specified indicators.” Scott Moss, SIMSOC list, June 2009, “what is the point?”

7. Samer Hassan WCSS 2010 7 A recurrent issue… in JASSS Joshua Epstein (2008): Prediction is one possible aim for ABM… among 16 others Explanation Guiding data collection Raise new questions Challenge theories … “Explanation does not imply Prediction” Tectonics explain earthquakes but cannot predict them

8. Samer Hassan WCSS 2010 8 A recurrent issue… in JASSS Joshua Epstein (2008): Prediction is one possible aim for ABM… among 15 others Explanation Guiding data collection Raise new questions Challenge theories … “Explanation does not imply Prediction” Tectonics explain earthquakes but cannot predict them Thompson & Derr (2009): “Good explanations predict” An explanatory model is valid only if it predicts real behaviour

9. Samer Hassan WCSS 2010 9 A recurrent issue… in JASSS Klaus Troitzsch (2009) Epstein & Thompson discuss different “Prediction levels”: 1)Prediction of the kind of behaviour of a system, under arbitrary parameter combinations and initial conditions oEarthquakes occur because X and Y 2)Prediction of the kind of behaviour of a system in the near future oRegion R is likely to suffer earthquakes in the following years because X and Y 3)Prediction of the state a system will reach in the near future oRegion R will suffer an earthquake of power P in expected day D with confidence C “Explanation does not imply 3rd level Prediction” “Good explanations usually imply 1st or 2nd level”

10. Samer Hassan WCSS 2010 10 A recurrent issue Agent-Based Modelling has multiple aims… …but still modellers might seek prediction… How could we help them?

11. Samer Hassan WCSS 2010 11 Contents A recurrent issue The Field of Forecasting Forecasting Principles into ABM Conclusions

12. Samer Hassan WCSS 2010 12 The field of Forecasting Forecasting A field focused on the study of prediction Specially aiming 3rd level 30 years experience Consolidated (journals, conferences) Formalised “Forecasting experiment”

13. Samer Hassan WCSS 2010 13 The field of Forecasting Using ABM as a Forecasting tool Setting up a forecasting experiment: Split data in two sets “Objective” error measures Compare the model Fair Comparison

14. Samer Hassan WCSS 2010 14 The field of Forecasting Using ABM as a Forecasting tool Setting up a forecasting experiment: Split data in two sets Training set (in-sample): calibration Test set (out-of-sample): validation “Objective” error measures Compare the model Fair Comparison

15. Samer Hassan WCSS 2010 15 The field of Forecasting Using ABM as a Forecasting tool Setting up a forecasting experiment: Split data in two sets “Objective” error measures Error(t)= forecasted(t) - actual_value(t) Aggregated Error of time series: Root Mean Square Error Mean Absolute Error… Compare the model Fair Comparison

16. Samer Hassan WCSS 2010 16 The field of Forecasting Using ABM as a Forecasting tool Setting up a forecasting experiment: Split data in two sets “Objective” error measures Compare the model Benchmarks: other models, not necessarily ABM Na ï ve method (at least): V ’ (t+1)= V(t) Fair Comparison

17. Samer Hassan WCSS 2010 17 The field of Forecasting Using ABM as a Forecasting tool Setting up a forecasting experiment: Split data in two sets “Objective” error measures Compare the model Fair Comparison Representative, large sample of forecasts Ex-ante: forecast of (t+1) uses info available until (t) Out-of-sample: not include training data in comparison

18. Samer Hassan WCSS 2010 18 Contents A recurrent issue The Field of Forecasting Forecasting Principles into ABM Conclusions

19. Samer Hassan WCSS 2010 19 Forecasting Principles into ABM Principles of Forecasting Armstrong (2001) with 40 authors Summarising the best practices Selection of subset for ABM Six topics: Modelling Process Use of data Space of solutions Stake-holders Validation Replication

20. Samer Hassan WCSS 2010 20 Forecasting Principles into ABM Modelling Process Decompose the problem into parts Bottom-up approach + combination of results Structure problems that involve causal chains Results of a (sub)model as input for next one More accurate than global simulation Consider the use of adaptive forecasting models ABM as adaptive systems

21. Samer Hassan WCSS 2010 21 Forecasting Principles into ABM Data-driven modelling Use theory to guide the search for information on explanatory variables Reduce complexity pruning design space in advance Use diverse data sources Increase of data reliability Keep forecasting method simple KISS Select simple methods unless empirical evidence calls for a more complex approach KISS + gradual increase of complexity on demand

22. Samer Hassan WCSS 2010 22 Forecasting Principles into ABM Space of solutions Identify possible outcomes prior to making forecasts Avoid biases Design test situations to match the forecasting problem Put forward scenarios to rehearse policies Adjust for events expected in the future Expectability should guide design space exploration and what-if questioning

23. Samer Hassan WCSS 2010 23 Forecasting Principles into ABM Stake-holders and Policy-makers Obtain decision makers' agreement on methods Ideally participatory simulation Ask unbiased experts to rate potential methods Emphasising their role Test the client's understanding of the methods Including limitations of the model Establish a formal review process to ensure that forecasts are used properly Policy deployment should be controlled

24. Samer Hassan WCSS 2010 24 Forecasting Principles into ABM Validation List all the important selection criteria before evaluating methods Temptation of redefining criteria to fit the outcomes Use “objective” tests of assumptions Quantitative approach to test assumptions when possible Use extensions of evaluations to better generalise about what methods are best for what situations Generalisation leads to applicability; based on what-if scenarios Use error measures that adjust for scale in the data Error measuring is as important as accuracy of data Establish a formal review process for forecasting methods Ensure verification, replication, trust

25. Samer Hassan WCSS 2010 25 Forecasting Principles into ABM Replication Compare track records of various forecasting methods The role of replication for ABM verification Assess acceptability and understandability of methods to users Sharing of models & code Describe potential biases of forecasters From both modellers and stakeholders How sensitive is the model to those biases?

26. Samer Hassan WCSS 2010 26 Contents A recurrent issue The Field of Forecasting Forecasting Principles into ABM Conclusions

27. Samer Hassan WCSS 2010 27 Conclusions The choice of Agent-Based Modelling implies Interest in the “what” is going to happen (Prediction) Interest in “how” the phenomenon occurs (Understanding) Prediction (3rd level) is a hard job Financial crisis Climate change Forecasting Principles Best practices, not a solution Helpful in seeking the “what”

28. Samer Hassan WCSS 2010 28 Thanks for your attention! Samer Hassan samer@fdi.ucm.es Universidad Complutense de Madrid

29. Samer Hassan WCSS 2010 29 Contents License This presentation is licensed under a Creative Commons Attribution 3.0 http://creativecommons.org/licenses/by/3.0/ You are free to copy, modify and distribute it as long as the original work and author are cited