1. Pat Langley School of Computing and Informatics Arizona State University Tempe, Arizona http://cll.stanford.edu/~langleylangley@asu.edu Computational Discovery of Explanatory Process Models Thanks to N. Asgharbeygi, K. Arrigo, D. Billman, S. Borrett, W. Bridewell, S. Dzeroski, O. Shiran, and L. Todorovski for their contributions to this research, which is funded by a grant from the National Science Foundation.

2. The Challenge of Systems Science focus on synthesis rather than analysis in their operation; focus on synthesis rather than analysis in their operation; develop system-level models with many variables / relations; develop system-level models with many variables / relations; rely on computational methods to aid in their construction. rely on computational methods to aid in their construction. Disciplines like Earth science and computational biology differ from traditional fields in that they: However, the key challenge involves search through the model space, not running rapid simulations or handling large data sets.

3. Example: Explain Data from the Ross Sea

4. A Model of the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Differential equation models of this sort are regularly used to explain observations and predict future behavior.

5. The Task of Model Construction Environmental scientists are confronted with a challenging task: Given: A set of variables of interest to the scientist; Given: A set of variables of interest to the scientist; Given: Observations of how these variables change over time; Given: Observations of how these variables change over time; Find: A model that explains these variations in plausible terms and that generalizes well to future observations. Find: A model that explains these variations in plausible terms and that generalizes well to future observations. Automating such model construction is a natural task for artificial intelligence and machine learning. We can develop algorithms that search the space of differential equation models, but this space is huge, so we need constraints.

6. Another Account of the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus As phytoplankton uptakes nitrogen, its concentration increases and nitrogen decreases. This continues until the nitrogen supply is exhausted, which leads to a phytoplankton die off. This produces detritus, which gradually remineralizes to replenish the nitrogen. Zooplankton grazes on phytoplankton, which slows the latters increase and also produces detritus.

7. Processes in the Ross Sea Ecosystem d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Here we highlight the terms related to phytoplantkon loss, which decreases phyto concentration and increases detritus. Knowledge about candidate processes requires that some terms occur either together or not at all.

8. d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus Processes in the Ross Sea Ecosystem We can use knowledge about processes to reorganize models and constrain search through the model space. Here we highlight terms related to zooplankton grazing, which decreases phyto but increases zoo and detritus.

9. A Process Model for the Ross Sea model Ross_Sea_Ecosystem variables: phyto, zoo, nitro, detritus observables: phyto, nitro process phyto_loss equations:d[phyto,t,1] = 0.307 phyto equations:d[phyto,t,1] = 0.307 phyto d[detritus,t,1] = 0.307 phyto process zoo_loss equations:d[zoo,t,1] = 0.251 zoo equations:d[zoo,t,1] = 0.251 zoo d[detritus,t,1] = 0.251 zoo process zoo_phyto_grazing equations:d[zoo,t,1] = 0.615 0.495 zoo equations:d[zoo,t,1] = 0.615 0.495 zoo d[detritus,t,1] = 0.385 0.495 zoo d[phyto,t,1] = 0.495 zoo process nitro_uptake equations:d[phyto,t,1] = 0.411 phyto equations:d[phyto,t,1] = 0.411 phyto d[nitro,t,1] = 0.098 0.411 phyto process nitro_remineralization; equations:d[nitro,t,1] = 0.005 detritus equations:d[nitro,t,1] = 0.005 detritus d[detritus,t,1 ] = 0.005 detritus This model is equivalent to a standard differential equation model, but it makes explicit assumptions about which processes are involved. For completeness, we must also make assumptions about how to combine influences from multiple processes.

10. The Task of Inductive Process Modeling We can use these ideas to reformulate the modeling problem: Given: A set of variables of interest to the scientist; Given: A set of variables of interest to the scientist; Given: Observations of how these variables change over time; Given: Observations of how these variables change over time; Given: Background knowledge about plausible processes; Given: Background knowledge about plausible processes; Find: A process model that explains these variations and that generalizes well to future observations. Find: A process model that explains these variations and that generalizes well to future observations. We can use background knowledge about candidate processes to make search much more tractable. Moreover, the resulting model will be consistent with this domain knowledge, making it more comprehensible.

11. Generic Processes as Background Knowledge the variables involved in a process and their types; the variables involved in a process and their types; the parameters appearing in a process and their ranges; the parameters appearing in a process and their ranges; the forms of conditions on the process; and the forms of conditions on the process; and the forms of associated equations and their parameters. the forms of associated equations and their parameters. We cast background knowledge as generic processes that specify: Generic processes are building blocks from which one can compose a specific process model.

12. Generic Processes for Aquatic Ecosystems generic process exponential_lossgeneric process remineralization variables: S{species}, D{detritus} variables: N{nutrient}, D{detritus} variables: S{species}, D{detritus} variables: N{nutrient}, D{detritus} parameters: [0, 1] parameters: [0, 1] parameters: [0, 1] parameters: [0, 1] equations:d[S,t,1] = 1 S equations:d[N, t,1] = D equations:d[S,t,1] = 1 S equations:d[N, t,1] = D d[D,t,1] = Sd[D, t,1] = 1 D generic process grazinggeneric process constant_inflow variables: S1{species}, S2{species}, D{detritus} variables: N{nutrient} variables: S1{species}, S2{species}, D{detritus} variables: N{nutrient} parameters: [0, 1], [0, 1] parameters: [0, 1] parameters: [0, 1], [0, 1] parameters: [0, 1] equations:d[S1,t,1] = S1 equations:d[N,t,1] = equations:d[S1,t,1] = S1 equations:d[N,t,1] = d[D,t,1] = (1 ) S1 d[S2,t,1] = 1 S1 generic process nutrient_uptake variables: S{species}, N{nutrient} variables: S{species}, N{nutrient} parameters: [0, ], [0, 1], [0, 1] parameters: [0, ], [0, 1], [0, 1] conditions:N > conditions:N > equations:d[S,t,1] = S equations:d[S,t,1] = S d[N,t,1] = 1 S Our current library contains about 20 generic processes, including ones with alternative functional forms for loss and grazing processes.

13. process exponential_growth variables: P {population} variables: P {population} equations: d[P,t] = [0, 1, ] P equations: d[P,t] = [0, 1, ] P process logistic_growth variables: P {population} variables: P {population} equations: d[P,t] = [0, 1, ] P (1 P / [0, 1, ]) equations: d[P,t] = [0, 1, ] P (1 P / [0, 1, ]) process constant_inflow variables: I {inorganic_nutrient} variables: I {inorganic_nutrient} equations: d[I,t] = [0, 1, ] equations: d[I,t] = [0, 1, ] process consumption variables: P1 {population}, P2 {population}, nutrient_P2 variables: P1 {population}, P2 {population}, nutrient_P2 equations: d[P1,t] = [0, 1, ] P1 nutrient_P2, equations: d[P1,t] = [0, 1, ] P1 nutrient_P2, d[P2,t] = [0, 1, ] P1 nutrient_P2 d[P2,t] = [0, 1, ] P1 nutrient_P2 process no_saturation variables: P {number}, nutrient_P {number} variables: P {number}, nutrient_P {number} equations: nutrient_P = P equations: nutrient_P = P process saturation variables: P {number}, nutrient_P {number} variables: P {number}, nutrient_P {number} equations: nutrient_P = P / (P + [0, 1, ]) equations: nutrient_P = P / (P + [0, 1, ]) Constructing Process Models model AquaticEcosystem variables: nitro, phyto, zoo, nutrient_nitro, nutrient_phyto observables: nitro, phyto, zoo process phyto_exponential_growth equations: d[phyto,t] = 0.1 phyto equations: d[phyto,t] = 0.1 phyto process zoo_logistic_growth equations: d[zoo,t] = 0.1 zoo / (1 zoo / 1.5) equations: d[zoo,t] = 0.1 zoo / (1 zoo / 1.5) process phyto_nitro_consumption equations: d[nitro,t] = 1 phyto nutrient_nitro, equations: d[nitro,t] = 1 phyto nutrient_nitro, d[phyto,t] = 1 phyto nutrient_nitro d[phyto,t] = 1 phyto nutrient_nitro process phyto_nitro_no_saturation equations: nutrient_nitro = nitro equations: nutrient_nitro = nitro process zoo_phyto_consumption equations: d[phyto,t] = 1 zoo nutrient_phyto, equations: d[phyto,t] = 1 zoo nutrient_phyto, d[zoo,t] = 1 zoo nutrient_phyto d[zoo,t] = 1 zoo nutrient_phyto process zoo_phyto_saturation equations: nutrient_phyto = phyto / (phyto + 0.5) equations: nutrient_phyto = phyto / (phyto + 0.5) HeuristicSearch observations generic processes process model phyto, nitro, zoo, nutrient_nitro, nutrient_phyto variables

14. A Method for Process Model Construction 1. Find all ways to instantiate known generic processes with specific variables, subject to type constraints; 2. Combine instantiated processes into candidate generic models subject to additional constraints (e.g., number of processes); 3. For each generic model, carry out search through parameter space to find good coefficients; 4. Return the parameterized model with the best overall score. Our initial system, IPM, constructs process models from generic components in four stages: Our typical evaluation metric is squared error, but we have also explored other measures of explanatory adequacy.

15. Results on Observations from Ross Sea We provided IPM with 188 samples of phytoplnkton, nitrate, and ice measures taken from the Ross Sea. From 2035 distinct model structures, it found accurate models that limited phyto growth by the nitrate and the light available. Some high-ranking models incorporated zooplankton, whereas others did not.

16. Results with Inductive Process Modeling population dynamics battery behavior hydrology biochemical kinetics

17. Extensions to Inductive Process Modeling heuristic beam search through the space of process models; heuristic beam search through the space of process models; hierarchical generic processes that further constrain search; hierarchical generic processes that further constrain search; an ensemble-like method that mitigates overfitting effects; an ensemble-like method that mitigates overfitting effects; an EM-like method that deals with missing observations. an EM-like method that deals with missing observations. In recent work, we have extended our system to incorporate: This approach has great potential to speed the construction of scientifc models – provided that domain users adopt it.

18. specify a quantitative process model of the target system; specify a quantitative process model of the target system; display and edit the models structure and details graphically; display and edit the models structure and details graphically; simulate the models behavior over time and situations; simulate the models behavior over time and situations; compare the models predicted behavior to observations; compare the models predicted behavior to observations; invoke a revision module in response to detected anomalies. invoke a revision module in response to detected anomalies. Because few scientists want to be replaced, we are developing an interactive environment, P ROMETHEUS, that lets users: The environment offers computational assistance in forming and evaluating models but lets the user retain control. Interfacing with Scientists

19. Viewing a Process Model Graphically

20. Viewing a Process Model as Equations

21. Adding a Process Manually

22. Requesting Automatic Model Revision

23. Results of Automatic Model Revision

24. Directions for Future Research provide better ways to visualize models, data, and their relation provide better ways to visualize models, data, and their relation offer users more natural ways to define the space of models offer users more natural ways to define the space of models specifying constraints on relations among entities and processes specifying constraints on relations among entities and processes characterizing subsystems that decompose complex models characterizing subsystems that decompose complex models incorporate intuitive metrics like match to trajectory shape incorporate intuitive metrics like match to trajectory shape more generally improve the usability of P ROMETHEUS more generally improve the usability of P ROMETHEUS Despite our progress to date, we need further work in order to: Taken together, these will make inductive process modeling a more robust approach to scientific model construction.

25. computational scientific discovery (e.g., Langley et al., 1983); computational scientific discovery (e.g., Langley et al., 1983); theory revision in machine learning (e.g., Towell, 1991); theory revision in machine learning (e.g., Towell, 1991); qualitative physics and simulation (e.g., Forbus, 1984); qualitative physics and simulation (e.g., Forbus, 1984); languages for scientific simulation (e.g., STELLA, MATLAB ); languages for scientific simulation (e.g., STELLA, MATLAB ); interactive tools for data analysis (e.g., Schneiderman, 2001). interactive tools for data analysis (e.g., Schneiderman, 2001). Intellectual Influences Our approach to aiding scientific model construction incorporates ideas from many traditions: Our work combines, in novel ways, insights from machine learning, AI, programming languages, and human-computer interaction.

26. Contributions of the Research incorporates a formalism that is familiar to many scientists; incorporates a formalism that is familiar to many scientists; takes into account background knowledge about the domain; takes into account background knowledge about the domain; produces meaningful results from small amounts of data; produces meaningful results from small amounts of data; generates models that explain rather than describe observations; generates models that explain rather than describe observations; provides an interactive environment for model construction. provides an interactive environment for model construction. In summary, our work on computational model construction has produced an approach that: We need much more research in computational systems science that addresses these challenges.

27. End of Presentation