1. NO EQUATIONS, NO VARIABLES NO PARAMETERS:
Cambridge (Newton Institute) December 2015
NO EQUATIONS, NO VARIABLES
NO PARAMETERS:
Data, and the computational modeling
of Complex/Multiscale Dynamical Systems
I.G. Kevrekidis, W. Gear, R. Coifman -and other good people
like Carmeline Dsilva and Ronen Talmon-
Department of Chemical Engineering, PACM & Mathematics
Princeton University, Princeton, NJ 08544
This semester: Hans Fischer Fellow, IAS-TUM Muenchen

3. Clustering and stirring in a plankton model An “equation-free” demo
Young, Roberts and Stuhne, Nature 2001

4. Dynamics of System with convection

5. Coarse Projective Integration
(coarse projective Forward Euler)

6. Projective Integration: From t=2,3,4,5 to 10

7. time Projective Integration - a sequence of outer integration steps
based on inner simulator + estimation (stochastic inference)
time
Accuracy and stability of these methods – NEC/TR 2001
(w/ C. W.Gear, SIAM J.Sci.Comp. 03, J.Comp.Phys. 03,
--and coarse projective integration (inner LB)
Comp.Chem.Eng. 2002
Time
Space
Value
Project forward in time
Projective methods in time:
perform detailed simulation for short periods
or use existing/legacy codes
- and then extrapolate forward over large steps

8. simulations in the full spatial and temporal domain is expensive
Patch Dynamics Computation
value
time
space
value
time
space
(I) Full Scale
time
space
value
running fine-scale
simulations in the full spatial and temporal
domain is expensive
(II) Coarse Projective Integration
time
value
space
time
space
value
value
time
space
value
time
space
run fine-scale
simulations in only a
fraction of the spatial
and temporal domain
(II) Gap-tooth
(II) Patch Dynamics

9. So, the main point:
You have a “detailed simulator” or an experiment
Somebody tells you what are good coarse variable(s)
Then you can use the IDEA
that a coarse equation exists
to accelerate the simulation/ extraction of information.
Equation-Free
BUT
How do we know what the right coarse variables are
Data mining techniques – Diffusion Maps

10. Where do good coarse variables come from?
Systematic hierarchy (and mathematics)
Fourier modes, moments…..
Experience/expertise/knowledge/brilliance
Human learning (“brain” data mining, observation, phase fields…)
Machine Learning
Data mining, manifold learning (here: diffusion maps)

12. and now for something completely different:
moving groups of individuals (T. Frewen)

14. Fish Schooling Models Couzin, Krause, Franks & Levin (2005)
Initial State
Position, Direction, Speed
INFORMED
UNINFORMED
Compute Desired Direction
Zone of Deflection Rij<
Zone of Attraction Rij<
Normalize 
Update Direction for Informed Individuals ONLY
Couzin, Krause,
Franks & Levin (2005)
Nature (433) 513
Update Positions

15. STUCK ~ typically around rxn coordinate value of about 0.5
INFORMED DIRN
STICK STATES
STUCK
~ typically around
rxn coordinate
value of about 0.5
INFORMED individual
close to front of group
away from centroid

16. close to group centroid
INFORMED DIRN
SLIP STATES
SLIP
~ wider range of
rxn coordinate values
for slip 00.35
INFORMED individual
close to group centroid

17. Dataset as Weighted Graph
edges 3
W23
weights 2
W12 1
vertices
parameter tR

18. Compute NN “neighborhood” matrix K
N datapoints
Compute NN “neighborhood” matrix K
Parameter
Local neighborhood size
Compute diagonal normalization matrix D
Compute Markov matrix M
Require: Eigenvalues λ and Eigenvectors Φ of M
Top
2nd
A few Eigenvalues\Eigenvectors provide
meaningful information on dataset geometry
Nth

19. Eigenfunctions of the Laplacian to Parameterize a Manifold
Consider a manifold
We want a “good“ parameterization of the manifold
Instead, we have data
sampled from the manifold and want to obtain a parameterization
These parameterizations are the
eigenfunctions of Δφ
We therefore approximate the Laplacian on the data
We obtain a similar parameterization for a curved manifold
Diffusion maps approximates the eigenfunctions for a manifold
using data sampled from the manifold

20. Diffusion Maps Dataset in x, y, z Dataset Diffusion Map N datapoints
eigencomputation
N datapoints
N datapoints
R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, and S. Zucker,
Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps.
PNAS 102 (2005).
B. Nadler, S. Lafon, R. Coifman, and I. G. Kevrekidis,
Diffusion maps, spectral clustering and reaction coordinates
of dynamical systems.
Appl. Comput. Harmon. Anal. 21 (2006).

21. 3 2 2 Diffusion Map (2, 3) ABSOLUTE Coordinates
Report absolute distance
of all uninformed individuals
to informed individual to DMAP routine
Report (signed) distance
of all uninformed individuals
to informed individual to DMAP routine
ABSOLUTE Coordinates
SIGNED Coordinates
Reaction
Coordinate 3
STICK
STICK
SLIP
SLIP 2 2

22. ABSOLUTE Coordinates
SIGNED Coordinates
MACHINE
MACHINE
MAN
MAN

23. Effective simplicity
Construct predictive models (deterministic, Markovian)
Get information from them: CALCULUS, Taylor series
Derivatives in time to jump in time
Derivatives in parameter space for sensitivity /optimization
Derivatives in phase space for contraction mappings
Derivatives in physical space for PDE discretizations
In complex systems --- no derivatives at the level we need them
sometimes no variables ---- no calculus
If we know what the right variables are, we can
PERFORM differential operations
on the right variables – A Calculus for Complex Systems

24. Nonlinear Intrinsic Variables
The key part of diffusion maps is constructing the appropriate distance metric for the problem
We would like to construct a distance metric that is invariant to the observation function (under some conditions), and instead respects the underlying dynamics of the system
A. Singer and R. R. Coifman, Applied and Computational Harmonic Analysis, 2008

25. Chemical Reaction Network Example
We simulate the dynamics of a chemical reaction network using the Gillespie Stochastic Simulation Algorithm
The rate constants are such that the reaction coordinates quickly approach an apparent two-dimensional manifold

26. Computing the Distance Metric from Empirical Data
We have data from a
trajectory or time series
For each data point, we look
at the local trajectory
in order to sample the
local noise
We estimate the
local covariance using the
local trajectory
We then consider the distance metric
where C is the estimated noise covariance.
This is the Mahalanobis distance and is invariant to the observation function
(provided the observation function is invertible).
We then use the Mahalanobis distance in a DMAPS computation.
We call the resulting DMAPS variables Nonlinear Intrinsic Variables (NIV).
C. J. Dsilva et al., The Journal of chemical physics, 2013.

28. Partial Observations and NIV embeddings
We simulate the dynamics of a chemical reaction network using the
Gillespie Stochastic Simulation Algorithm
The rate constants are such that the reaction coordinates quickly approach an apparent two-dimensional manifold
We observe the concentrations
of E, S, S : E, and F : E*
We observe the concentrations
of E, S, E : S, and D : S*
We obtain a consistent embedding for the two data sets
D. T. Gillespie, The Journal of Physical Chemistry, 1977

29. Reconstructing Missing Components
We would like to exploit the consistency of the embeddings
to “fill in” the missing components in each data set.
Train a function
f (x) : NIV space → chemical space
using data set 1 as training data
We use a multiscale Laplacian Pyramids algorithm for training. We could use splines, nearest neighbors, etc.
Compute NIV embedding of
data set 2
Use f(x) to estimate components
in data set 2
We can successfully predict
the concentrations of the
missing components
in data set 2
N. Rabin and R. R. Coifman, SDM, 2012.

30. Another data problem, and a small miracle
NIVs also help solve a major multiscale data problem
There are two kinds of reduction:
one in which the value of the fast variable
(its QSS) becomes slaved to the slow variable
and one in which the statistics of the fast variable
(its quasi-invariant measure) becomes slaved to the slow variable.

31. Two types of reduction (“ODE” and “SDE”)

32. Manifold Learning Applied to Reducible SDEs
Consider the SDE
If ε < 1, then x is the only slow variable
Issue: y is still O(1),
so we cannot recover
only x using DMAPS
a = 3, ε=1e-3

33. Rescaling to Recover the Slow Variables
Consider the transformation
z = y √ε
Then the SDEs become
Now z is of a much smaller scale than x,
and DMAPS will recover x
Note: after this transformation,
the two SDEs both have
noise with unit variance

34. NIVs to Recover the Slow Variables with Nonlinearities
We can also recover NIVs when the data is obscured by a nonlinear measurement function
Look at local clouds
in the data
We again recover the slow variable x

37. 1 2 3 We are studying the accuracy and stability of these methods
Reverse Projective Integration –
a sequence of outer integration steps backward;
based on forward steps + estimation 1 2 3
Reverse Integration: a little forward,
and then a lot backward !
We are studying the accuracy and stability of these methods

39. Free Energy Landscape

40. Right-handed -helical minimum

41. Just like a stable manifold computation
Instantaneous ring position
consists of set of replicas (nodes)
Periodic Ring BCs
(arc length)
Evolve ring with
normal velocity u:
define the ring mathematically
periodic BCs – use existing material
Backward Integration of:
additional term controls
nodal distribution along ring

42. Parametrization: as in string method E, Ren, Vanden-Eijnden
Protocols for Coarse MD using Reverse Ring Integration
MD Simulations
run FORWARD
in Time
Initialize ring nodes
Step Ring
BACKWARD
in “time” or effective potential
Parametrization: as in string method E, Ren, Vanden-Eijnden

43. Stretch points close to edges to avoid extension issues
USUAL DMAP + BOUNDARY DETECTION
Stretch points close to edges to avoid extension issues

44. An algorithm for boundary point detection
Legend:
Previously detected boundary point
Nearest neighbor of
Small neighborhood of
Center of mass of neighborhood
First edge point to be guessed (if DMAP is used, take min/max)
1) Compute:
2) New boundary points:
3) Algorithm parameters:

45. USE NEW COORDINATES FOR EXTENSION PURPOSES
Both restriction and lifting were made by local RBF (over 40 nearest neighbours)

46. Out of sample extension
Physical high-dimensional space
START
END
Reduced low-dimensional space (DMAP, PCA)
Out-of-sample (extended, on-manifold) reduced sample
extend

49. Sine-DMAP
Standard DMAP algorithm automatically imposes a zero-flux (Neumann) condition at the manifold boundary (without even explicitly knowing where the boundary is!);
There is a possibility of imposing instead Dirichlet-boundary conditions (provided we are able to detect in advance the manifold boundary);
Simple example: Points on a segment of length 1 (physical coordinate 0<x<1)
One-to-one but…trying to extend f at the boundary:
Cosine-based
DMAPs 1
Cosine-DMAP coordinate
Ok at the boundary!
Edge detection
Sine-based
DMAPs
Sine-DMAP coordinate

50. Initial configuration
1) Alanine dipeptide in implicit solvent (Still algorithm) by GROMACS;
2) dt=0.001 ps, Nsteps= ;
3) Temperature: 300 K (v-rescale);
4) PBC, with a simulation box of (2nm)^3;
5) Cut-off for non-bonded interactions: 0.8 nm;
6) Force field: Amber03.

51. Edges can be automatically detected using local PCA and by computing the Boundary Propensity Field (see Gear’s notes)

52. Restart from here
A new energy well is discovered starting from the considered extended edge after a very short MD run (Nsteps= 20000)!

53. Restart from here
No new wells are found starting from a different extended edge after a very short MD run (Nsteps= 20000)

56. Effective simplicity
Construct predictive models (deterministic, Markovian)
Get information from them: CALCULUS, Taylor series
Derivatives in time to jump in time
Derivatives in parameter space for sensitivity /optimization
Derivatives in phase space for contraction mappings
Derivatives in physical space for PDE discretizations
In complex systems --- no derivatives at the level we need them
sometimes no variables ---- no calculus
If we know what the right variables are, we can
PERFORM differential operations
on the right variables – A Calculus for Complex Systems

57. Coming full circle PRECONDITIONING, B A x = B b
No equations ? Isn’t that a little medieval ? Equations =“Understanding”
AGAIN matrix free iterative linear algebra
A x = b
PRECONDITIONING, B A x = B b
B approximate inverse of A
Use “the best equation you have”
to precondition equation-free computations.
With enough initialization authority: equation free laboratory experiments
Samaey et al., JCompPhys (LBM for streamer fronts) –
physics/ math.DS/ q-bio.QM/

58. Computer-Aided Analysis of Nonlinear Problems in Transport Phenomena Robert A. Brown, L. E. Scriven and William J. Silliman in HOLMES, P.J., New Approaches to Nonlinear Problems in Dynamics, 1980
ABSTRACT The nonlinear partial differential equations of mass, momentum, energy,
Species and charge transport…. can be solved in terms of functions of limited differentiability,
no more than the physics warrants, rather than the analytic functions of classical analysis…
….. basis sets consisting of low-order polynomials. …. systematically generating and
analyzing solutions by fast computers employing modern matrix techniques.
….. nonlinear algebraic equations by the Newton-Raphson method. … The Newton-Raphson
technique is greatly preferred because the Jacobian of the solution is a treasure trove, not only
for continuation, but also for analysing stability of solutions, for detecting bifurcations of
solution families, and for computing asymptotic estimates of the effects, on any solution, of
small changes in parameters, boundary conditions, and boundary shape……
In what we do, not only the analysis, but the equations themselves are obtained on the
computer, from short experiments with an alternative, microscopic description.