Presentation on theme: "Cost of surrogates In linear regression, the process of fitting involves solving a set of linear equations once. For moving least squares, we need to form."— Presentation transcript:
1Cost of surrogatesIn linear regression, the process of fitting involves solving a set of linear equations once.For moving least squares, we need to form and solve the system at every prediction point.With radial basis neural networks we have to optimize the selection of neurons, which will again entail multiple solutions of the linear system. We may find the best spread by minimizing cross-validation errors.Kriging, our next surrogate is even more expensive, we have a spread constant in every direction and we have to perform optimization to calculate the best set of constants.With many hundreds of data points this can become significant computational burden.
2Kriging philosophyWe assume that the data is sampled from an unknown function that obeys simple correlation rules.The value of the function at a point is correlated to the values at neighboring points based on their separation in different directions.The correlation is strong to nearby points and weak with far away points, but strength does not change based on location.This is often grossly wrong because a function may be fast undulating in one corner of the design space and vary slowly in another corner.Still, Kriging is a good surrogate, and it may be the most popular surrogate in academia.Normally Kriging is used with the assumption that there is no noise so that it interpolates exactly the function values.It works out to be a local surrogate, and it uses functions that are very similar to the radial basis functions.
3Reminder: Covariance and Correlation Covariance of two random variables X and YThe covariance of a random variable with itself is the square of the standard deviationCovariance matrix for a vector contains the covariances of the componentsCorrelationThe correlation matrix has 1 on the diagonal.
4Correlation between function values at nearby points x=10*rand(1,10)xnear=x+0.1; xfar=x+1;ynear=sin(xnear)y=sin(x)yfar=sin(xfar)rfar=corrcoef(y,yfar)0.4229r=corrcoef(y,ynear)0.9894
5Gaussian correlation function Correlation between point x and point sy10=sin(10*x); y10near=sin(10*xnear)r10=corrcoef(y10,y10near)0.4264For the function we would like to estimate
6Universal KrigingLinear trend modelSystematic departurexyKrigingSampling data pointsSystematic DepartureLinear Trend ModelNamed after a South African mining engineer D. G. KrigeAssumption: Systematic departures Z(x) are correlatedGaussian correlation function C(x,s,θ) is most popular
7Simple KrigingKriging started without the trend, and it is not clear that one cannot get by without it.Simple Kriging is uses a covariance structure with a constant standard deviation.The most popular correlation structure is GaussianThe standard deviation measures the uncertainty in function values. If we have dense data, that uncertainty will be small, and if the data is sparse the uncertaty will be large.How do you decide whether the data is sparse or dense?
8Prediction and shape functions Simple Kriging prediction formulaR is the correlation matrix of the data points.The equation is linear in r, which means that the basis functions are the exponentialsThe equation is linear in y, which is in common with linear regression.
9Prediction varianceSquare root of variance is still called standard errorThe uncertainty at any x is still normally distributed.9
10Finding the thetas The thetas and sigma must be found by optimization. Maximize the likelihood of the data.For a given curve, we can calculate the probability that if the curve is exact, we would have sampled the data.Minimize the cross-validation error.Each set of theta acts like a different surrogate.Both problems are ill-conditioned and expensive for large number of data points.Watch for thetas reaching their higher bounds!Prediction variance equation does not count for the uncertainty in the theta values.