0. Extremum Seeking Control for Real-Time Optimization
Miroslav Krstic
UC San Diego
IEEE Advanced Process Control Applications for Industry Workshop Vancouver, 2007

1. Example of Single-Parameter Maximum Seeking
Plant

2. Example of Single-Parameter Maximum Seeking

3. Topics - Theory History
Single parameter ES, how it works, and stability analysis by averaging
Multi-parameter ES
ES in discrete time
ES with plant dynamics and compensators for performance improvement
Internal model principle for tracking parameter changes
Slope seeking
Limit cycle minimization via ES

4. Topics - Applications PID tuning
Internal combustion (HCCI) engine fuel consumption minimization
Compressor instabilities in jet engines
Combustion instabilities
Formation flight
Fusion reflected RF power
Thermoacoustic coolers
Beam matching in particle accelerators
Flow separation control in diffusers
Autonomous vehicles without position sensing

5. History Leblanc (1922) - electric railways
Early Russian literature (1940’s) - many papers
Drapper and Li (1951) - application to IC engine spark timing tuning
Tsien (1954) - a chapter in his book on Engineering Cybernetics
Feldbaum (1959) - book Computers in Automatic Control Systems
Blackman (1962 chap. in book by Westcott) - nice intuitive presentation of ES
Wilde (1964) - a book
Chinaev (1969) - a handbook on self-tuning systems
Papers by[Morosanov], [Ostrovskii], [Pervozvanskii], [Kazakevich], [Frey, Deem, and Altpeter], [Jacobs and Shering], [Korovin and Utkin] - late 50s - early 70’s
Meerkov (1967, 1968) - papers with averaging analysis
Sternby (1980) - survey
Astrom and Wittenmark (1995 book) - rates ES as one of the most promising areas for adaptive control

6. Recent Developments
Krstic and Wang (2000, Automatica) - stability proof for single-parameter general dynamic nonlinear plants
Choi, Ariyur, Wang, Krstic - discrete-time, limit cycle minimization, IMC for parameter tracking, etc.
Rotea; Walsh; Ariyur - multi-parameter ES
Ariyur - slope seeking
Tan, Nesic, Mareels (2005) - semi-global stability of ES
Other approaches: Guay, Dochain, Titica, and coworkers; Zak, Ozguner, and coworkers; Banavar, Chichka, Speyer; Popovic, Teel; etc.
Applications not presented in this workshop:
Electromechanical valve actuator (Peterson and Stephanopoulou)
Artificial heart (Antaki and Paden)
Exercise machine (Zhang and Dawson)
Shape optimization for magnetic head in hard disk drives (UCSD)
Shape optimization of airfoils and automotive vehicles (King, UT Berlin)

7. ES Book

8. Tutorial Topics Covered in the Book
Introduction, history, single-parameter stability analysis
Plant dynamics, compensators, and IMC for tracking parameter changes
Limit cycle minimization via ES
Multi-parameter ES
ES in discrete time
Slope seeking
Compressor instabilities in jet engines
Combustion instabilities
Formation flight
Anti-skid braking
Bioreactor
Thermoacoustic coolers
Internal combustion engines
Flow separation control in diffusers
Beam matching in particle accelerators
PID tuning
Autonomous vehicles without position sensing

9. Basic Extremum Seeking - Static Map
Plant

10. How Does It Work? Plant Estimation error: Loc. Analysis - neglect
quadratic terms:

11. How Does It Work?
Plant

12. How Does It Work?
Plant
Demodulation:

13. How Does It Work?
Plant
Since
then

14. How Does It Work?
Plant
high frequency terms - attenuated by integrator

15. How Does It Work?
Plant
Stable because

16. Stability Proof by Averaging
Plant
Full nonlinear time-varying model:

17. Stability Proof by Averaging
Plant
Average system:
Average equilibrium:

18. Stability Proof by Averaging
Plant
Jacobian of the average system:

19. Stability Proof by Averaging
Plant
Theorem. For sufficiently large w there exists a unique exponentially stable periodic solution of period 2p/w and it satisfies
Speed of convergence proportional to 1/w, a2, k,

20. Stability Proof by Averaging
Plant
Output performance:

21. Based on contributions by: Nick Killingsworth
PID Tuning Using ES
Based on contributions by: Nick Killingsworth

22. Background & Motivation
Proportional-Integral-Derivative (PID) Control
Consists of the sum of three control terms
- Proportional term:
- Integral term:
- Derivative term:
Often poorly tuned (Astrom [1995], etc.)
e(t) = r(t) – y(t)
r(t) reference signal
y(t) measured output

23. Background – PID + - We use a two degree of freedom controller
The derivative term only acts on y(t)
This avoids large control effort when there is a step change in the reference signal + -

24. Extremum Seeking Algorithm
Tuning Scheme
Continuous Time
Step
function
r(t)
u(t)
y(t)
J(qk) G - qk
Extremum Seeking Algorithm
Discrete Time

25. Extremum Seeking
Simple - three lines of code

26. Extremum Seeking Tuning Scheme
Implementation
Run Step response experiment with ZN PID parameters
Calculate J

27. Extremum Seeking Tuning Scheme
Implementation
Run Step response experiment with ZN PID parameters
Calculate J
Calculate next set of PID parameters using discrete ES tuning method

28. Extremum Seeking Tuning Scheme
Implementation
Run Step response experiment with ZN PID parameters
Calculate J
Calculate next set of PID parameters using discrete ES tuning method
Run another step response experiment with new PID parameters

29. Extremum Seeking Tuning Scheme
Implementation
Run Step response experiment with ZN PID parameters
Calculate J
Calculate next set of PID parameters using discrete ES tuning method
Run another step response experiment with new PID parameters
Repeat 2-4 set number of times or until J falls below a set value
Repeat

30. Implementation – Cost Function
Cost Function J(qk)
Used to quantify the controller’s performance
Constructed from the output error of the plant and the control effort during a step response experiment
Has discrete values at the completion of each step response experiment
where T is the total sample time of each step response experiment
q is a vector containing the PID parameters:

31. Implementation – Cost Function
Cost Function J(qk)
t0 is the time up until which zero weightings are placed on the error.
This shifts the emphasis of the PID controller from the transient phase of the response to that of minimizing the tracking error after the initial transient portion of the response to

32. Example Plants
Four systems with dynamics typical of some industrial plants have been used to test the ES PID tuning method
Time delay
Large time delay
Single pole of order eight
Unstable zero

33. Results
Ziegler-Nichols values used as initial conditions in the ES tuning algorithm
Results compared to three other popular PID tuning methods:
- Ziegler-Nichols (ZN)
- Internal model control (IMC)
- Iterative feedback tuning (IFT, Gevers, ‘94, ‘98)

34. Results - a) Evolution of Cost Function b) Evolution of PID Parameters
c) Step Response of output
d) Step Response of controller

35. Results - a) Evolution of Cost Function b) Evolution of PID Parameters
c) Step Response of output
d) Step Response of controller

36. Results - a) Evolution of Cost Function b) Evolution of PID Parameters
c) Step Response of output
d) Step Response of controller

37. Results - a) Evolution of Cost Function b) Evolution of PID Parameters
c) Step Response of output
d) Step Response of controller

38. Results – Cost Function Comparison
Step Response of output
The following cost functions
were minimized using ES:

39. Results – Cost Function Comparison
Step Response of output
The following cost functions
were minimized using ES:

40. Results – Cost Function Comparison
Step Response of output
The following cost functions
were minimized using ES:

41. Results – Cost Function Comparison
Step Response of output
The following cost functions
were minimized using ES:

42. Results – Cost Function Comparison
Step Response of output
The following cost functions
were minimized using ES:

43. Actuator Saturation
Saturation of 1.6 applied to control signal for plant G1
ES and IMC compared with and without the addition of an anti windup scheme
Tracking anti-windup scheme

44. Actuator Saturation Step response of output
Control signal during step response

45. Effects of Noise Band-limited white noise has been added to output
Power spectral density =
Correlation time = 0.2
Independent noise signal for each iteration
Simulations on plant G1

46. Effects of Noise a) Evolution of Cost Function
b) Evolution of PID Parameters
c) Step Response of output
d) Step Response of controller

47. Selecting Parameters of ES Scheme
Must select
a, perturbation step size
g, adaptation gain
w, perturbation frequency
h, high-pass filter cut-off frequency
Looks like have more parameters to pick than we started out with!
However, ES tuning is less sensitive to parameters than PID controller.

48. Selecting Parameters of ES Scheme
ES Tuning Parameters K Ti Td
1.01
31.5
7.16
1.00
31.1
7.6
31.3
7.54
31.0
7.65

49. Selecting Parameters of ES Scheme
Need to select an adaptation gain g and perturbation amplitude a for EACH parameter to be estimated
In the case of a PID controller, q = [K, Ti, Td], so we need three of each.
The modulation frequency is determined by:
where 0 < a < 1
The highpass filter (z-1)/(z+h) is designed with 0<h<1
with the cutoff frequency well below the modulation frequency .
Convergence rate is directly affected by choice of a and g, as well as by cost function shape near minimizer.

50. Example of ES-PID tuner GUI

51. Punch Line
ES yields performance as good as the best of the other popular tuning methods
Can handle some nonlinearities and noise.
The cost function can be modified such that different performance attributes are emphasized

52. Control of HCCI Engines
Based on contributions by: Nick Killingsworth (UCSD),
Dan Flowers and Sal Aceves (Livermore Lab),
and Mrdjan Jankovic (Ford)

53. HCCI = ? HCCI = Homogeneous Charge Compression Ignition
Low NOx emissions like spark-ignition engines
High efficiency like Diesel engines
More promising in near term than fuel cell/hydrogen engines

54. HCCI Engine Applications
Distributed power generation
Automotive hybrid powertrain

55. What is the difference between Spark Ignition, Diesel, and HCCI engines?

56. Direct injection engine
Categories of Engines
Compression Ignition
Spark ignition
Homogeneous charge
HCCI
Spark ignition engine
Inhomogeneous charge
Diesel
Direct injection engine

57. Spark Ignition Engine
Basic engine thermodynamics: engine efficiency increases as the compression ratio and =cp/cv (ratio of specific heats) increase
g = 1.4 for air
g = 1.35 for fuel and air mixture
Engine Efficiency
SI engines 1 3 5 7 9 11 13 15 17 19
compression ratio
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
engine indicated efficiency
g=1.3
g=1.4

58. Diesel Engine
Highly efficient because they compress only air ( is high) and are not restricted by knock (compression ratio is high)
g = 1.4 for air
g = 1.35 for fuel and air mixture
Engine Efficiency 1 3 5 7 9 11 13 15 17 19
compression ratio
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
engine indicated efficiency
g=1.3
g=1.4
Diesel engines
SI engines

59. Diesel and HCCI engines
Compression ratio not restricted by “knock” (autoignition of gas ahead of flame in spark ignition engines) a efficiency comparable to Diesel
Diesel and HCCI engines
g = 1.4 for air
g = 1.35 for fuel and air mixture
Engine Efficiency 1 3 5 7 9 11 13 15 17 19
compression ratio
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
engine indicated efficiency
g=1.3
g=1.4
SI engines

60. HCCI Engine Potential for high efficiency (Diesel-like)
Low NOx and PM (unlike Diesel)
BUT, no direct trigger for ignition - requires feedback to control the timing of ignition!

61. Experiment at Livermore Lab
Caterpillar 3406 natural gas spark ignited engine converted to HCCI
Set up for stationary power generation (not automotive)

62. Actuators Combustion timing (output) is very sensitive to
intake temperature (input)
Heated
Intake Air
Cooled
Mixing “Tees”(x 6)
Controlled Intake Temperature
to Individual Cylinders
Cold Manifold
Hot Manifold
Valve Actuators

63. Overall Architecture: Sensors and Software
User interface
Valve Position
Real-Time Controller
PC running Labview RT OS
Cylinder Pressure
Crank Angle Position

64. ES used to MINIMIZE FUEL CONSUMPTION of HCCI engine by tuning combustion timing setpoint
CA50 e
CA50 +
Extremum
Seeking m
fuel . T
intake
HCCI Engine S PI -
CA50 SP

65. ES delays the combustion timing 6 crank angle degrees, reducing fuel consumption by > 10%
The PI controller found using ES is applied and tested during a load disturbance. The load starts at 21 kW and is increased via a step change to 25 kW around 30 seconds. The dip in the rpm is a result of the extra load being applied to the engine. PI: 1600 rpm, load = kW, theta = [2.53,0.005,0] Kf = 0

66. Larger adaptive gain: ES finds same minimizer, but much more quickly
The PI controller found using ES is applied and tested during a load disturbance. The load starts at 21 kW and is increased via a step change to 25 kW around 30 seconds. The dip in the rpm is a result of the extra load being applied to the engine. PI: 1600 rpm, load = kW, theta = [2.53,0.005,0] Kf = 0

67. Axial Flow (Jet Engine-Like) Compressor Control
Problem Statement
time
Pressure Rise
Experimental Results
Active controls for rotating stall only reduce the stall oscillations but they do not bring them to zero nor do they maximize pressure rise.
Extremum seeking to optimize compressor operating point.
Extremum seeking stabilizes the maximum pressure rise.
bleed valve
Pressure rise
Caltech
COMPRESSOR
Smaller, lighter compressors; higher payload in aircraft
Motivation
Active controls for compressor rotating stall ensure that the descent into rotating stall is not abrupt, but they do not eliminate it completely, nor do they ensure that the pressure rise is maximal possible.
Extremum seeking tunes the compressor operating point so that it be kept at the stall inception point where stall oscillations are absent and the pressure rise is the maximal physically possible. This tuning ensures that the operating point tracks variations in the environment, pilot commands, and aging. The maximization of pressure allows the reduction of compressor size, and an increase in the thrust-to-weight ratio in aircraft.
The time trace shows an extremum seeking transient. The compressor starts from a regime where the stall controller is already active but it is not optimal, so rotating stall, although moderate, is present. The extremum seeker forces the system completely out of rotating stall but it also keeps it from sliding down the stable side of the compressor characteristic into low values of pressure rise. It keeps stall at zero and the pressure at its maximum. The oscillations present after the transient are just measurement noise.
The experiment shown was performed on an axial-flow compressor at Caltech, in the laboratory of Prof. Richard Murray.
The complete block diagram of the system shows that a stall controller implemented through air injector is in the inner regulation loop, while the outer, optimization loop, is the extremum seeker. The method uses sinusoidal probing to estimate the gradient of the compressor operating map, and drives the compressor to an operating point where the gradient is zero.
Extremum seeking is implemented via a bleed valve (low bandwidth, cheap).
Professor Krstic has patented this compressor optimization scheme through University of California.
EXTREMUM
SEEKER
Air Injection
Stall Controller
washout
filter
sin wt

68. Combustion Instability Control
Problem Statement
time
ext. seeking suppresses oscillations
Experiment on UTRC 4MW combustor
Rayleigh criterion-based controllers, which use phase-shifted pressure measurements and fuel modulation, have emerged as prevalent
The length of the phase needed varies with operating conditions. The tuning method must be non-model based.
Tuning allows operation with minimum oscillations at lean conditions
Reduced engine size, fuel consumption and NOx emissions
Motivation
sin wt
Pressure
washout
filter
COMBUSTOR
Phase-Shifting
Controller
Frequency/
amplitude
observer
fuel
Thermoacoustic oscillations are common in combustion chambers operating at high power or in a lean (low emission) regime.
Active control that uses fuel modulation as actuation and a phase-shifted measurement of pressure has emerged as the most practical way of mitigating those oscillations. This approach is directly related to the classical Rayleigh’s instability criterion.
The length of the phase needed varies with operating conditions and cannot be determined a priori using a model.
The extremum seeking was used in this program to tune the phase of the phase-shifting controller designed by Dr. Andrzej Banaszuk. He performed the experiments on an industrial size combustor at UTRC.
The time trace shows how the bulk mode of the pressure oscillations gets dramatically supressed after turning on the extremum seeker.
The complete block diagram of the system shows that the phase shifting controller performs the task of local regulation and the extremum seeker performs its optimization in an outer loop.
The extremum seeking scheme estimates the gradient of the map of oscillation amplitudes and drives the phase shift to a value where the gradient of the map is zero, i.e., the optimal phase shift.
Extremum seeking minimizes oscillations under varying operating conditions, pilot commands, aging. It allow the reduction in engine size (increased thrust/weight ratio), in fuel consumption, and in NOx emissions.
phase
EXTREMUM SEEKER

69. Formation Flight Engine Output Minimization
Classical linear decoupled aircraft model with conventional controls—deviation from trim conditions of aircraft states are small
Order of the model is 16, states are the conventional rigid body parameters, plus the actuators
The reference flight condition used for linearization is cruise at M=0.77, ft and a weight of 650 klb
Aircraft dynamics in free flight is the basis for modeling aircraft dynamics in the wake
Experimental wake data of the C-5 used for generating simulation model
All forces and moments are assumed to depend only upon relative positions through the feedback mechanism shown in this diagram
Specifically, upwash-induced longitudinal forces and pitching moment are modeled by averaging the upwash along the wingspan
Upwash-induced rolling moment is calculated with modified strip theory and
Sidewash-induced sideforce, yawing and rolling moment are modeled by assuming a uniform sidewash equal to that at the centerline
Aerodynamic interference results from flying in formation and is modeled as three nonlinear static maps … As shown in this block diagram, to which I will go back in a few moments
Tune reference inputs yref and zref to the autopilot of the wingman to maximize its downward pitch angle or to minimize its engine output

70. Simulation of C-5 Galaxy transport airplane for a brief encounter of “clear air turbulence”
Dryden Vertical and Lateral Turbulence
ES is switched to autopilot after 3 seconds of CAT, enough to detect it by nz increase

71. Thermoacoustic Cooler (M. Rotea)
Electric energy
Acoustic energy
Heat pumping
Standing sound wave creates the refrigeration cycle
Resonance tube
Stack
Hot-end heat exchangers
Electro-dynamic driver
Cold-end heat exchangers
Pressurized He-Ar mixture 3 2
Heat Pumping
1-2: adiabatic compression and displacement
2-3: isobaric heat transfer (gas to solid)
3-4: adiabatic expansion and displacement
4-1: isobaric heat transfer (solid to gas)
Solid surface (stack plate)
Gas particle in a standing wave 1 4 QL QH

72. Thermoacoustic Cooler
Moving piston (varying resonator’s stiffness)
Heat exchangers
Helmholtz resonator
Neck (mass)
Volume (stiffness)
Electro-dynamic
Driver
Tuning Variables
Piston position (acoustic impedance)
Driver frequency

73. ES with PD compensator + POS Command + + + FREQ Command Cooling Power
LPF
Integrator PD + + PD
Integrator
LPF +
FREQ Command
Cooling
Power
Calculation
Tunable
Cooler
HPF

74. Experiment – Fixed Operating Condition
Cooling Performance with ESC
POS in.
FREQ Hz
POWER W 4
141
22.65
142
29.92
143
35.67
144
28.63
145
21.25 5
15.89
34.12
39.68
146
35.12
148
19.34 6
140
4.95
9.00
18.55
23.86
35.99
147
41.28
38.00
149
30.36
150
19.36 7
16.34
33.34
41.21
40.70
151
34.69
153
19.63 8
32.16
152
35.60
31.74 50
100
150
200
250
300 20 40 60
Cooling Power (Watt) 2 4 6 8
Piston Position (in)
140
145
Driving Frequency (Hz)
Time (sec)
ESC ON
The first test was done for Esc with fixed conditions, which means all other operating parameters are fixed, e.g.,mean pressure, flow rates, inlet temperatures of 2nd loop, gas mixture are not changed throughout the process. Before we did the test, the static map was measured. Here x is the piston position measured from the motor side, f is the driving freq, and Qc_dot is the cooling power. The static map shows that the optimal cooling power is about 41 Watt, located around 6~7 inch and 147~149 Hz
ESC quickly finds optimum operating point (41.3W, 147Hz, 6.2in)

75. Experiment – Varying Operating Condition50
100
150
200
250
300
350
400
Cooling Power (Watt) 4 6 8 10 12
Piston Position (in)
140
145
155
Driving Frequency (Hz) 20 40 60 80
Time (sec)
Flow Rate (ml/s)
Cold Side
Hot Side
ESC ON
Flow Rate Change
ESC tracks optimum after cold-side flow rate is increased

76. ES for the Plasma Control in the Frascati Fusion Reactor
Contribution by Luca Zaccarian (U. Rome, Tor Vergata)

77. Optimization Objective
Framework:
Additional Radio Frequency heating injected in the plasma by way of Lower Hybrid (LH) antennas: plasma reflects some power
Goal:
Optimize coupling between the Lower Hybrid antenna and tha plasma, during the LH pulse

78. Reflected Power Map Reflected power: Convex fcn of edge density
Convex fcn of edge position
Possible approaches to optimize:
1. Move the antenna
(too slow!)
2. Move the plasma
(viable – adopted here)

79. Probing not Allowed - Modified ES Scheme
Knob
Extracted
Input sinusoid
Extracted
Output sinusoid

80. Experimental results with medium gain
K = 300
Safety saturation limits performance
Control action is quite aggressive.

81. Experimental results with lower gain
K = 200
(Antenna has been moved)
Graceful convergence to the minimum reflected power

82. Gain too high - instability
K = 350
Instability
Gain is
too large

83. Experiments - Summary Input/output plane representation:
K = 300: saturation prevents reaching the minimum
K = 200: graceful convergence to minimum (slight overshoot)
K = 350: gain too high – all the curve is explored