1. Fixed-Point Iteration Douglas Wilhelm Harder Department of Electrical and Computer Engineering University of Waterloo Copyright © 2007 by Douglas Wilhelm Harder. All rights reserved. ECE 204 Numerical Methods for Computer Engineers

2. Fixed-Point Iteration In this topic, we will look at: –how fixed-point iteration can be used to solve certain problems numerically, and –some of the various limitations of iterative numerical methods including: determining convergence, slow convergence, and non-convergence

3. Fixed-Point Iteration Everyone has at some point punched in an arbitrary number on a scientific calculator and started pushing one of the buttons We will look at two examples: sin and cos

4. Fixed-Point Iteration Starting with x = 0, if I repeatedly hit the cos button, I get the sequence: x = 0 x = 1 x = 0.540302305868140 x = 0.857553215846393 x = 0.654289790497779 x = 0.793480358742566 x = 0.701368773622757 x = 0.763959682900654 x = 0.722102425026708 x = 0.750417761763761 x = 0.731404042422510 x = 0.744237354900557 x = 0.735604740436347 x = 0.741425086610109 x = 0.737506890513243 x = 0.740147335567876 x = 0.738369204122323 x = 0.739567202212256 x = 0.738760319874211

5. Fixed-Point Iteration Looking at the first three digits of the last three iterations, we note that they appear to be converging: 0.738369204122323 0.739567202212256 0.738760319874211

6. Fixed-Point Iteration After applying cos 100 times using Matlab, I get the sequence: x = 0.739085133215166 x = 0.739085133215157 x = 0.739085133215163 x = 0.739085133215159 x = 0.739085133215162 x = 0.739085133215160 x = 0.739085133215161 x = 0.739085133215160 x = 0.739085133215161 x = 0.739085133215160 x = 0.739085133215161

7. Fixed-Point Iteration If, however, you set your calculator to degrees instead of radians, you would have found a different sequence: >> x = 0; >> for i = 1:6 x = cos(x/180*pi) end x = 1 x = 0.999847695156391 x = 0.999847741545212 x = 0.999847741531084 x = 0.999847741531088

8. Fixed-Point Iteration In this case, the sequence converged significantly faster, but to what?

9. Fixed-Point Iteration Given an equation of the form x = f(x) if |f (1) (x)| < 1 in a neighbourhood of a solution to this equation, then for any initial point x 0 in this neighbourhood, the sequence x k + 1 = f(x k ) will converge to the solution to this equation

10. Fixed-Point Iteration The examples we have been looking at were finding points such that: x = cos(x) This will differ depending on whether you are using radians or degrees:

11. Fixed-Point Iteration We will now look at one of the longest known examples of fixed-point iteration Starting with any value and repeated apply the function f(x) = x/2 + 1/x For centuries, it has been known that this converges to

12. Fixed-Point Iteration To see why, recall that this converges to a solution of x = x/2 + 1/x which is also a solution of x 2 = x 2 /2 + 1 or x 2 /2 = 1 x 2 = 2

13. Newton’s Method We also get this sequence by applying Newton’s method which was covered in first year The function f(x) = x 2 – 2 has two roots, and we can find these roots using Newton’s method:

14. Newton’s Method Thus, we iterate Looking at the right hand side, it simplifies to:

15. Fixed-Point Iteration For example, consider the equation x = 2x(x – 1) This has two solutions, namely x = 0 and x = 0.5 If we start with x 0 = 0.1 and define f(x) = 2x(x – 1), we get the sequence

16. Fixed-Point Iteration Sequence is kxkxk |x k – x k – 1 | 00.1– 10.180.08 20.29520.1152 30.416113920.1209 40.48592625116446720.06981 50.49960385918742870.01368 6 0.49999968614491320.0003958 7 0.49999999999980300.0000003139 80.50.0000000000001970 90.50

17. Fixed-Point Iteration To solve any such problem, we could continue to iterate until the sequence no longer changes, however, this does not always work Consider the slight modification, solving x = 2.4x(x – 1) which has the solution x = 7/12 ≈ 0.58333 ⋅⋅

18. Fixed-Point Iteration Starting with x 0 = 0.1, we converge much more slowly, and after 32 iterations: kxkxk |x k – x k – 1 | 320.58333333333334570.432 10 –11 330.58333333333332840.173 10 –11 340.58333333333333540.070 10 –11 350.58333333333333250.029 10 –11 360.58333333333333370.012 10 –11 370.58333333333333320.005 10 –11 380.58333333333333340.002 10 –11 390.58333333333333320.002 10 –11 400.58333333333333340.002 10 –11

19. Fixed-Point Iteration This sequence will never converge, and therefore we need a weaker halting condition We will continue iterating until: |x k – x k – 1 | <  step where  step is a suitably chosen predetermined criteria

20. Fixed-Point Iteration Unfortunately, this may also lead to erroneous results

21. Fixed-Point Iteration Suppose we are trying to solve x = sin(x) which has only one solution x = 0 After 1000 iterations, we still have: x 1000 = 0.04802218067056788 x 1001 = 0.04800372523478609 x 1002 = 0.04798529106705650 x 1003 = 0.04796687812655410

22. Fixed-Point Iteration In this case, the rate of convergence is very, very slow The following table lists the number of iterations required to satisfy the terminating condition:

23. Fixed-Point Iteration The following table shows how many iterations are required  step Iterations xkxk 10 –4.. 1230.0841946299701288 10 –5.. 16580.03913176037625081 10 –6.. 87850.01816976184230423 10 –7.. 418700.008434193162746495 10 –8.. 1954410.003914852569606909 10 –9.. 9082570.001817119078995468 10 –11 42168590.0008434325248239295 10 –12 195740330.0003914867496521149

24. Fixed-Point Iteration Notice that a small step size does not suggest that we are anywhere near the solution? After 20 million iterations: –the step size is less than 10 –12, but –the value 0.0003915867·· has only three zeros after the decimal place

25. Fixed-Point Iteration Therefore, we must restrict the number of iterations required to some fixed amount, say N Thus, if we iterate N times and have not satisfied our halting conditions, we stop and indicate that the method failed to converge

26. Fixed-Point Iteration If we stop due to iterating N times, this may mean that either: –a solution does not exist, –we chose a poor initial condition, or –we are using an inappropriate method for finding a solution

27. Fixed-Point Iteration Alternatively, to find a solution to x = sin(x), we could define f(x) = sin(x) – x and then iterate Thus, we get: x 0 = 0.1 x 1 = – 0.0000333452420668 x 2 = 0.00000000001074791837

28. Fixed-Point Iteration In this topic, we have seen a number of examples of fixed-point iteration –it may be possible to solve equations of the form x = f(x) by repeated applying f(x) –the form of the iteration will affect the convergence –it is necessary to define a halting condition stopping if the difference is less than  step –the sequence may not converge so we may have to halt after some fixed number of iterations

29. Usage Notes These slides are made publicly available on the web for anyone to use If you choose to use them, or a part thereof, for a course at another institution, I ask only three things: –that you inform me that you are using the slides, –that you acknowledge my work, and –that you alert me of any mistakes which I made or changes which you make, and allow me the option of incorporating such changes (with an acknowledgment) in my set of slides Sincerely, Douglas Wilhelm Harder, MMath dwharder@alumni.uwaterloo.ca