Numerical ordinary differential equations
272829
225586144
2008-07-14T12:52:02Z
70.168.100.70
/* The Euler method */
[[Image:Numerical integration illustration, h=1.png|right|thumb|Illustration of numerical integration for the differential equation <math>y'=y, y(0)=1.</math> Blue: the [[Euler method]], green: the [[midpoint method]], red: the exact solution, <math>y=e^t.</math> The step size is <math>h=1.0.</math>]]
[[Image:Numerical integration illustration, h=0.25.png|right|thumb|The same illustration for <math>h=0.25.</math> It is seen that the midpoint method converges faster than the Euler method.]]
'''Numerical ordinary differential equations''' is the part of [[numerical analysis]] which studies the numerical solution of [[differential equation|ordinary differential equations]] (ODEs). This field is also known under the name ''[[numerical integration]],'' but some people reserve this term for the computation of [[integral]]s.
Many differential equations cannot be solved analytically, in which case we have to satisfy ourselves with an approximation to the solution. The [[algorithm]]s studied here can be used to compute such an approximation. An alternative method is to use techniques from [[calculus]] to obtain a [[series expansion]] of the solution.
Ordinary differential equations occur in many scientific disciplines, for instance in [[mechanics]], [[chemistry]], [[biology]], and [[economics]]. In addition, some methods in [[numerical partial differential equations]] convert the [[partial differential equation]] into an ordinary differential equation, which must then be solved.
== The problem ==
We want to approximate the solution of the differential equation
:<math>y'(t) = f(t,y(t)), \qquad y(t_0)=y_0, \qquad\qquad (1)</math>
where ''f'' is a function that maps [''t''<sub>0</sub>,∞) × '''R'''<sup>d</sup> to '''R'''<sup>d</sup>, and the initial condition ''y''<sub>0</sub> ∈ '''R'''<sup>d</sup> is a given vector.
The above formulation is called an ''[[initial value problem]]'' (IVP). The [[Picard-Lindelöf theorem]] states that there is a unique solution, if ''f'' is [[Lipschitz continuous]]. In contrast, ''[[boundary value problem]]s'' (BVPs) specify (components of) the solution ''y'' at more than one point. Different methods need to be used to solve BVPs, for example the [[shooting method]], [[multiple shooting]] or global methods like [[finite difference]]s or [[collocation method]]s.
Note that we restrict ourselves to ''first-order'' differential equations (meaning that only the first derivative of ''y'' appears in the equation, and no higher derivatives). However, a higher-order equation can easily be converted to a first-order equation by introducing extra variables. For example, the second-order equation ''y''<nowiki>''</nowiki> = −''y'' can be rewritten as two first-order equations: ''y''<nowiki>'</nowiki> = ''z'' and ''z''<nowiki>'</nowiki> = −''y''.
== Methods ==
Two elementary methods are discussed to give the reader a feeling for the subject. After that, pointers are provided to other methods (which are generally more accurate and efficient). The methods mentioned here are analysed in the next section.
=== The Euler method ===<!-- This section is linked from [[Generating trigonometric tables]] -->
{{details|Euler method}}
A brief explanation:
From any point on a curve, you can find an approximation of a nearby point on the curve by moving a short distance along a line [[tangent]] to the curve.
Rigorous development:
Starting with the differential equation (1), we replace the derivative ''y''<nowiki>'</nowiki> by the [[finite difference]] approximation
:<math> y'(t) \approx \frac{y(t+h) - y(t)}{h}, \qquad\qquad (2) </math>
which when re-arranged yields the following formula
:<math> y(t+h) \approx y(t) + hy'(t) \qquad\qquad </math>
and using (1) gives:
:<math> y(t+h) \approx y(t) + hf(t,y(t)). \qquad\qquad (3) </math>
This formula is usually applied in the following way. We choose a step size ''h'', and we construct the sequence ''t''<sub>0</sub>, ''t''<sub>1</sub> = ''t''<sub>0</sub> + ''h'', ''t''<sub>2</sub> = ''t''<sub>0</sub> + 2''h'', … We denote by ''y''<sub>''n''</sub> a numerical estimate of the exact solution ''y''(''t''<sub>''n''</sub>). Motivated by (3), we compute these estimates by the following [[recursion|recursive]] scheme
:<math> y_{n+1} = y_n + hf(t_n,y_n). \qquad\qquad (4) </math>
This is the ''[[Euler method]]'' (or ''[[forward Euler method]]'', in contrast with the ''backward Euler method'', to be described below). The method is named after [[Leonhard Euler]] who described it in [[1768]].
The Euler method is an example of an [[explicit and implicit methods|''explicit'']] method. This means that the new value ''y''<sub>''n+1''</sub> is defined in terms of things that are already known, like ''y''<sub>''n''</sub>.
=== The backward Euler method ===
If, instead of (2), we use the approximation
:<math> y'(t) \approx \frac{y(t) - y(t-h)}{h}, \qquad\qquad (5)</math>
we get the ''backward Euler method'':
:<math> y_{n+1} = y_n + hf(t_{n+1},y_{n+1}). \qquad\qquad (6)</math>
The backward Euler method is an [[explicit and implicit methods|''implicit'']] method, meaning that we have to solve an equation to find ''y''<sub>''n''+1</sub>. One often uses [[fixed point iteration]] or (some modification of) the [[Newton's method|Newton–Raphson method]] to achieve this. Of course, it costs time to solve this equation; this cost must be taken into consideration when one selects the method to use. The advantage of implicit methods such as (6) is that they are usually more stable for solving a [[stiff equation]], meaning that a larger step size ''h'' can be used.
=== Generalizations ===
The Euler method is often not accurate enough. In more precise terms, it only has order one (the concept of ''order'' is explained below). This caused mathematicians to look for higher-order methods.
One possibility is to use not only the previously computed value ''y''<sub>''n''</sub> to determine ''y''<sub>''n''+1</sub>, but to make the solution depend on more past values. This yields a so-called ''multistep method''. Perhaps the simplest is the [[Leapfrog method]] which is second order and (roughly speaking) relies on two time values.
Almost all practical multistep methods fall within the family of [[linear multistep method]]s, which have the form
:<math> \alpha_k y_{n+k} + \alpha_{k-1} y_{n+k-1} + \cdots
+ \alpha_0 y_n</math>
::<math> = h \left[ \beta_k f(t_{n+k},y_{n+k}) + \beta_{k-1}
f(t_{n+k-1},y_{n+k-1}) + \cdots + \beta_0 f(t_n,y_n) \right]. </math>
Another possibility is to use more points in the interval [''t''<sub>''n''</sub>,''t''<sub>''n''+1</sub>]. This leads to the family of [[Runge–Kutta method]]s, named after [[Carl David Tolmé Runge|Carl Runge]] and [[Martin Wilhelm Kutta|Martin Kutta]]. One of their fourth-order methods is especially popular.
Both ideas can also be combined. The resulting methods are called ''general linear methods''.
=== Advanced features ===
A good implementation of one of these methods for solving an ODE entails more than the time-stepping formula.
It is often inefficient to use the same step size all the time, so ''variable step-size methods'' have been developed. Usually, the step size is chosen such that the (local) error per step is below some tolerance level. This means that the methods must also compute an ''error indicator'', an estimate of the local error.
An extension of this idea is to choose dynamically between different methods of different orders (this is called a
''variable order method''). [[Extrapolation method]]s are often used to construct various methods of different orders.
Other desirable features include:
* ''dense output'': cheap numerical approximations for the whole integration interval, and not only at the points ''t''<sub>0</sub>, ''t''<sub>1</sub>, ''t''<sub>2</sub>, ...
* ''event location'': finding the times where, say, a particular function vanishes.
* support for [[parallel computing]].
* when used for integrating with respect to time, time reversibility
=== Alternative methods ===
Many methods do not fall within the framework discussed here. Some classes of alternative methods are:
* ''multiderivative methods'', which use not only the function ''f'' but also its derivatives. This class includes ''Hermite–Obreschkoff methods'' and ''[[Runge–Kutta–Fehlberg method|Fehlberg methods]]'', as well as methods like the [[Parker–Sochacki method]], which compute the coefficients of the [[Taylor series]] of the solution ''y'' recursively.
* ''methods for second order ODEs''. We said that all higher-order ODEs can be transformed to first-order ODEs of the form (1). While this is certainly true, it may not be the best way to proceed. In particular, ''Nyström methods'' work directly with second-order equations.
* ''[[geometric integrator|geometric integration methods]]'' are especially designed for special classes of ODEs (e.g., [[symplectic integrator]]s for the solution of [[Hamiltonian mechanics|Hamiltonian equations]]). They take care that the numerical solution respects the underlying structure or geometry of these classes.
== Analysis ==
[[Numerical analysis]] is not only the design of numerical methods, but also their analysis. Three central concepts in this analysis are:
* ''convergence'': whether the method approximates the solution,
* ''order'': how well it approximates the solution, and
* [[numerical stability|''stability'']]: whether errors are damped out.
=== Convergence ===
A numerical method is said to be ''convergent'' if the numerical solution approaches the exact solution as the step size ''h'' goes to 0. More precisely, we require that for every ODE (1) with a [[Lipschitz continuous|Lipschitz]] function ''f'' and every ''t''<sup>*</sup> > 0,
:<math> \lim_{h\to0+} \max_{n=0,1,\dots,\lfloor t^*/h\rfloor} \| y_{n,h} - y(t_n) \| = 0. </math>
All the methods mentioned above are convergent. In fact, convergence is a condition ''[[sine qua non]]'' for any numerical scheme.
=== Consistency and order ===
Suppose the numerical method is
:<math> y_{n+k} = \Psi(t_{n+k}; y_n, y_{n+1}, \dots, y_{n+k-1}; h). \, </math>
The ''local error'' of the method is the error committed by one step of the method. That is, it is the difference between the result given by the method, assuming that no error was made in earlier steps, and the exact solution:
:<math> \delta^h_{n+k} = \Psi \left( t_{n+k}; y(t_n), y(t_{n+1}), \dots, y(t_{n+k-1}); h \right) - y(t_{n+k}). </math>
The method is said to be ''consistent'' if
:<math> \lim_{h\to 0} \frac{\delta^h_{n+k}}{h} = 0. </math>
The method has ''order'' <math>p</math> if
:<math> \delta^h_{n+k} = O(h^{p+1}) \quad\mbox{as } h\to0. </math>
Hence a method is consistent if it has an order greater than 0. The (forward) Euler method (4) and the backward Euler method (6) introduced above both have order 1, so they are consistent. Most methods being used in practice attain higher order. Consistency is a necessary condition for convergence, but not sufficient; for a method to be convergent, it must be both consistent and [[zero-stable]].
A related concept is the ''global error'', the error sustained in all the steps one needs to reach a fixed time ''t''. Explicitly, the global error at time ''t'' is ''y''<sub>''N''</sub> − ''y''(''t'') where ''N'' = (''t''−''t''<sub>0</sub>)/''h''. The global error of a ''p''th order one-step method is O(''h''<sup>''p''</sup>); in particular, such a method is convergent. This statement is not necessarily true for multi-step methods.
=== Stability and stiffness ===
: ''Main article: [[Stiff equation]]''
For some differential equations, application of standard methods—such as the Euler method, explicit [[Runge–Kutta methods]], or [[multistep method]]s (e.g., Adams–Bashforth methods)—exhibit instability in the solutions, though other methods may produce stable solutions. This "difficult behaviour" in the equation (which may not necessarily be complex itself) is described as ''stiffness'', and is often caused by the presence of different time scales in the underlying problem. Stiff problems are ubiquitous in [[chemical kinetics]], [[control theory]], [[solid mechanics]], [[weather|weather prediction]], [[biology]], and [[electronics]].
== History ==
Below is a [[Chronology|timeline]] of some important developments in this field.
* [[1768]] - [[Leonhard Euler]] publishes his method.
* [[1824]] - [[Augustin Louis Cauchy]] proves convergence of the Euler method. In this proof, Cauchy uses the implicit Euler method.
* [[1855]] - First mention of the [[multistep method]]s of [[John Couch Adams]] in a letter written by [[F. Bashforth]].
* [[1895]] - [[Carl David Tolmé Runge|Carl Runge]] publishes the first [[Runge–Kutta method]].
* [[1905]] - [[Martin Wilhelm Kutta|Martin Kutta]] describes the popular fourth-order [[Runge–Kutta method]].
* [[1910]] - [[Lewis Fry Richardson]] announces his [[extrapolation method]], [[Richardson extrapolation]].
* [[1952]] - [[Charles F. Curtiss]] and [[Joseph Oakland Hirschfelder]] coin the term ''[[stiff equation]]s''.
== See also ==
* [[Trapezoidal rule]]
* [[Courant–Friedrichs–Lewy condition]]
== References ==
*J. C. Butcher, ''Numerical methods for ordinary differential equations'', ISBN 0471967580
*Ernst Hairer, Syvert Paul Nørsett and Gerhard Wanner, ''Solving ordinary differential equations I: Nonstiff problems,'' second edition, Springer Verlag, Berlin, 1993. ISBN 3-540-56670-8.
*Ernst Hairer and Gerhard Wanner, ''Solving ordinary differential equations II: Stiff and differential-algebraic problems,'' second edition, Springer Verlag, Berlin, 1996. ISBN 3-540-60452-9. <br> ''(This two-volume monograph systematically covers all aspects of the field.)''
*Arieh Iserles, ''A First Course in the Numerical Analysis of Differential Equations,'' Cambridge University Press, 1996. ISBN 0-521-55376-8 (hardback), ISBN 0-521-55655-4 (paperback). <br> ''(Textbook, targeting advanced undergraduate and postgraduate students in mathematics, which also discusses [[numerical partial differential equations]].)''
*John Denholm Lambert, ''Numerical Methods for Ordinary Differential Systems,'' John Wiley & Sons, Chichester, 1991. ISBN 0-471-92990-5. <br> ''(Textbook, slightly more demanding than the book by Iserles.)''
==External links ==
* Joseph W. Rudmin, ''[http://csma31.csm.jmu.edu/physics/rudmin/ps.pdf Application of the Parker–Sochacki Method to Celestial Mechanics]'', 1998.
* Dominique Tournès, ''[http://www.rehseis.cnrs.fr/calculsavant/Equipe/dominiquetournes.html L'intégration approchée des équations différentielles ordinaires (1671-1914)]'', thèse de doctorat de l'université Paris 7 - Denis Diderot, juin 1996. Réimp. Villeneuve d'Ascq : Presses universitaires du Septentrion, 1997, 468 p. (Extensive online material on ODE numerical analysis history, for English-language material on the history of ODE numerical analysis, see eg the paper books by Chabert and Goldstine quoted by him.)
{{Numerical integrators}}
[[Category:Numerical differential equations| ]]
[[Category:Ordinary differential equations]]
[[cs:Numerické řešení obyčejných diferenciálních rovnic]]
[[fa:حل عددی معادلات دیفرانسیل معمولی]]
[[fr:Résolution numérique des équations différentielles]]