Convergence of random variables 50723 215690961 2008-05-29T08:08:16Z Zvika 413489 /* See also */ In [[probability theory]], there exist several different notions of [[convergence]] of [[random variable]]s. The convergence (in one of the senses presented below) of [[sequence]]s of random variables to some [[Limit (mathematics)|limit]]ing random variable is an important concept in probability theory, and its applications to [[statistics]] and [[stochastic process]]es. For example, if the average of ''n'' [[uncorrelated]] random variables ''Y''<sub>''i''</sub>, ''i'' = 1, ..., ''n'', is given by :<math>X_n = \frac{1}{n}\sum_{i=1}^n Y_i\,,</math> then as ''n'' goes to infinity, ''X''<sub>''n''</sub> converges ''in probability'' (see below) to the common [[mean]], &mu;, of the random variables ''Y''<sub>''i''</sub>. This result is known as the [[weak law of large numbers]]. Other forms of convergence are important in other useful theorems, including the [[central limit theorem]]. Throughout the following, we assume that (''X''<sub>''n''</sub>) is a sequence of random variables, and ''X'' is a random variable, and all of them are defined on the same [[probability space]] (&Omega;, ''F'', P). == Convergence in distribution == Suppose that ''F''<sub>1</sub>, ''F''<sub>2</sub>, ... is a [[sequence]] of [[cumulative distribution function]]s corresponding to random variables ''X''<sub>1</sub>, ''X''<sub>2</sub>, ..., and that ''F'' is a distribution function corresponding to a random variable ''X''. We say that the sequence ''X''<sub>''n''</sub> converges towards ''X'' '''in distribution''', if ::<math>\lim_{n\rightarrow\infty} F_n(a) = F(a),</math> for every [[real number]] ''a'' at which ''F'' is [[continuous function|continuous]]. Since ''F''(a) = Pr(''X'' &le; a), this means that the probability that the value of ''X'' is in a given range is very similar to the probability that the value of ''X''<sub>''n''</sub> is in that range, provided ''n'' is [[sufficiently large]]. Convergence in distribution is often denoted by adding the letter <math>\mathcal D</math> over an arrow indicating convergence: ::<math>X_n \, \xrightarrow{\mathcal D} \, X</math> Small <math>d</math> is also possible, although less common. Convergence in distribution is the weakest form of convergence, and is sometimes called '''weak convergence''' (main article: [[weak convergence of measures]]). It does not, in general, imply any other mode of convergence. However, convergence in distribution ''is'' implied by all other modes of convergence mentioned in this article, and hence, it is the most common and often the most useful form of convergence of random variables. It is the notion of convergence used in the [[central limit theorem]]. A useful result, which may be employed in conjunction with law of large numbers and the [[central limit theorem]], is that if a function &nbsp;''g'': '''R''' &rarr; '''R'''&nbsp; is continuous, then if &nbsp;''X''<sub>''n''</sub>&nbsp; converges in distribution to &nbsp;''X'', then so too does &nbsp;''g''(''X''<sub>''n''</sub>)&nbsp; converge in distribution to &nbsp;''g''(''X''). (This may be proved using [[Skorokhod's representation theorem]].) This fact could be taken as a definition for the convergence in distribution. Convergence in distribution is also called '''convergence in law''', since the word "law" is sometimes used as a synonym of "probability distribution." == Convergence in probability == To say that the sequence ''X''<sub>''n''</sub> converges towards ''X'' '''in probability''' means :<math>\lim_{n\rightarrow\infty}\Pr\left(\left|X_n-X\right|\geq\varepsilon\right)=0</math> for every &epsilon; > 0. Formally, pick any &epsilon; > 0 and any &delta; > 0. Let ''P''<sub>''n''</sub> be the probability that ''X''<sub>''n''</sub> is outside a tolerance &epsilon; of ''X''. Then, if ''X''<sub>''n''</sub> converges in probability to ''X'' then there exists a value ''N'' such that, for all ''n'' &ge; ''N'', ''P''<sub>''n''</sub> is itself less than &delta;. Convergence in probability is often denoted by adding the letter 'P' over an arrow indicating convergence: ::<math>X_n \, \xrightarrow{P} \, X</math> It is also denoted <math>\mbox{p}\!\!\!\lim_{n\rightarrow\infty\;} X_n = X \,</math> or simply <math>\operatorname{plim}\; X_n = X. \, </math> Convergence in probability is the notion of convergence used in the [[weak law of large numbers]]. Convergence in probability implies convergence in distribution. <sup>[[Convergence_of_random_variables/Proof_that_convergence_in_probability_implies_convergence_in_distribution |proof]]</sup> In the special case when <math>X=c</math> is a constant (that is, a random variable that attains the deterministic value <math>c</math> with probability one), convergence of <math>X_n</math> to <math>c</math> in probability is equivalent to convergence to <math>c</math> in distribution. == Almost sure convergence == To say that the sequence ''X''<sub>''n''</sub> converges '''almost surely''' or '''almost everywhere''' or '''with probability 1''' or '''strongly''' towards ''X'' means :<math>\Pr\left(\lim_{n\rightarrow\infty}X_n=X\right)=1.</math> This means that the values of ''X''<sub>''n''</sub> approach the value of ''X'', in the sense (see [[almost surely]]) that events for which ''X''<sub>''n''</sub> does not converge to ''X'' have probability 0. Using the probability space (&Omega;, ''F'', P) and the concept of the random variable as a function from &Omega; to '''R''', this is equivalent to the statement :<math>\Pr\left(\big\{\omega \in \Omega \, | \, \lim_{n \to \infty}X_n(\omega) = X(\omega) \big\}\right) = 1.</math> Almost sure convergence is often denoted by adding the letters ''a.s.'' over an arrow indicating convergence: ::<math>X_n \, \xrightarrow{\mathrm{a.s.}} \, X</math> Almost sure convergence implies convergence in probability, and hence implies convergence in distribution. It is the notion of convergence used in the strong [[law of large numbers]]. == Sure convergence == To say that the sequence or [[random variables]] (''X''<sub>''n''</sub>) defined over the same [[probability space]] (i.e., a [[random process]]) converges '''surely''' or '''everywhere''' or '''pointwise''' towards ''X'' means :<math>\lim_{n\rightarrow\infty}X_n(\omega)=X(\omega), \, \, \forall \omega \in \Omega.</math> where <math>\Omega</math> is the [[sample space]] of the underlying [[probability space]] over which the random variables are defined. This is the notion of [[pointwise convergence]] of sequence functions extended to sequence of [[random variables]]. (Note that random variables themselves are functions). :<math>\big\{\omega \in \Omega \, | \, \lim_{n \to \infty}X_n(\omega) = X(\omega) \big\} = \Omega.</math> Sure convergence of a random variable implies all the other kinds of convergence stated above, but there is no payoff in [[probability theory]] by using sure convergence compared to using almost sure convergence. The difference between the two only exists on sets with probability zero. This is why the concept sure convergence of random variables is very rarely used. == Convergence in mean == We say that the sequence ''X''<sub>''n''</sub> converges '''in the ''r''-th mean''' or '''in the''' ''[[Lp space|L<sup>r</sup>'' '''norm]]''' towards ''X'', if ''r'' &ge; 1, E|''X''<sub>''n''</sub>|<sup>''r''</sup> < &infin; for all ''n'', and ::<math>\lim_{n\rightarrow\infty}\mathrm{E}\left(\left|X_n-X\right|^r\right)=0</math> where the operator E denotes the [[expected value]]. Convergence in ''r''th mean tells us that the expectation of the ''r''-th power of the difference between ''X''<sub>''n''</sub> and ''X'' converges to zero. This type of convergence is often denoted by adding the letter ''L<sup>r</sup>'' over an arrow indicating convergence: ::<math>X_n \, \xrightarrow{L^r} \, X.</math> The most important cases of convergence in ''r''-th mean are: * When ''X''<sub>''n''</sub> converges in ''r''-th mean to ''X'' for ''r'' = 1, we say that ''X''<sub>''n''</sub> converges '''in mean''' to ''X''. * When ''X''<sub>''n''</sub> converges in ''r''-th mean to ''X'' for ''r'' = 2, we say that ''X''<sub>''n''</sub> converges '''in mean square''' to ''X''. This is also sometimes referred to as convergence '''in mean''', and is sometimes denoted<ref>{{cite book | last = Porat | first = B. | title = Digital Processing of Random Signals: Theory & Methods | date = 1994 | publisher = Prentice Hall | isbn = 0130637513 | pages=19 }}</ref> <math>\mathrm{l.i.m.}_{n \rightarrow \infty} X_n = X.\,\!</math> Convergence in the ''r''-th mean, for ''r'' &gt; 0, implies convergence in probability (by [[Markov's inequality]]), while if ''r'' > ''s'' &ge; 1, convergence in ''r''-th mean implies convergence in ''s''-th mean. Hence, convergence in mean square implies convergence in mean. == Implications == The chain of implications between the various notions of convergence are noted in their respective sections. They are, using the arrow notation :*<math> \xrightarrow{\textrm{a.s.}} \quad \Rightarrow \quad \xrightarrow{P} \quad \Rightarrow \quad \xrightarrow{\mathcal D} </math> :*<math> \forall r>0:\quad\xrightarrow{L^r} \quad \Rightarrow \quad \xrightarrow{P} </math> :*<math>\forall r>s\geq1:\quad\xrightarrow{L^r} \quad \Rightarrow \quad \xrightarrow{L^s}</math> No other implications other than these hold in general, but a number of special cases do permit the converse implications: * If ''X''<sub>''n''</sub> converges in distribution to a constant ''c'', then ''X''<sub>''n''</sub> converges in probability to ''c''. * If ''X''<sub>''n''</sub> converges in probability to ''X'', and if Pr(|''X''<sub>''n''</sub>| &le; ''b'') = 1 for all ''n'' and some ''b'', then ''X''<sub>''n''</sub> converges in ''r''th mean to ''X'' for all ''r'' &ge; 1. In other words, if ''X''<sub>''n''</sub> converges in probability to ''X'' and all random variables ''X''<sub>''n''</sub> are almost surely bounded above and below, then ''X''<sub>''n''</sub> converges to ''X'' also in any ''r''th mean. * If for all &epsilon; > 0, ::<math>\sum_n P\left(|X_n - X| > \varepsilon\right) < \infty,</math> :then we say that ''X''<sub>''n''</sub> ''converges almost completely'', or ''fast in probability'' towards ''X''. When ''X''<sub>''n''</sub> converges almost completely towards ''X'' then it also converges almost surely to ''X''. In other words, if ''X''<sub>''n''</sub> converges in probability to ''X'' sufficiently quickly (''i''.''e''. the above sequence of tail probabilities is summable for all &epsilon; > 0), then ''X''<sub>''n''</sub> also converges almost surely to ''X''. This is a direct implication from the [[Borel-Cantelli lemma]]. * If ''S''<sub>''n''</sub> is a sum of ''n'' real independent random variables: ::<math>S_n = X_1+\cdots+X_n</math> :then ''S''<sub>''n''</sub> converges almost surely if and only if ''S''<sub>''n''</sub> converges in probability. * [[Lévy's convergence theorem]] gives sufficient conditions for almost sure convergence to imply ''L''<sup>1</sup>-convergence: ::<math> \left. \begin{array}{ccc} X_n\xrightarrow{a.s.} X \\ \\ |X_n| < Y \\ \\ \mathrm{E}(Y) < \infty \end{array}\right\} \quad\Rightarrow \quad X_n\xrightarrow{L^1} X </math> ==See also== * [[Continuous stochastic process]]: the question of continuity of a [[stochastic process]] is essentially a question of convergence, and many of the same concepts and relationships used above apply to the continuity question. * [[Slutsky's theorem]] == Notes == {{reflist}} == External links == * [[b:Econometric Theory/Asymptotic Convergence|Convergence Wikibook]] == References == * Billingsley, Patrick (1999). ''Convergence of Probability Measures''. John Wiley & Sons, 2nd edition, pp. 1-28 ISBN 0471197459 * G.R. Grimmett and D.R. Stirzaker (1992). ''Probability and Random Processes, 2nd Edition''. Clarendon Press, Oxford, pp 271--285. ISBN 0-19-853665-8. * M. Jacobsen (1992). ''Videregående Sandsynlighedsregning (Advanced Probability Theory) 3rd Edition''. HCØ-tryk, Copenhagen, pp 18--20. ISBN 87-91180-71-6. [[Category:probability theory]] [[de:Konvergenz (Stochastik)]] [[fr:Convergence de variables aléatoires]] [[it:Convergenza di variabili casuali]] [[he:התכנסות (סטטיסטיקה)]] [[pl:Zbieżność według rozkładu]] [[vi:Sự hội tụ của các biến ngẫu nhiên]]