About Opentopia
Opentopia Directory Encyclopedia Tools

Linear regression

Encyclopedia : L : LI : LIN : Linear regression

In statistics, linear regression is a method of estimating the conditional expected value of one variable y given the values of some other variable or variables x. The variable of interest, y, is conventionally called the "response variable". The terms "endogenous variable" and "output variable" are also used. The other variables x are called the explanatory variables. The terms "exogenous variables" and "input variables" are also used, along with "predictor variables". The term independent variables is sometimes used, but should be avoided as the variables are not necessarily statistically independent. The explanatory and response variables may be scalars or vectors. Multiple regression includes cases with more than one explanatory variable.

The term explanatory variable suggests that its value can be chosen at will, and the response variable is an effect, i.e., causally dependent on the explanatory variable, as in a stimulus-response model. Although many linear regression models are formulated as models of cause and effect, the direction of causation may just as well go the other way, or indeed there need not be any causal relation at all. For that reason, one may prefer the terms "predictor / response" or "endogenous / exogenous," which do not imply causality.

Regression, in general, is the problem of estimating a conditional expected value.

It is often erroneously thought that the reason the technique is called "linear regression" is that the graph of [y = \alpha + \beta x] is a line. But in fact, if the model is

[y_i = \alpha + \beta x_i + \gamma x_i^2 + \epsilon_i]
(in which case we have put the vector [(x_i, x_i^2)] in the role formerly played by [x_i] and the vector [(\beta, \gamma)] in the role formerly played by [\beta]), then the problem is still one of linear regression, even though the graph is not a straight line. The rationale for this terminology will be explained below.

Linear regression is called "linear" because the relation of the response to the explanatory variables is assumed to be a linear function of some parameters. Regression models which are not a linear function of the parameters are called nonlinear regression models. A neural network is an example of a nonlinear regression model.

Still more generally, regression may be viewed as a special case of density estimation. The joint distribution of the response and explanatory variables can be constructed from the conditional distribution of the response variable and the marginal distribution of the explanatory variables. In some problems, it is convenient to work in the other direction: from the joint distribution, the conditional distribution of the response variable can be derived.

Contents

Historical remarks

The earliest form of linear regression was the method of least squares, which was published by Legendre in 1805, and by Gauss in 1809. The term "least squares" is from Legendre's term, moindres carrés. However, Gauss claimed that he had known the method since 1795.

Legendre and Gauss both applied the method to the problem of determining, from astronomical observations, the orbits of bodies about the sun. Euler had worked on the same problem (1748) without success. Gauss published a further development of the theory of least squares in 1821, including a version of the Gauss-Markov theorem.

The term "reversion" was used in the nineteenth century to describe a biological phenomenon, namely that the progeny of exceptional individuals tend on average to be less exceptional than their parents, and more like their more distant ancestors. Francis Galton studied this phenomenon, and applied the slightly misleading term "regression towards mediocrity" to it (parents of exceptional individuals also tend on average to be less exceptional than their children). For Galton, regression had only this biological meaning, but his work (1877, 1885) was extended by Karl Pearson and Udny Yule to a more general statistical context (1897, 1903). In the work of Pearson and Yule, the joint distribution of the response and explanatory variables is assumed to be Gaussian. This assumption was weakened by R.A. Fisher in his works of 1922 and 1925. Fisher assumed that the conditional distribution of the response variable is Gaussian, but the joint distribution need not be. In this respect, Fisher's assumption is closer to Gauss's formulation of 1821.

Statement of the linear regression model

A linear regression model is typically stated in the form

[ y = \alpha + \beta x + \varepsilon. \, ]
The right hand side may take other forms, but generally comprises a linear combination of the parameters, here denoted α and β. The term ε represents the unpredicted or unexplained variation in the response variable; it is conventionally called the "error" whether it is really a measurement error or not. The error term is conventionally assumed to have expected value equal to zero, as a nonzero expected value could be absorbed into α. See also errors and residuals in statistics; the difference between an error and a residual is also dealt with below. It is also assumed that [\varepsilon] is independent of [x].

An equivalent formulation which explicitly shows the linear regression as a model of conditional expectation is

[ \mbox(y|x) = \alpha + \beta x \, ]
with the conditional distribution of y given x essentially the same as the distribution of the error term.

A linear regression model need not be affine, let alone linear, in the explanatory variables x. For example,

[y = \alpha + \beta x + \gamma x^2 + \varepsilon]
is a linear regression model, for the right-hand side is a linear combination of the parameters α, β, and γ. In this case it is useful to think of x2 as a new explanatory variable, formed by modifying the original variable x. Indeed, any linear combination of functions f(x), g(x), h(x), ..., is linear regression model, so long as these functions do not have any free parameters (otherwise the model is generally a nonlinear regression model). The least-squares estimates of α, β, and γ are linear in the response variable y, and nonlinear in x (they are nonlinear in x even if the γ and α terms are absent; if only β were present then doubling all observed x values would multiply the least-squares estimate of β by 1/2).

Often in linear regression problems statisticians rely on the Gauss-Markov assumptions:

(See also Gauss-Markov theorem. That result says that under the assumptions above, least-squares estimators are in a certain sense optimal.)

Sometimes stronger assumptions are relied on:

If xi is a vector we can take the product βxi to be a scalar product (see "dot product").

A statistician will usually estimate the unobservable values of the parameters α and β by the method of least squares, which consists of finding the values of a and b that minimize the sum of squares of the residuals

[e_i = y_i - (a + bx_i). \,]
Those values of a and b are the "least-squares estimates." The residuals may be regarded as estimates of the errors; see also errors and residuals in statistics.

Notice that, whereas the errors are independent, the residuals cannot be independent because the use of least-squares estimates implies that the sum of the residuals must be 0, and the scalar product of the vector of residuals with the vector of x-values must be 0, i.e., we must have

[e_1 + \cdots + e_n = 0 \,]
and

[e_1 x_1 + \cdots + e_n x_n = 0. \,]
These two linear constraints imply that the vector of residuals must lie within a certain (n − 2)-dimensional subspace of Rn; hence we say that there are "n − 2 degrees of freedom for error". If one assumes the errors are normally distributed and independent, then it can be shown to follow that 1) the sum of squares of residuals

[e_1^2 + \cdots + e_n^2 \,]
is distributed as

[\sigma^2 \chi^2_, \,]
So we have : These facts make it possible to use Student's t-distribution with n − 2 degrees of freedom (so named in honor of the pseudonymous "Student") to find confidence intervals for α and β.

Parameter estimation

By recognizing that the [y_i = \alpha + \beta x_i + \varepsilon_i ] regression model is a system of linear equatioins we can express the model using data matrix X, target vector Y and parameter vector [\delta]. The ith row of X and Y will contain the x and y value for the tth data sample. Then the model can be written as

[ \begin y_1\\ y_2\\ \vdots\\ y_n \end= \begin 1 & x_1\\ 1 & x_2\\ \vdots & \vdots\\ 1 & x_n \end \begin \alpha \\ \beta \end + \begin \varepsilon_1\\ \varepsilon_2\\ \vdots\\ \varepsilon_n \end ]
which when using pure matrix notation becomes
[Y = X \delta + \varepsilon \,]
where ε is normally distributed with expected value 0 (i.e., a column vector of 0s) and variance σ2 In, where In is the n×n identity matrix. The matrix [X\widehat] (where (remember) [\widehat] is the vector of estimates of the components of δ) is then the orthogonal projection of Y onto the column space of X.

Then it can be shown that

[\widehat = (X' X)^\; X' Y \,]
(where X′ is the transpose of X) and the sum of squares of residuals is

[Y' (I_n - X (X' X)^ X')\, Y.]
The fact that the matrix X(X′X)−1X′ is a symmetric idempotent matrix is incessantly relied on both in computations and in proofs of theorems. The linearity of [\widehat] as a function of the vector Y, expressed above by saying

[\widehat = (X'X)^X'Y,\,]
is the reason why this is called "linear" regression. Nonlinear regression uses nonlinear methods of estimation.

The matrix In − X (X′ X)−1 X′ that appears above is a symmetric idempotent matrix of rank n − 2. Here is an example of the use of that fact in the theory of linear regression. The finite-dimensional spectral theorem of linear algebra says that any real symmetric matrix M can be diagonalized by an orthogonal matrix G, i.e., the matrix G′MG is a diagonal matrix. If the matrix M is also idempotent, then the diagonal entries in G′MG must be idempotent numbers. Only two real numbers are idempotent: 0 and 1. So In − X(X′X) 1X′, after diagonalization, has n − 2 0s and two 1s on the diagonal. That is most of the work in showing that the sum of squares of residuals has a chi-square distribution with n−2 degrees of freedom.

Regression parameters can also be estimated by Bayesian methods. This has the advantages that

Suppose that in the linear regression

[ y = \alpha + \beta x + \varepsilon \, ]
we know from domain knowledge that alpha can only take one of the values but we do not know which. We can build this information into the analysis by choosing a prior for alpha which is a discrete distribution with a probability of 0.5 on −1 and 0.5 on +1. The posterior for alpha will also be a discrete distribution on , but the probability weights will change to reflect the evidence from the data.

Robust regression

A useful alternative to linear regression is robust regression in which mean absolute error, or some other function of the residuals, is minimized instead of mean squared error as in linear regression. Robust regression is computationally much more intensive than linear regression and is somewhat more difficult to implement as well. However, least squares estimates are known to be very poor under certain circumstances, particularly when outliers are present. Since most real data are non-normal and often subject to outliers, robust estimation will frequently be preferable to least squares in practice.

Robust regression sometimes means linear regression with robust (Huber-White) standard errors (e.g. relaxing the assumption of homoskedasticity).

Summarizing the data

We sum the observations, the squares of the Ys and Xs and the products XY to obtain the following quantities.

[S_X = x_1 + x_2 + \cdots + x_n \,]
and [S_Y] similarly.
[S_ = x_1^2 + x_2^2 + \cdots + x_n^2 \,]
and SYY similarly.
[S_ = x_1 y_1 + x_2 y_2 + \cdots + x_n y_n. \,]

Estimating beta

We use the summary statistics above to calculate b, the estimate of β.

[b = - S_X S_Y \over n S_ - S_X S_X}. \,]

Estimating alpha

We use the estimate of β and the other statistics to estimate α by:

[a = . \,]
A consequence of this estimate is that the regression line will always pass through the "center" [(\bar,\bar) = (S_X/n, S_Y/n)].

Displaying the residuals

The first method of displaying the residuals use the histogram or cumulative distribution to depict the similarity (or lack thereof) to a normal distribution. Non-normality suggests that the model may not be a good summary description of the data.

We plot the residuals,

[e_i=y_i - a - bx_i \,]
against the explanatory variable, x. There should be no discernible trend or pattern if the model is satisfactory for these data. Some of the possible problems are:
Studentized residuals can be used in outlier detection.
  • The vertical spread of the residuals increases as the explanatory variable increases (funnel-shaped plot) —indicates that the homoskedasticity assumption is violated (i.e. there is heteroskedasticity: the variability of the response depends on the value of x.
  • *Transform the data: for example, the logarithm or logit transformations are often useful.
  • *Use a more general modeling approach that can account for non-constant variance, for example a general linear model or a generalized linear model.

    • Ancillary statistics

    The sum of squared deviations can be partitioned as in ANOVA to indicate what part of the dispersion of the response variable is explained by the explanatory variable.

    The correlation coefficient, r, can be calculated by

    [r = - S_X S_Y \over \sqrt - S_X^2) (n S_ - S_Y^2)}}. \,]
    This statistic is a measure of how well a straight line describes the data. Values near zero suggest that the model is ineffective. r2 is frequently interpreted as the fraction of the variability explained by the explanatory variable, X.

    Multiple linear regression

    Linear regression can be extended to functions of two or more variables, for example

    [Z = a X + b Y + c +\varepsilon. \,]
    Here X and Y are explanatory variables. The values of the parameters a, b and c are estimated by the method of least squares, that minimize the sum of squares of the residuals

    [S_r = \sum_^n (a x_i + b y_i + c - z_i)^2. \,]
    Take derivatives with respect to a, b and c in [S_r] and set equal to zero. This leads to a system of three linear equations i.e. the normal equations to find estimates of parameters a, b and c:

    [ \begin \sum_^n x_i^2 & \sum_^n x_i y_i & \sum_^n x_i \\ \\ \sum_^n x_i y_i & \sum_^n y_i^2 & \sum_^n y_i \\ \\ \sum_^n x_i & \sum_^n y_i & n \\ \end \begin a \\ \\ b \\ \\ c \end= \begin \sum_^n x_i z_i \\ \\ \sum_^n y_i z_i \\ \\ \sum_^n z_i \end]

    Scientific applications of regression

    Linear regression is widely used in biological and behavioural sciences to describe relationships between variables. It ranks as one of the most important tools used in these disciplines. For example, early evidence relating cigarette smoking to mortality came from studies employing regression. Researchers usually include several variables in their regression analysis in an effort to remove factors that might produce spurious correlations. For the cigarette smoking example, researchers might include socio-economic status in addition to smoking to ensure that any observed effect of smoking on mortality is not due to some effect of education or income. However, it is never possible to include all possible confounding variables in a study employing regression. For the smoking example, a hypothetical gene might increase mortality and also cause people to smoke more. For this reason, randomized experiments are considered to be more trustworthy than a regression analysis.

    See also

    References

    Historical

    Modern theory

    Modern practice

    External links

     

    From Wikipedia, the Free Encyclopedia. Original article here. Support Wikipedia by contributing or donating.
    All text is available under the terms of the GNU Free Documentation License See Wikipedia Copyrights for details.

    Search Titles
    0123456789
    ABCDEFGHIJ
    KLMNOPQRST
    UVWXYZ?

    E-mail this article to:

    Personal Message: