# Derivation: Ordinary Least Squares Solution and Normal Equations

In a linear regression framework, we assume some output variable is a linear combination of some independent input variables plus some independent noise . The way the independent variables are combined is defined by a parameter vector :

We also assume that the noise term is drawn from a standard Normal distribution:

For some estimate of the model parameters , the model’s prediction errors/residuals are the difference between the model prediction and the observed ouput values

The Ordinary Least Squares (OLS) solution to the problem (i.e. determining an optimal solution for ) involves minimizing the sum of the squared errors with respect to the model parameters, . The sum of squared errors is equal to the inner product of the residuals vector with itself :

To determine the parameters, , we minimize the sum of squared residuals with respect to the parameters.

due to the identity , for vectors and . This relationship is matrix form of the Normal Equations. Solving for gives the analytical solution to the Ordinary Least Squares problem.

Boom.

Posted on September 1, 2012, in Derivations, Regression, Statistics, Theory, Uncategorized and tagged Derivation of OLS Solution, Mean Squared Error, Normal Equations, OLS. Bookmark the permalink. 3 Comments.

Reblogged this on Growth Hacks For Geeks and commented:

A very intuitive proof of normal equations in a linear regression framework

Pingback: fMRI in Neuroscience: Modeling the HRF With FIR Basis Functions « The Clever Machine

Pingback: Derivation: The Covariance Matrix of an OLS Estimator (and applications to GLS) « The Clever Machine