Linear Regression is a simple and powerful model for predicting a numeric response from a set of one or more independent variables. This article will focus mostly on how the method is used in machine learning, so we won't cover common use cases like causal inference or experimental design. And although it may seem like linear regression is overlooked in modern machine learning's ever-increasing world of complex neural network architectures, the algorithm is still widely used across a large number of domains because it is effective, easy to interpret, and easy to extend. The key ideas in linear regression are recycled everywhere, so understanding the algorithm is a must-have for a strong foundation in machine learning.

Let's Be More Specific
Linear regression is a supervised algorithm^[ℹ] that learns to model a dependent variable, $y$, as a function of some independent variables (aka "features"), $x_i$, by finding a line (or surface) that best "fits" the data. In general, we assume $y$ to be some number and each $x_i$ can be basically anything. For example: predicting the price of a house using the number of rooms in that house ($y$: price, $x_1$: number of rooms) or predicting weight from height and age ($y$: weight, $x_1$: height, $x_2$: age).

In general, the equation for linear regression is

$$y=\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_px_p + \epsilon$$

where:

• $y$: the dependent variable; the thing we are trying to predict.^[ℹ]
• $x_i$: the independent variables: the features our model uses to model y.^[ℹ]
• $\beta_i$: the coefficients (aka "weights") of our regression model. These are the foundations of our model. They are what our model "learns" during optimization.^[ℹ]
• $\epsilon$: the residual (or "error") of our model. Our model will not make perfect predictions, so we compute this term by subtracting the predicted value from the actual value.

Fitting a linear regression model is all about finding the set of coefficients that best model $y$ as a function of our features. We may never know the true parameters for our model, but we can estimate them (more on this later). Once we've estimated these coefficients, $\hat{\beta_i}$, we predict future values, $\hat{y}$, as:

$$\hat{y}=\hat{\beta_0} + \hat{\beta_1}x_1 + \hat{\beta_2}x_2 + ... + \hat{\beta_p}x_p$$

So predicting future values (often called inference), is as simple as plugging the values of our features $x_i$ into our equation!

Model Evaluation

To train an accurate linear regression model, we need a way to quantify how good (or bad) our model performs. In machine learning, we call such performance-measuring functions loss functions. Several popular loss functions exist for regression problems.^[ℹ] To measure our model's performance, we'll use one of the most popular: mean-squared error (MSE).

Mean-Squared Error (MSE)
MSE quantifies how close a predicted value is to the true value, so we'll use it to quantify how close a regression line is to a set of points. MSE works by squaring the distance between each data point and the regression line (the red residuals in the graphs above), summing the squared values, and then dividing by the number of data points: $$\begin{aligned} MSE = \frac{1}{n} \Sigma^{n}_{i=1}(y_i - \hat{y_i})^2 \end{aligned}$$ The name is quite literal: take the mean of the squared errors. The squaring of errors prevents negative and positive terms from canceling out in the sum,^[ℹ] and gives more weight to points further from the regression line, punishing outliers. In practice, we'll fit our regression model to a set training data, and evaluate it's performance using MSE on the test dataset.

R-Squared
Regression models may also be evaluated with the so-called goodness of fit measures, which summarize how well a model fits a set of data. The most popular goodness of fit measure for linear regression is r-squared, a metric that represents the percentage of the variance in $y$ explained by our features $x$.^[ℹ] More specifically, r-squared measures the percentage of variance explained normalized against the baseline variance of our model (which is just the variance of the the trivial model that always predicts the mean): $$\begin{aligned} R^2 = 1 - \frac{\Sigma^{n}_{i=1}(y_i - \hat{y_i})^2 }{\Sigma^{n}_{i=1}(y_i - \bar{y})^2 } \end{aligned}$$ The highest possible value for r-squared is 1, representing a model that captures 100% of the variance. A negative r-squared means that our model is doing worse (capturing less variance) than a flat line through mean of our data would. (The name "r-squared" falsely implies that it would not have a negative value.)

To build intuition for yourself, try changing the weight and intercept terms below to see how the MSE and r-squared change across different possible models for a toy dataset (click Shuffle Data to make a new toy dataset):

You will often see R-Squared referenced in statistical contexts as a way to assess model fit.

Selecting An Evaluation Metric
Many methods exist for evaluating regression models, each with different concerns around interpretability, theory, and usability. The evaluation metric should reflect whatever it is you actually care about when making predictions. For example, when we use MSE, we are implicitly saying that we think the cost of our prediction error should reflect the quadratic (squared) distance between what we predicted and what is correct. This may work well if we want to punish outliers or if our data is minimized by the mean, but this comes at the cost of interpretability: we output our error in squared units (though this may be fixed with RMSE). If instead we wanted our error to reflect the linear distance between what we predicted and what is correct, or we wanted our data minimized by the median, we could try something like Mean Absolute Error (MAE). Whatever the case, you should be thinking of your evaluation metric as part of your modeling process, and select the best metric based on the specific concerns of your use-case.

A Closed-Form Solution
We'd be remiss not to mention the Normal Equation, a widely taught method for obtaining estimates for our linear regression coefficients. The Normal Equation is a closed-form solution that allows us to estimate our coefficients directly by minimizing the residual sum of squares (RSS) of our data: $$\begin{aligned} RSS = \sum^{n}_{i=1}(y_i - \hat{y_i})^2 \end{aligned}$$ The RSS should look familiar - it was a key piece in both the MSE and r-squared formulas that represents our model's total squared error: $$\begin{aligned} \hat{\beta} = (X^{T}X)^{-1}X^{T}Y \end{aligned}$$ Add circles to the chart below to see how the Normal Equation calculates two features, the intercept and weight, for the corresponding regression model.

Despite providing a convenient closed-form solution for finding our optimal coefficients, the Normal Equation estimates are often not used in practice, because of the computational complexity required to invert a matrix with too many features. While our two feature example above runs fast (we can run it in the browser!), most machine learning models are more complicated. For this reason, we often just use gradient descent.

Are Our Coefficients Valid?
In research publications and statistical software, coefficients of regression models are often presented with associated p-values. These p-values come from traditional null hypothesis statistical tests: t-tests are used to measure whether a given coefficient is significantly different than zero (the null hypothesis that a particular coefficient $\beta_i$ equals zero), while F tests are used to measure whether any of the terms in a regression model are significantly different from zero. Different opinions exist on the utility of such tests (e.g. chapter 10.7 of [1] maintains they're not super important). We don't take a strong stance on this issue, but believe practitioners should always assess the standard error around any parameter estimates for themselves and present them in their research.