Cook's distance

Data points with large residuals (outliers) and/or high leverage may distort the outcome and accuracy of a regression. Cook's distance measures the effect of deleting a given observation. Points with a large Cook's distance are considered to merit closer examination in the analysis.

For the algebraic expression, first define

${\displaystyle {\underset {n\times 1}{\mathbf {y} }}={\underset {n\times p}{\mathbf {X} }}\quad {\underset {p\times 1}{\boldsymbol {\beta }}}\quad +\quad {\underset {n\times 1}{\boldsymbol {\varepsilon }}}}$

where ${\displaystyle {\boldsymbol {\varepsilon }}\sim {\mathcal {N}}\left(0,\sigma ^{2}\mathbf {I} \right)}$ is the error term, ${\displaystyle {\boldsymbol {\beta }}=\left[\beta _{0}\,\beta _{1}\dots \beta _{p-1}\right]^{\mathsf {T}}}$ is the coefficient matrix, ${\displaystyle p}$ is the number of covariates or predictors for each observation, and ${\displaystyle \mathbf {X} }$ is the design matrix including a constant. The least squares estimator then is ${\displaystyle \mathbf {b} =\left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} }$, and consequently the fitted (predicted) values for the mean of ${\displaystyle \mathbf {y} }$ are

${\displaystyle \mathbf {\widehat {y}} =\mathbf {X} \mathbf {b} =\mathbf {X} \left(\mathbf {X} ^{\mathsf {T}}\mathbf {X} \right)^{-1}\mathbf {X} ^{\mathsf {T}}\mathbf {y} =\mathbf {H} \mathbf {y} }$

where ${\displaystyle \mathbf {H} \equiv \mathbf {X} (\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {X} ^{\mathsf {T}}}$ is the projection matrix (or hat matrix). The ${\displaystyle i}$-th diagonal element of ${\displaystyle \mathbf {H} \,}$, given by ${\displaystyle h_{ii}\equiv \mathbf {x} _{i}^{\mathsf {T}}(\mathbf {X} ^{\mathsf {T}}\mathbf {X} )^{-1}\mathbf {x} _{i}}$,[4] is known as the leverage of the ${\displaystyle i}$-th observation. Similarly, the ${\displaystyle i}$-th element of the residual vector ${\displaystyle \mathbf {e} =\mathbf {y} -\mathbf {\widehat {y\,}} =\left(\mathbf {I} -\mathbf {H} \right)\mathbf {y} }$ is denoted by ${\displaystyle e_{i}}$.

Cook's distance ${\displaystyle D_{i}}$ of observation ${\displaystyle i\;({\text{for }}i=1,\dots ,n)}$ is defined as the sum of all the changes in the regression model when observation ${\displaystyle i}$ is removed from it[5]

${\displaystyle D_{i}={\frac {\sum _{j=1}^{n}\left({\widehat {y\,}}_{j}-{\widehat {y\,}}_{j(i)}\right)^{2}}{ps^{2}}}}$

where ${\displaystyle {\widehat {y\,}}_{j(i)}}$ is the fitted response value obtained when excluding ${\displaystyle i}$, and ${\displaystyle s^{2}={\frac {\mathbf {e} ^{\top }\mathbf {e} }{n-p}}}$ is the mean squared error of the regression model.[6]

Equivalently, it can be expressed using the leverage[5] (${\displaystyle h_{ii}}$):

${\displaystyle D_{i}={\frac {e_{i}^{2}}{ps^{2}}}\left[{\frac {h_{ii}}{(1-h_{ii})^{2}}}\right].}$

There are different opinions regarding what cut-off values to use for spotting highly influential points. Since Cook's distance is in the metric of an F distribution with ${\displaystyle p}$ and ${\displaystyle n-p}$ (as defined for the design matrix ${\displaystyle \mathbf {X} }$ above) degrees of freedom, the median point (i.e., ${\displaystyle F_{0.5}(p,n-p)}$) can be used as a cut-off.[7] Since this value is close to 1 for large ${\displaystyle n}$, a simple operational guideline of ${\displaystyle D_{i}>1}$ has been suggested.[8] Note that the Cook's distance measure does not always correctly identify influential observations.[9]

${\displaystyle D_{i}}$ can be expressed using the Leverage[5] (${\displaystyle 0\leq h_{ii}\leq 1}$) and the square of the internally Studentized residual (${\displaystyle 0\leq t_{i}^{2}}$), as follows:

${\displaystyle {\begin{aligned}D_{i}&={\frac {e_{i}^{2}}{ps^{2}}}\left[{\frac {h_{ii}}{(1-h_{ii})^{2}}}\right]={\frac {1}{p}}{\frac {e_{i}^{2}}{{1 \over n-p}\sum _{j=1}^{n}{\widehat {\varepsilon \,}}_{j}^{\,2}(1-h_{ii})}}\left[{\frac {h_{ii}}{(1-h_{ii})}}\right]\\&=\left[{\frac {1}{p}}\right]t_{i}^{2}{\frac {h_{ii}}{1-h_{ii}}}.\end{aligned}}}$

The benefit in the last formulation is that it clearly shows the relationship between ${\displaystyle t_{i}^{2}}$ and ${\displaystyle h_{ii}}$ to ${\displaystyle D_{i}}$ (while p and n are the same for all observations). If ${\displaystyle t_{i}^{2}}$ is large then it (for non-extreme values of ${\displaystyle h_{ii}}$) will increase ${\displaystyle D_{i}}$. If ${\displaystyle h_{ii}}$ is close to 0 than ${\displaystyle D_{i}}$ will be small, while if ${\displaystyle h_{ii}}$ is close to 1 then ${\displaystyle D_{i}}$ will become very large (as long as ${\displaystyle t_{i}^{2}>0}$, i.e.: that the observation ${\displaystyle i}$ is not exactly on the regression line that was fitted without observation ${\displaystyle i}$).

${\displaystyle D_{i}}$ is related to DFFITS through the following relationship (note that ${\displaystyle {{\widehat {\sigma }} \over {\widehat {\sigma }}_{(i)}}t_{i}=t_{i(i)}}$ is the externally studentized residual, and ${\displaystyle {\widehat {\sigma }},{\widehat {\sigma }}_{(i)}}$ are defined here):

${\displaystyle {\begin{aligned}D_{i}&=\left[{\frac {1}{p}}\right]t_{i}^{2}{\frac {h_{ii}}{1-h_{ii}}}\\&=\left[{\frac {1}{p}}\right]{{\widehat {\sigma }}_{(i)}^{2} \over {\widehat {\sigma }}^{2}}{{\widehat {\sigma }}^{2} \over {\widehat {\sigma }}_{(i)}^{2}}t_{i}^{2}{\frac {h_{ii}}{1-h_{ii}}}=\left[{\frac {1}{p}}\right]{{\widehat {\sigma }}_{(i)}^{2} \over {\widehat {\sigma }}^{2}}\left(t_{i(i)}{\sqrt {\frac {h_{ii}}{1-h_{ii}}}}\right)^{2}\\&=\left[{\frac {1}{p}}\right]{{\widehat {\sigma }}_{(i)}^{2} \over {\widehat {\sigma }}^{2}}{\text{DFFITS}}^{2}\end{aligned}}}$

${\displaystyle D_{i}}$ can be interpreted as the distance one's estimates move within the confidence ellipsoid that represents a region of plausible values for the parameters. This is shown by an alternative but equivalent representation of Cook's distance in terms of changes to the estimates of the regression parameters between the cases, where the particular observation is either included or excluded from the regression analysis.

Many programs and statistics packages, such as R, Python, etc., include implementations of Cook's distance.

High dimensional Influence Measure (HIM), is an alternative to Cook's distance for when ${\displaystyle p>n}$ (i.e.: more predictors than observations) [10]. While the Cook’s distance quantifies the individual observation’s influence on the least squares regression coefficient estimate, the HIM measures the influence of an observation on the marginal correlations.

• Outlier
• Leverage (statistics)
• Partial leverage
• DFFITS
• Studentized residual

• Atkinson, Anthony; Riani, Marco (2000). "Deletion Diagnostics". Robust Diagnostics and Regression Analysis. New York: Springer. pp. 22–25. ISBN 0-387-95017-6.
• Heiberger, Richard M.; Holland, Burt (2013). "Case Statistics". Statistical Analysis and Data Display. Springer Science & Business Media. pp. 312–27. ISBN 9781475742848.
• Krasker, William S.; Kuh, Edwin; Welsch, Roy E. (1983). "Estimation for dirty data and flawed models". Handbook of Econometrics. 1. Elsevier. pp. 651–698. doi:10.1016/S1573-4412(83)01015-6.
• Aguinis, Herman; Gottfredson, Ryan K.; Joo, Harry (2013). "Best-Practice Recommendations for Defining Identifying and Handling Outliers" (PDF). Organizational Research Methods. Sage. 16 (2): 270–301. doi:10.1177/1094428112470848. Retrieved 4 December 2015.