Projection (linear algebra)

• In the finite-dimensional case, a square matrix ${\displaystyle P}$ is called a projection matrix if it is equal to its square, i.e. if ${\displaystyle P^{2}=P}$.[2]^{:p. 38}
• A square matrix ${\displaystyle P}$ is called an orthogonal projection matrix if ${\displaystyle P^{2}=P=P^{\mathrm {T} }}$ for a real matrix, and respectively ${\displaystyle P^{2}=P=P^{\mathrm {H} }}$ for a complex matrix, where ${\displaystyle P^{\mathrm {T} }}$ denotes the transpose of ${\displaystyle P}$ and ${\displaystyle P^{\mathrm {H} }}$ denotes the Hermitian transpose of ${\displaystyle P}$.[2]^{:p. 223}
• A projection matrix that is not an orthogonal projection matrix is called an oblique projection matrix.

The eigenvalues of a projection matrix must be 0 or 1.

For example, the function which maps the point ${\displaystyle (x,y,z)}$ in three-dimensional space ${\displaystyle \mathbb {R} ^{3}}$ to the point ${\displaystyle (x,y,0)}$ is an orthogonal projection onto the x–y plane. This function is represented by the matrix

${\displaystyle P={\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}}.}$

The action of this matrix on an arbitrary vector is

${\displaystyle P{\begin{pmatrix}x\\y\\z\end{pmatrix}}={\begin{pmatrix}x\\y\\0\end{pmatrix}}.}$

To see that ${\displaystyle P}$ is indeed a projection, i.e., ${\displaystyle P=P^{2}}$, we compute

${\displaystyle P^{2}{\begin{pmatrix}x\\y\\z\end{pmatrix}}=P{\begin{pmatrix}x\\y\\0\end{pmatrix}}={\begin{pmatrix}x\\y\\0\end{pmatrix}}=P{\begin{pmatrix}x\\y\\z\end{pmatrix}}}$.

Observing that ${\displaystyle P^{\mathrm {T} }=P}$ shows that the projection is an orthogonal projection.

A simple example of a non-orthogonal (oblique) projection (for definition see below) is

${\displaystyle P={\begin{bmatrix}0&0\\\alpha &1\end{bmatrix}}.}$

Via matrix multiplication, one sees that

${\displaystyle P^{2}={\begin{bmatrix}0&0\\\alpha &1\end{bmatrix}}{\begin{bmatrix}0&0\\\alpha &1\end{bmatrix}}={\begin{bmatrix}0&0\\\alpha &1\end{bmatrix}}=P.}$

proving that ${\displaystyle P}$ is indeed a projection.

The projection ${\displaystyle P}$ is orthogonal if and only if ${\displaystyle \alpha =0}$ because only then ${\displaystyle P^{\mathrm {T} }=P}$.

By definition, a projection ${\displaystyle P}$ is idempotent (i.e. ${\displaystyle P^{2}=P}$).

Let ${\displaystyle W}$ be a finite dimensional vector space and ${\displaystyle P}$ be a projection on ${\displaystyle W}$. Suppose the subspaces ${\displaystyle U}$ and ${\displaystyle V}$ are the range and kernel of ${\displaystyle P}$ respectively. Then ${\displaystyle P}$ has the following properties:

1. ${\displaystyle P}$ is the identity operator ${\displaystyle I}$ on ${\displaystyle U}$

   ${\displaystyle \forall x\in U:Px=x}$.

2. We have a direct sum ${\displaystyle W=U\oplus V}$. Every vector ${\displaystyle x\in W}$ may be decomposed uniquely as ${\displaystyle x=u+v}$ with ${\displaystyle u=Px}$ and ${\displaystyle v=x-Px=(I-P)x}$, and where ${\displaystyle u\in U,v\in V}$.

The range and kernel of a projection are complementary, as are ${\displaystyle P}$ and ${\displaystyle Q=I-P}$. The operator ${\displaystyle Q}$ is also a projection as the range and kernel of ${\displaystyle P}$ become the kernel and range of ${\displaystyle Q}$ and vice versa. We say ${\displaystyle P}$ is a projection along ${\displaystyle V}$ onto ${\displaystyle U}$ (kernel/range) and ${\displaystyle Q}$ is a projection along ${\displaystyle U}$ onto ${\displaystyle V}$.

In infinite dimensional vector spaces, the spectrum of a projection is contained in ${\displaystyle \{0,1\}}$ as

${\displaystyle (\lambda I-P)^{-1}={\frac {1}{\lambda }}I+{\frac {1}{\lambda (\lambda -1)}}P.}$

Only 0 or 1 can be an eigenvalue of a projection, implying that ${\displaystyle P}$ is always a positive semi-definite matrix. The corresponding eigenspaces are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique in general. Therefore, given a subspace ${\displaystyle V}$, there may be many projections whose range (or kernel) is ${\displaystyle V}$.

If a projection is nontrivial it has minimal polynomial ${\displaystyle x^{2}-x=x(x-1)}$, which factors into distinct roots, and thus ${\displaystyle P}$ is diagonalizable.

The product of projections is not in general a projection, even if they are orthogonal. If two projections commute then their product is a projection, but the converse if false: the product of two non-commuting projections may be a projection .

If two orthogonal projections commute then their product is an orthogonal projection. If the product of two orthogonal projections is an orthogonal projection, then the two orthogonal projections commute (more generally: two self-adjoint endomorphisms commute if and only if their product is self-adjoint).

When the vector space ${\displaystyle W}$ has an inner product and is complete (is a Hilbert space) the concept of orthogonality can be used. An orthogonal projection is a projection for which the range ${\displaystyle U}$ and the null space ${\displaystyle V}$ are orthogonal subspaces. Thus, for every ${\displaystyle x}$ and ${\displaystyle y}$ in ${\displaystyle W}$, ${\displaystyle \langle Px,(y-Py)\rangle =\langle (x-Px),Py\rangle =0}$. Equivalently:

${\displaystyle \langle x,Py\rangle =\langle Px,Py\rangle =\langle Px,y\rangle }$.

A projection is orthogonal if and only if it is self-adjoint. Using the self-adjoint and idempotent properties of ${\displaystyle P}$, for any ${\displaystyle x}$ and ${\displaystyle y}$ in ${\displaystyle W}$ we have ${\displaystyle Px\in U}$, ${\displaystyle y-Py\in V}$, and

${\displaystyle \langle Px,y-Py\rangle =\langle P^{2}x,y-Py\rangle =\langle Px,P(I-P)y\rangle =\langle Px,(P-P^{2})y\rangle =0\,}$

where ${\displaystyle \langle \cdot ,\cdot \rangle }$ is the inner product associated with ${\displaystyle W}$. Therefore, ${\displaystyle Px}$ and ${\displaystyle y-Py}$ are orthogonal projections.[3] The other direction, namely that if ${\displaystyle P}$ is orthogonal then it is self-adjoint, follows from

${\displaystyle \langle x,Py\rangle =\langle Px,y\rangle =\langle x,P^{*}y\rangle }$

for every ${\displaystyle x}$ and ${\displaystyle y}$ in ${\displaystyle W}$; thus ${\displaystyle P=P^{*}}$.

Proof of existence

Let ${\displaystyle H}$ be a complete metric space with an inner product, and let ${\displaystyle U}$ be a closed linear subspace of ${\displaystyle H}$ (and hence complete as well). For every ${\displaystyle x}$ the following set of non-negative norms ${\displaystyle \{\|x-u\||u\in U\}}$ has an infimum, and due to the completeness of ${\displaystyle U}$ it is a minimum. We define ${\displaystyle Px}$ as the point in ${\displaystyle U}$ where this minimum is obtained. Obviously ${\displaystyle Px}$ is in ${\displaystyle U}$. It remains to show that ${\displaystyle Px}$ satisfies ${\displaystyle \langle x-Px,Px\rangle =0}$ and that it is linear. Let us define ${\displaystyle a=x-Px}$. For every non-zero ${\displaystyle v}$ in ${\displaystyle U}$, the following holds: ${\displaystyle \|a-{\frac {\langle a,v\rangle }{\|v\|^{2}}}v\|^{2}=\|a\|^{2}-{\frac {{\langle a,v\rangle }^{2}}{\|v\|^{2}}}}$ By defining ${\displaystyle w=Px+{\frac {\langle a,v\rangle }{\|v\|^{2}}}v}$ we see that ${\displaystyle \|x-w\|<\|x-Px\|}$ unless ${\displaystyle \langle a,v\rangle }$ vanishes. Since ${\displaystyle Px}$ was chosen as the minimum of the abovementioned set, it follows that ${\displaystyle \langle a,v\rangle }$ indeed vanishes. In particular, (for ${\displaystyle y=Px}$): ${\displaystyle \langle x-Px,Px\rangle =0}$. Linearity follows from the vanishing of ${\displaystyle \langle x-Px,v\rangle }$ for every ${\displaystyle v\in U}$: ${\displaystyle \langle \left(x+y\right)-P\left(x+y\right),v\rangle =0}$ ${\displaystyle \langle \left(x-Px\right)+\left(y-Py\right),v\rangle =0}$ By taking the difference between the equations we have ${\displaystyle \langle Px+Py-P\left(x+y\right),v\rangle =0}$ But since we may choose ${\displaystyle v=Px+Py-P(x+y)}$ (as it is itself in ${\displaystyle U}$) it follows that ${\displaystyle Px+Py=P(x+y)}$. Similarly we have ${\displaystyle \lambda Px=P(\lambda x)}$ for every scalar ${\displaystyle \lambda }$.

An orthogonal projection is a bounded operator. This is because for every ${\displaystyle v}$ in the vector space we have, by Cauchy–Schwarz inequality:

${\displaystyle \|Pv\|^{2}=\langle Pv,Pv\rangle =\langle Pv,v\rangle \leq \|Pv\|\cdot \|v\|}$

Thus ${\displaystyle \|Pv\|\leq \|v\|}$.

For finite dimensional complex or real vector spaces, the standard inner product can be substituted for ${\displaystyle \langle \cdot ,\cdot \rangle }$.

A simple case occurs when the orthogonal projection is onto a line. If ${\displaystyle u}$ is a unit vector on the line, then the projection is given by the outer product

${\displaystyle P_{u}=uu^{\mathrm {T} }.}$

(If ${\displaystyle u}$ is complex-valued, the transpose in the above equation is replaced by a Hermitian transpose). This operator leaves u invariant, and it annihilates all vectors orthogonal to ${\displaystyle u}$, proving that it is indeed the orthogonal projection onto the line containing u.[4] A simple way to see this is to consider an arbitrary vector ${\displaystyle x}$ as the sum of a component on the line (i.e. the projected vector we seek) and another perpendicular to it, ${\displaystyle x=x_{\parallel }+x_{\perp }}$. Applying projection, we get

${\displaystyle P_{u}x=uu^{\mathrm {T} }x_{\parallel }+uu^{\mathrm {T} }x_{\perp }=u\left(\mathrm {sign} (u^{\mathrm {T} }x_{\parallel })\|x_{\parallel }\|\right)+u\cdot 0=x_{\parallel }}$

by the properties of the dot product of parallel and perpendicular vectors.

This formula can be generalized to orthogonal projections on a subspace of arbitrary dimension. Let ${\displaystyle u_{1},\ldots ,u_{k}}$ be an orthonormal basis of the subspace ${\displaystyle U}$, and let ${\displaystyle A}$ denote the ${\displaystyle n\times k}$ matrix whose columns are ${\displaystyle u_{1},\ldots ,u_{k}}$, i.e ${\displaystyle A={\begin{bmatrix}u_{1}&\ldots &u_{k}\end{bmatrix}}}$. Then the projection is given by:[5]

${\displaystyle P_{A}=AA^{\mathrm {T} }}$

which can be rewritten as

${\displaystyle P_{A}=\sum _{i}\langle u_{i},\cdot \rangle u_{i}.}$

The matrix ${\displaystyle A^{\mathrm {T} }}$ is the partial isometry that vanishes on the orthogonal complement of ${\displaystyle U}$ and ${\displaystyle A}$ is the isometry that embeds ${\displaystyle U}$ into the underlying vector space. The range of ${\displaystyle P_{A}}$ is therefore the final space of ${\displaystyle A}$. It is also clear that ${\displaystyle AA^{\mathrm {T} }}$ is the identity operator on ${\displaystyle U}$.

The orthonormality condition can also be dropped. If ${\displaystyle u_{1},\ldots ,u_{k}}$ is a (not necessarily orthonormal) basis, and ${\displaystyle A}$ is the matrix with these vectors as columns, then the projection is:[6][7]

${\displaystyle P_{A}=A(A^{\mathrm {T} }A)^{-1}A^{\mathrm {T} }.}$

The matrix ${\displaystyle A}$ still embeds ${\displaystyle U}$ into the underlying vector space but is no longer an isometry in general. The matrix ${\displaystyle (A^{\mathrm {T} }A)^{-1}}$ is a "normalizing factor" that recovers the norm. For example, the rank-1 operator ${\displaystyle uu^{\mathrm {T} }}$ is not a projection if ${\displaystyle \|u\|\neq 1.}$ After dividing by ${\displaystyle u^{\mathrm {T} }u=\|u\|^{2},}$ we obtain the projection ${\displaystyle u(u^{\mathrm {T} }u)^{-1}u^{\mathrm {T} }}$ onto the subspace spanned by ${\displaystyle u}$.

In the general case, we can have an arbitrary positive definite matrix ${\displaystyle D}$ defining an inner product ${\displaystyle \langle x,y\rangle _{D}=y^{\dagger }Dx}$, and the projection ${\displaystyle P_{A}}$ is given by ${\displaystyle P_{A}x=\mathrm {argmin} _{y\in \mathrm {range} (A)}\|x-y\|_{D}^{2}}$. Then

${\displaystyle P_{A}=A(A^{\mathrm {T} }DA)^{-1}A^{\mathrm {T} }D.}$

When the range space of the projection is generated by a frame (i.e. the number of generators is greater than its dimension), the formula for the projection takes the form: ${\displaystyle P_{A}=AA^{+}}$. Here ${\displaystyle A^{+}}$ stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator.

If ${\displaystyle {\begin{bmatrix}A&B\end{bmatrix}}}$ is a non-singular matrix and ${\displaystyle A^{\mathrm {T} }B=0}$ (i.e., ${\displaystyle B}$ is the null space matrix of ${\displaystyle A}$),[8] the following holds:

${\displaystyle {\begin{aligned}I&={\begin{bmatrix}A&B\end{bmatrix}}{\begin{bmatrix}A&B\end{bmatrix}}^{-1}{\begin{bmatrix}A^{\mathrm {T} }\\B^{\mathrm {T} }\end{bmatrix}}^{-1}{\begin{bmatrix}A^{\mathrm {T} }\\B^{\mathrm {T} }\end{bmatrix}}\\&={\begin{bmatrix}A&B\end{bmatrix}}\left({\begin{bmatrix}A^{\mathrm {T} }\\B^{\mathrm {T} }\end{bmatrix}}{\begin{bmatrix}A&B\end{bmatrix}}\right)^{-1}{\begin{bmatrix}A^{\mathrm {T} }\\B^{\mathrm {T} }\end{bmatrix}}\\&={\begin{bmatrix}A&B\end{bmatrix}}{\begin{bmatrix}A^{\mathrm {T} }A&O\\O&B^{\mathrm {T} }B\end{bmatrix}}^{-1}{\begin{bmatrix}A^{\mathrm {T} }\\B^{\mathrm {T} }\end{bmatrix}}\\[4pt]&=A(A^{\mathrm {T} }A)^{-1}A^{\mathrm {T} }+B(B^{\mathrm {T} }B)^{-1}B^{\mathrm {T} }\end{aligned}}}$

If the orthogonal condition is enhanced to ${\displaystyle A^{\mathrm {T} }WB=A^{\mathrm {T} }W^{\mathrm {T} }B=0}$ with ${\displaystyle W}$ non-singular, the following holds:

${\displaystyle I={\begin{bmatrix}A&B\end{bmatrix}}{\begin{bmatrix}(A^{\mathrm {T} }WA)^{-1}A^{\mathrm {T} }\\(B^{\mathrm {T} }WB)^{-1}B^{\mathrm {T} }\end{bmatrix}}W.}$

All these formulas also hold for complex inner product spaces, provided that the conjugate transpose is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014).[9] Also see Banerjee (2004)[10] for application of sums of projectors in basic spherical trigonometry.

The term oblique projections is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see oblique projection), though not as frequently as orthogonal projections. Whereas calculating the fitted value of an ordinary least squares regression requires an orthogonal projection, calculating the fitted value of an instrumental variables regression requires an oblique projection.

Projections are defined by their null space and the basis vectors used to characterize their range (which is the complement of the null space). When these basis vectors are orthogonal to the null space, then the projection is an orthogonal projection. When these basis vectors are not orthogonal to the null space, the projection is an oblique projection. Let the vectors ${\displaystyle u_{1},\ldots ,u_{k}}$ form a basis for the range of the projection, and assemble these vectors in the ${\displaystyle n\times k}$ matrix ${\displaystyle A}$. The range and the null space are complementary spaces, so the null space has dimension ${\displaystyle n-k}$. It follows that the orthogonal complement of the null space has dimension ${\displaystyle k}$. Let ${\displaystyle v_{1},\ldots ,v_{k}}$ form a basis for the orthogonal complement of the null space of the projection, and assemble these vectors in the matrix ${\displaystyle B}$. Then the projection is defined by

${\displaystyle P=A(B^{\mathrm {T} }A)^{-1}B^{\mathrm {T} }.}$

This expression generalizes the formula for orthogonal projections given above.[11][12]

Let ${\displaystyle V}$ be a vector space (in this case a plane) spanned by orthogonal vectors ${\displaystyle u_{1},u_{2},\cdots ,u_{p}}$. Let ${\displaystyle y}$ be a vector. One can define a projection of ${\displaystyle y}$ onto ${\displaystyle V}$ as

${\displaystyle \operatorname {proj} _{V}y={\frac {y\cdot u^{i}}{u^{j}\cdot u^{j}}}u^{i}}$

where the ${\displaystyle i}$ 's and ${\displaystyle j}$ 's imply Einstein sum notation. The vector ${\displaystyle y}$ can be written as an orthogonal sum such that ${\displaystyle y=\operatorname {proj} _{V}y+z}$. ${\displaystyle \operatorname {proj} _{V}y}$ is sometimes denoted as ${\displaystyle {\hat {y}}}$. There is a theorem in Linear Algebra that states that this ${\displaystyle z}$ is the shortest distance from ${\displaystyle y}$ to ${\displaystyle V}$ and is commonly used in areas such as machine learning.

y is being projected onto the vector space V.