Constrained least squares

In constrained least squares one solves a linear least squares problem with an additional constraint on the solution.[1] I.e., the unconstrained equation ${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\mathbf {y} }$ must be fit as closely as possible (in the least squares sense) while ensuring that some other property of ${\displaystyle {\boldsymbol {\beta }}}$ is maintained.

There are often special-purpose algorithms for solving such problems efficiently. Some examples of constraints are given below:

• Equality constrained least squares: the elements of ${\displaystyle {\boldsymbol {\beta }}}$ must exactly satisfy ${\displaystyle \mathbf {L} {\boldsymbol {\beta }}=\mathbf {d} }$ (see Ordinary least squares).
• Regularized least squares: the elements of ${\displaystyle {\boldsymbol {\beta }}}$ must satisfy ${\displaystyle \|\mathbf {L} {\boldsymbol {\beta }}-\mathbf {y} \|\leq \alpha }$ (choosing ${\displaystyle \alpha }$ in proportion to the noise standard deviation of y prevents over-fitting).
• Non-negative least squares (NNLS): The vector ${\displaystyle {\boldsymbol {\beta }}}$ must satisfy the vector inequality ${\displaystyle {\boldsymbol {\beta }}\geq {\boldsymbol {0}}}$ defined componentwise—that is, each component must be either positive or zero.
• Box-constrained least squares: The vector ${\displaystyle {\boldsymbol {\beta }}}$ must satisfy the vector inequalities ${\displaystyle {\boldsymbol {lb}}\leq {\boldsymbol {\beta }}\leq {\boldsymbol {ub}}}$, each of which is defined componentwise.
• Integer-constrained least squares: all elements of ${\displaystyle {\boldsymbol {\beta }}}$ must be integers (instead of real numbers).
• Phase-constrained least squares: all elements of ${\displaystyle {\boldsymbol {\beta }}}$ must be real numbers, all multiplied by a same complex number of unit modulus.

When the constraint only applies to some of the variables, the mixed problem may be solved using separable least squares by letting ${\displaystyle \mathbf {X} =[\mathbf {X_{1}} \mathbf {X_{2}} ]}$ and ${\displaystyle \mathbf {\beta } ^{\rm {T}}=[\mathbf {\beta _{1}} ^{\rm {T}}\mathbf {\beta _{2}} ^{\rm {T}}]}$ represent the unconstrained (1) and constrained (2) components. Then substituting the least-squares solution for ${\displaystyle \mathbf {\beta _{1}} }$, i.e.

${\displaystyle {\hat {\boldsymbol {\beta }}}_{1}=\mathbf {X} _{1}^{+}(\mathbf {y} -\mathbf {X} _{2}{\boldsymbol {\beta }}_{2})}$

(where ⁺ indicates the Moore–Penrose pseudoinverse) back into the original expression gives (following some rearrangement) an equation that can be solved as a purely constrained problem in ${\displaystyle \mathbf {\beta } _{2}}$.

${\displaystyle \mathbf {P} \mathbf {X} _{2}{\boldsymbol {\beta }}_{2}=\mathbf {P} \mathbf {y} ,}$

where ${\displaystyle \mathbf {P} :=\mathbf {I} -\mathbf {X} _{1}\mathbf {X} _{1}^{+}}$ is a projection matrix. Following the constrained estimation of ${\displaystyle {\hat {\boldsymbol {\beta }}}_{2}}$ the vector ${\displaystyle {\hat {\boldsymbol {\beta }}}_{1}}$ is obtained from the expression above.