Point Estimates

Definition: Suppose a random variable ${\displaystyle X\,}$ follows a statistical distribution or a law ${\displaystyle {\mathcal {L}}_{\theta }}$ indexed by parameter ${\displaystyle \theta \,}$. Then a function ${\displaystyle g:X\rightarrow {\hat {\theta }}}$ from the sample space to the parameter space is called a point estimator of ${\displaystyle \theta \,}$.

In general, let ${\displaystyle f(\theta )\,}$ be any function of ${\displaystyle \theta \,}$. Then any function from the sample space to the domain of ${\displaystyle f\,}$ will be called a point estimator of ${\displaystyle f(\theta )\,}$.

Definition: If ${\displaystyle h\,}$ is a point estimator for ${\displaystyle \theta \,}$, then for a realization ${\displaystyle x\,}$ of the random variable ${\displaystyle X\,}$, the quantity ${\displaystyle h(x)\,}$ is called a point estimate of ${\displaystyle \theta \,}$ and is denoted as ${\displaystyle {\hat {\theta }}\,}$.

Notice that the estimate ${\displaystyle {\hat {\theta }}\,}$ is a random variable (unlike the true parameter ${\displaystyle \theta \,}$), since it depends on ${\displaystyle X\,}$.

Examples

1. Suppose ${\displaystyle X_{1},X_{2},...,X_{n}\,}$ follow independent Normal(μ,σ²). Then an estimator for the mean μ is the sample mean ${\displaystyle \sum X_{i} \over n\,}$.
2. Suppose ${\displaystyle X_{1},X_{2},...,X_{n}\,}$ follow Uniform[θ,θ+1]. Then an estimator for θ is ${\displaystyle \min X_{i}\,}$. Another is ${\displaystyle \max X_{i}-1\,}$. Yet another is ${\displaystyle {\frac {\sum X_{i}}{n}}-{\tfrac {1}{2}}\,}$

Notice that the above definition does not restrict the point estimator to only the "good" ones. For example, according to the definition it is perfectly fine to estimate the mean ${\displaystyle \mu \,}$ in the above example as something absurd, like ${\displaystyle 10\sum X_{i}^{2}+\exp(X_{1})\,}$. This freedom is in the definition is deliberate. In general, however, when we form point estimators we take some measure of goodness into account. It should be kept in mind that the point estimators will always be targeted to be close to the parameter it estimates, intuitively and if possible, formally.

A variety of methods are used to evaluate the effectiveness of a particular estimator. One such measure of goodness is called the bias. Bias is defined as in terms of the closeness of the expectation for the estimator to the actual parameter value. For example if estimating a parameter, ${\displaystyle \theta }$ with an estimator, ${\displaystyle W(x)}$ then ${\displaystyle bias(W(x))=E[W(x)-\theta ]}$. When bias is zero, the estimator is called unbiased.

As an example we can show that the sample mean is unbiased for distribution mean. ${\displaystyle E[{\frac {1}{n}}\sum X_{i}]={\frac {1}{n}}E[\sum X_{i}]={\frac {1}{n}}\sum {E[X_{i}]}={\frac {n}{n}}E[X]=\mu }$.

Unbiasedness alone means the positive and negative errors balance out, but it is not the only measure of quality. This can be thought of as accuracy, but we might also be concerned about precision. Precision is often measured by the mean squared error, or MSE for short. ${\displaystyle MSE=E[(W(x)-\theta )^{2}]}$