5.5 Estimator Property: Consistency

Foundations makes another of their jokes when they write, on page 105,

“Consistency is a simple notion: if we had enough data, the probability that our estimate \(\hat{\theta}\) would be far from the truth, \(\theta\), would be small.”

How do we determine if a particular estimator, \(\hat{\theta}\) is a consistent estimator for our parameter of interest?

There are at least two ways:

  1. The estimator is unbiased, and has a sampling variance that decreases as we add data; or,
  2. We can use Chebyshev’s to place a bound on the estimator, showing that as we add data, the estimator converges in probability to \(\theta\).

The first notion of convergence requires an understanding of sampling variance:

The sampling variance of an estimator is a statement about how much dispersion due to random sampling, is present in the estimator. We defined the variance of a random variable to be \(E\left[(X - E[X])^{2}\right]\), The sampling variance uses this same definition, but we work with it slightly differently when we are considering sampling variance.

In particular, when we are considering sampling variance, we do not typically got as far as actually computing the variance of the underlying random variable? Why? Because, if we’re working in a sampling scenario, it is unlikely that we have access to the underlying function that governs the PDF of the random variable.

Instead, we typically start from a statement of the estimator that is under consideration, and apply the variance operator against that estimator. Consider, for example, forming a statement about the sampling variance of the sample average.

Let \(\overline{X} \equiv\) “sample average” \(\equiv \frac{1}{n}\sum_{i=1}^{n}X_{i}\) be the normal form of the sample average.

Earlier, we proved that \(\overline{X}\) is an unbiased estimator of \(E[X]\).

What is the sampling variance of the sample average?

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Using the statement that you have just produced, would you say that the sample average is a consistent estimator for the population expectation of a random variable?