Interval Estimation

“확률과 통계(MATH230)” 수업에서 배운 것과 공부한 것을 정리한 포스트입니다. 전체 포스트는 Probability and Statistics에서 확인하실 수 있습니다 🎲

Introduction to Interval Estimation

Let $X_1, X_2, \dots, X_n$ be a random sample with $X_i \sim f(x; \theta)$, and $x_1, x_2, \dots, x_n$ be the values of the sample.

Here $\theta$ is unknown. The <interval estimation> is to find $(\hat{\theta}_L, \hat{\theta}_U)$ in which we expect to find the true value of the parameter $\theta$.

Q. How to find the interval?

A1. $\theta \in (-\infty, \infty)$ → bad! 🤬

A2. We would construct two estimators $\hat{\theta}_L$ and $\hat{\theta}_U$ from the random sample such that $P(\hat{\theta}_L < \theta < \hat{\theta}_U) = 1 - \alpha$.

Here, $(\hat{\theta}_L, \hat{\theta}_U)$ is called an <interval estimator> of $\theta$, $(\hat{\theta}_L, \hat{\theta}_U)$ is called a $100 \cdot (1 - \alpha)$% confidence interval. 🔥

Also, $1-\alpha$ is called the <confidence coefficient> or <confidence level>. 🔥

We usually take $\alpha = 0.01, \; 0.05, \; 0.1$.

💥 Note that $(\hat{\theta}_L, \hat{\theta}_U)$ is not unique!! (꼭 대칭일 필요는 없다는 말)

---

Interval Estimation

이제 상황에 따른 <Interval Estimation> 방법을 살펴보겠다!

• Estimate $\mu$ when $\sigma^2$ is known
• Estimate $\mu$ when $\sigma^2$ is unknown

---

z-value: Estimate $\mu$ when $\sigma^2$ is known

Example.

• $n=100$, $\bar{X} = 170$.
• $\sigma = 20$

1. We use $\bar{X}$ as a point estimator for $\mu$.

2. We can use CLT here.

\[\frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \overset{D}{\approx} N(0, 1)\] \[\begin{aligned} 0.95 &= P(\hat{\mu}_L < \mu < \hat{\mu}_U) \\ &= P(-z_{0.025} \le z \le z_{0.025}) \\ &\approx P \left(-1.96 \le \frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \le 1.96 \right) \\ &= P \left( \bar{X} - 1.96 \frac{\sigma}{\sqrt{n}} \le \mu \le \bar{X} + 1.96 \frac{\sigma}{\sqrt{n}} \right) \end{aligned}\]

Here, we have $\hat{\mu}_L := \bar{X} - 1.96 \dfrac{\sigma}{\sqrt{n}}$, $\hat{\mu}_U := \bar{X} + 1.96 \dfrac{\sigma}{\sqrt{n}}$

$\therefore$ A 95% confidence interval would be $(170 - 3.92, 170 + 3.92)$.

Remark. Confidence Interval on $\mu$ when $\sigma^2$ is known

Let $x_1, \dots, x_n$ be given data points from a random sample $X_1, \dots, X_n$ with known population variance $\sigma^2$ and unknown population mean $\mu$.

If $\bar{x}$ is the sample mean, a $100(1-\alpha)\%$ confidence interval for $\mu$ is given by

\[\left( \bar{x} - z_{\alpha/2} \frac{\sigma}{\sqrt{n}} , \; \bar{x} + z_{\alpha/2} \frac{\sigma}{\sqrt{n}} \right)\]

💥 Note that this is an approximate interval unless $X_i \sim N(\mu, \sigma^2)$.

Q1. Is it true that $P \left( \mu \in (\hat{\mu}_L, \hat{\mu}_U) \right) \overset{?}{=} 0.95$? 🔥

A1. No!! $\mu$ is not a random variable!

Q2. Then what does the confidence interval really mean?

A2. Its the number of counts that $\mu$ is actually belong to sampled interval! 🔥

Let $x_1, \dots, x_n$ be a random sample. Supp. we obtain 1,000 samples and each sample has size $n$. Then, we have the following:

1st sample: $x_{11}, x_{12}, \dots, x_{1n}$ → $\bar{x}_1$ → confidence \((\bar{\mu}_{1L}, \bar{\mu}_{1U})\)

2nd sample: $x_{21}, x_{22}, \dots, x_{2n}$ → $\bar{x}_1$ → confidence \((\bar{\mu}_{2L}, \bar{\mu}_{2U})\)

$\vdots$

1000th sample: $x_{1000, 1}, x_{1000, 2}, \dots, x_{1000, n}$ → $\bar{x}_1$ → confidence \((\bar{\mu}_{1000, L}, \bar{\mu}_{1000, U})\)

이렇게 얻은 1,000개의 interval estimation에 대해, 정확히 95%의 비율, 즉 950개의 interval에 true parameter $\mu$가 실제로 포함되어 있다는 말!

---

Error of Interval Estimation

Definition. Error of estimation

Now, let’s consider the error $| \bar{x} - \mu |$.

For an estimated interval $\left( \bar{x} - z_{\alpha/2} \frac{\sigma}{\sqrt{n}}, \bar{x} + z_{\alpha/2} \frac{\sigma}{\sqrt{n}}\right)$, the error is

\[\left| \bar{x} - \mu \right| \le z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}\]

Theorem.

If $\bar{x}$ is used as an estimate of $\mu$, we can be $100(1-\alpha)\%$ confident that the error will not exceed $z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$.

Theorem.

Q. How large can the sample size be if the error is at most $\epsilon$?

A. We want $\text{Err} = z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$ to be less than $\epsilon$.

\[\text{Err} = z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \le \epsilon\]

Solve the inequality for $n$.

\[n \ge \left[ \frac{z_{\alpha/2} \cdot \sigma}{\epsilon} \right]^2\]

---

One-sided Confidence Bounds

지금까지 우리는 양 끝의 상황을 살펴보는 Two-sided Confidence Interval을 살펴보았다. 그러나 때로는 한쪽의 상황만 관심의 대상이 될 수도 있다! 그래서 아래와 같이 One-side에 대한 Confidence Interval을 구해야 할 수도 있다.

\[P(\hat{\theta}_L \le \mu) = 1 - \alpha\]

사실 two-sided에서 약간만 수정해주면 된다. two-sided에서의 Confidence Interval이 아래와 같다면,

\[\bar{x} - z_{\alpha/2} \frac{\sigma}{\sqrt{n}} \; \le \; \mu \; \le \; \bar{x} + z_{\alpha/2} \frac{\sigma}{\sqrt{n}}\]

여기에서 한쪽만 취해 $\alpha$를 사용하면 된다. 즉,

\[\bar{x} - z_{\textcolor{red}{\alpha}} \frac{\sigma}{\sqrt{n}} \; \le \; \mu\]

이것은 곧

\[P \left(\bar{x} - z_{\textcolor{red}{\alpha}} \frac{\sigma}{\sqrt{n}} \; \le \; \mu \right) = 1 - \alpha\]

와 같다!

---

t-value: Estimate $\mu$ when $\sigma^2$ is unknown

앞에서 진행했던 과정을 다시 살펴보자. 우리는 CLT를 사용해 sample mean $\bar{X}$를 Normal 분포로 근사했다.

\[Z = \frac{\bar{X} - \mu}{\sigma / \sqrt{n}}\]

그런데! population variance $\sigma$를 모르는 지금 상황에서는 위와 같이 접근할 수 없다!! 😲 $\sigma$를 모르기 때문에 CLT 근사식에서 분모 부분에 $\sigma$를 쓸 수 없기 때문이다!

우리가 그나마 $\sigma^2$와 비슷하다고 생각하는 것이 있다. 바로 “sample variance” $S^2$! 이 녀석으로 $\sigma$를 대체해 식을 다시 써보자.

\[\frac{\bar{X} - \mu}{S / \sqrt{n}}\]

어라! 이 식은 student’s t-distribution에서 이미 살펴보았다.

\[\frac{\bar{X} - \mu}{S / \sqrt{n}} \; \overset{D}{\sim} \; t(n-1)\]

그래서 $t(n-1)$ distribution에서 $100(1-\alpha)\%$ confidence interval을 구하면,

\[P \left( -t_{\alpha/2} (n-1) < \frac{\bar{X} - \mu}{S / \sqrt{n}} < t_{\alpha/2}(n-1) \right) = 1 - \alpha\]

가 된다!

Remark. Confidence Interval on $\mu$ when $\sigma^2$ is unknown

Let $x_1, \dots, x_n$ be given data points from a normal random sample $X_1, \dots, X_n$ with mean $\mu$ and variance $\sigma^2$.

Here, the population mean $\mu$ and the population variance $\sigma$ are both unknown.

If $\bar{x}$ is the sample mean and $s^2$ is the sample variance, then a $100(1-\alpha)\%$ confidence interval for $\mu$ is given by

\[\left( \bar{x} - t_{\alpha/2}(n-1) \frac{s}{\sqrt{n}} , \; \bar{x} + t_{\alpha/2}(n-1) \frac{s}{\sqrt{n}} \right)\]

Remark.

1. The width of the interval is random!

\[\left| \bar{x} - \mu \right| < t_{\alpha/2}(n-1) \cdot \frac{s}{\sqrt{n}}\]

2. This confidence interval is not an approximation, since we assume sample $X_i$ as iid. normal $\mu$, $\sigma^2$.

---

Compare Point Estimator and Interval Estimator

Q. Does confidence interval give us more information about $\mu$ than a point estimator $\bar{x}$?

A. Not really… 🤔

[Point Estimator]

For a point estimator $\bar{x}$,

by LLN, $\bar{x} \rightarrow \mu$ as $n\rightarrow\infty$.

And the variance $\text{Var}(\bar{x}) = \sigma^2/n$.

[Interval Estimator]

For an interval estimator $\left(\bar{x} - z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}, \; \bar{x} + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \right)$,

the error is $\left| \bar{x} - \mu \right| \le z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$.

💥 This means the width of the confidence interval is determined by the standard deviation of the point estimator!! 이것은 둘 중 한 Estimator가 개선되면, 다른 하나의 성능도 개선됨을 말한다.

---

이어지는 포스트들에서는 상황별로 <Interval Estimation>을 수행하는 방법을 살펴볼 예정이다! 🤩

• Prediction & Tolerance Estimation
• Two Samples Estimation: Diff Btw Two Means
• Two Samples Estimation: Paired Observations
• Proportion Estimation
  • Single Sample Estimation: Proportion Estimation
  • Two Samples Estimation: Diff btw Two Proportions
• Variance Estimation
  • Single Sample Estimation: Variance Estimation
  • Two Samples Estimation: The Ratio of Two Variances