StatisticsProbability

Poisson Errors: Estimating Typos Missed by Two Proofreaders

Estimate the unseen typo distribution with Poisson splitting and two readers, then calculate the expected benefit of a third reader.

Question

A text's total error count follows Poisson⁡(λ)\operatorname{Poisson}(\lambda). One motivating approximation uses 100 pages, 100 lines on each page, and 100 character positions per line: one million independent, identically distributed Bernoulli error opportunities. For a portion A with k character positions, the approximate mean error count is λk/106\lambda k/10^6. Two proofreaders independently inspect the text. Each error is detected independently by reader i with probability pip_i, i=1,2. Define X1X_1 as errors found only by reader 1, X2X_2 as errors found only by reader 2, X3X_3 as errors found by both, and X4X_4 as errors found by neither. Estimate the distribution of X4X_4 and the benefit of an independent third reader who detects each error with probability qq.

(a) Specify the joint law of X1X_1, X2X_2, X3X_3, X4X_4.

(b) Verify E[X1]/E[X3]=(1−p2)/p2E[X_1]/E[X_3]=(1-p_2)/p_2 and E[X2]/E[X3]=(1−p1)/p1E[X_2]/E[X_3]=(1-p_1)/p_1.

(c) Estimate p1p_1, p2p_2, λ\lambda by replacing E[Xi]E[X_i] with the observed X_i.

(d) Estimate the mean number of errors missed by both readers.

(e) With X1=60X_1=60, X2=30X_2=30, X3=40X_3=40, estimate the distribution of X4X_4.

(f) For these counts, an independent third reader has qq=0.9. By how much does the expected missed-error count decrease?

(a) Joint distribution

Let N∼Poisson⁡(λ)N\sim\operatorname{Poisson}(\lambda) be the total count. Each error enters exactly one of four categories, with probabilities

(r1,r2,r3,r4)=(p1(1−p2),(1−p1)p2,p1p2,(1−p1)(1−p2)).(r_1,r_2,r_3,r_4)=\bigl(p_1(1-p_2),(1-p_1)p_2,p_1p_2,(1-p_1)(1-p_2)\bigr).

Conditional on N=nN=n, the four counts have a multinomial distribution and sum to nn. Without conditioning on NN, Poisson splitting gives mutually independent counts Xj∼Poisson⁡(λrj)X_j\sim\operatorname{Poisson}(\lambda r_j). Their joint probability is

P(X1=x1,…,X4=x4)=∏j=14e−λrj(λrj)xjxj!,xj=0,1,2,….P(X_1=x_1,\ldots,X_4=x_4)=\prod_{j=1}^{4}e^{-\lambda r_j}\frac{(\lambda r_j)^{x_j}}{x_j!},\qquad x_j=0,1,2,\ldots.

To see the independence, multiply the Poisson probability for N=∑xjN=\sum x_j by the multinomial probability for those category counts. The factorial for NN cancels, and ∑rj=1\sum r_j=1 factors the exponential into four terms. Independence here is unconditional; fixed-total multinomial counts are generally dependent.

(b) Ratios of means

Writing μj=E[Xj]\mu_j=E[X_j] and assuming λp1p2>0\lambda p_1p_2>0,

μ1μ3=λp1(1−p2)λp1p2=1−p2p2,μ2μ3=λ(1−p1)p2λp1p2=1−p1p1.\frac{\mu_1}{\mu_3}=\frac{\lambda p_1(1-p_2)}{\lambda p_1p_2}=\frac{1-p_2}{p_2},\qquad \frac{\mu_2}{\mu_3}=\frac{\lambda(1-p_1)p_2}{\lambda p_1p_2}=\frac{1-p_1}{p_1}.

(c) Estimates of the detection probabilities and total mean

For observed counts x1,x2,x3x_1,x_2,x_3 with x3>0x_3>0, replacing means by counts gives

p^1=x3x2+x3,p^2=x3x1+x3,λ^=(x1+x3)(x2+x3)x3.\widehat p_1=\frac{x_3}{x_2+x_3},\qquad \widehat p_2=\frac{x_3}{x_1+x_3},\qquad \widehat\lambda=\frac{(x_1+x_3)(x_2+x_3)}{x_3}.

Reader 1's probability is estimated using the proportion of reader 2's discoveries also found by reader 1. The formula for λ^\widehat\lambda follows from μ3=λp1p2\mu_3=\lambda p_1p_2. These are model-based plug-in estimates, not assertions about a known total number of typos. The zero-overlap case x3=0x_3=0 makes this estimation formula unavailable; do not divide by zero or infer that there are no unseen errors.

(d) Mean number missed by both

μ^4=λ^(1−p^1)(1−p^2)=x1x2x3=λ^−(x1+x2+x3).\widehat\mu_4=\widehat\lambda(1-\widehat p_1)(1-\widehat p_2) =\frac{x_1x_2}{x_3} =\widehat\lambda-(x_1+x_2+x_3).

The resulting estimated law is Poisson⁡(μ^4)\operatorname{Poisson}(\widehat\mu_4).

(e) Estimated distribution for the supplied counts

For (x1,x2,x3)=(60,30,40)(x_1,x_2,x_3)=(60,30,40),

p^1=4070=47,p^2=40100=25,λ^=100⋅7040=175,μ^4=60⋅3040=45.\widehat p_1=\frac{40}{70}=\frac47,\quad \widehat p_2=\frac{40}{100}=\frac25,\quad \widehat\lambda=\frac{100\cdot70}{40}=175,\quad \widehat\mu_4=\frac{60\cdot30}{40}=45.

Thus the plug-in distribution for X4X_4 is Poisson with mean 45:

P^(X4=k)=e−4545kk!,k=0,1,2,….\widehat P(X_4=k)=e^{-45}\frac{45^k}{k!},\qquad k=0,1,2,\ldots.

The estimate does not say that exactly 45 errors remain. It is a distribution with estimated mean and variance 45.

(f) Benefit of a third independent proofreader

Let YY be the number still missed after the third reader. Given X4X_4, Y∣X4∼Binomial⁡(X4,1−q)Y\mid X_4\sim\operatorname{Binomial}(X_4,1-q). Thinning therefore gives Y∼Poisson⁡(μ4(1−q))Y\sim\operatorname{Poisson}(\mu_4(1-q)). With q=0.9q=0.9, the estimated remaining mean is

45(1−0.9)=4.5.45(1-0.9)=4.5.

The expected missed-error count decreases by 45−4.5=40.545-4.5=40.5 errors, a 90% reduction. These are expected counts, so they need not be integers. This conclusion assumes independent detection with the same stated probability for every remaining error.

Evidence boundary

This is the six-part March 10, 2011 Columbia IEOR 4106 version. The final part is labeled (f) here to distinguish the source’s two successive (e) labels. Counts 60, 30, 40 and detection probability 0.9 are supplied by the exercise. The Poisson laws and plug-in estimates rely on independent, homogeneous error detection; estimated means are not observed remaining counts. The formulas for parameter estimation require positive observed overlap.

Sources

These references support the concepts and methods used in the explanation above.