Deletion audits

The population audit was run against the depth column: fresh member, non-member and audited samples each replicate, the audit feature a count of 2-moves over the three positions after the first, 200 replicates per cell at audit sizes 50, 200 and 1000, the exponents (a, b) placing the target at 1/2, 1/5 and 4/5. The estimator fits the audited set's COVARIANCE — a variance here, the feature being a single number — against the mixture's predicted covariance, and does not use its own model's mean equation, though that equation identifies the weight linearly. Two consequences print.

The flip. The predicted covariance is invariant under π ↔ 1 − π whenever the two class covariances are equal, so there the objective has twin minima and no sample size separates them: on equal-variance Gaussian cells — no worlds involved — the estimate lands nearer 1 − π than π in about half the replicates at every audit size, a truth of 0.2 read as 0.8. In the world cells the two class variances differ substantially — by factors of about 1.4 and 1.75 — and the audit there is identified outright: the objective's second solution lies far outside the interval the weight is searched in, so there is no second minimum to land on. The flip persists anyway, fading with data while staying substantial at working sizes: 28% of replicates at one target placement at audit size 200, 10% at 1000. What produces it there is not a rival minimum but the ordinary spread of a single number. Where the objective's vertex falls outside the interval the weight is searched in, the estimate is a monotone function of one quantity — the audited set's variance less the member class's — so it crosses the midpoint exactly when that quantity crosses a threshold the two class samples fix, and a flip is the solution itself drifting past the midpoint rather than any silent fallback. The rate then follows from the exact class moments with nothing fitted: 0.387, 0.283 and 0.099 predicted at the three audit sizes against 0.385, 0.280 and 0.100 measured.

The band. The bootstrap band covers the exact Fraction at 69–86% across the world cells at every audit size, never near the 90% its quantile levels would suggest — and the band NARROWS with data while the coverage stays where it was: a stated precision tightening while the miss it should cover does not shrink.

Scope. Observation: one audit tool's low-order-moment estimator, one seed stream, scalar features, one world family. The twin-minima invariance follows from that estimator's objective alone and is reproduced ring-free on Gaussian cells. The mixture model holds exactly at fiber inputs by construction; nothing here scores an audit whose feature model must be taken on faith.

verifiers: explore_deletion_ruler.py, explore_flip_level.py

That identification is a fact about the population, and it is not what a sample hands the tool. Write out what the estimator solves. With A the audited set's variance less the member class's, B the squared gap between the two class means and G the class-variance gap plus B, the audit minimizes (A − πG + π²B)² over the box, the interval [0, 1] the weight is searched in. Only A moves with the audited set. So the equation the tool means to solve has a solution in the box exactly when the measured A falls in the range that A(p) = Gp − Bp² sweeps over the box — and when it does not, the tool returns the box-constrained minimizer with no solution behind it, silently. That is not rare at working sizes: at one target placement it happens in 0.525, 0.400 and 0.237 of fresh replicates at audit sizes 50, 200 and 1000, and in 0.564, 0.431 and 0.301 of the resamples the band is built from.

The rate is set by a margin in the coefficients and carries no term for the audit feature at all: it is a normal tail at the distance from A to the nearer edge of that range, over the spread of the measured A. Across eighteen cells the largest deviation is 0.125 at the smallest audit size and 0.069 from size 200 up. The three cells left out are the ones with a binary feature, where the margin is exactly zero: the objective is tangent to zero at the truth instead of crossing it, so the fluctuation that can leave the range is one-sided and the measured rate is 0.010 where a two-sided tail says 0.500. Coarsening the audit feature acts through the margin and through nothing else wherever the objective still crosses — quantizing a continuous feature to three levels shrinks the margin twenty-two-fold and reproduces the world cells' plateau — and inducing that tangency is the only other route it has, which is why the rate is not monotone in the number of levels.

Two different failures answer to the one name. Where the two solutions are complex the objective has a strict interior minimum and the fallback lands on the vertex, the turning point G/2B; where they are real and both outside the box the fallback lands on a box endpoint instead. The first is 99–100% of the events at every quantized cell and the second 16–65% of them at the world cells. And at both world cells the endpoint it lands on is the one nearest the truth — 0 against a truth of 1/5, 1 against 4/5 — so this failure is invisible in a flip count, which asks only whether the estimate crossed the midpoint.

The two are cuts of the same scalar at two different levels. No solution is the measured A past an edge of the range; a flip is that same number past its half-box value G/2 − B/4. Whenever the vertex lies outside the box the half-box value is strictly interior to the range — by construction, the midpoint being interior to the box — so the two events can overlap only beyond the other edge — the one whose fallback lands away from the truth. That overlap is not uniformly small: 1.8% of replicates and 7% of flips at audit size 200, but 16.5% and 37% at size 50, where the far-edge fallback is fat.

Scope. Observation for the rate, over 18 of 21 cells: two settings of a continuous Gaussian dial, that dial's feature quantized to 2, 3 and 4 levels, and the two world cells, each at three audit sizes, with 200 replicates and 200 resamples per replicate — the audited tool's own counts — on one seed stream. The tangent case is named and derived but not modelled: its rate turns on which side of the feature's own variance map the population sits on, which no margin in the coefficients carries, and no cell here dials through a tangency. The two-level geometry is a property of the objective. Controls read before any of it: the exact hypergeometric moments reproduce the two world cells' population solutions in exact rational arithmetic with zero difference, the samplers recover the exact class variances to 0.0022, and the continuous sweep reproduces the audit's own recorded trajectory. The frozen tolerance was a single bar over all cells and it failed at the smallest audit size in both directions — over by 0.125 at one cell, under by 0.111 at another — so what degrades there is the normal approximation and not a term with a sign; two candidate causes act only at that size and this rig does not separate them. One audit tool, scalar features, one world family. Toy scale.

verifiers: explore_noroot_margin.py, explore_flip_level.py

That half-box cut has four inputs, and three of them are available to an auditor who has no truth: the measured A from the audited sample, and G and B from the two class samples. The fourth is the cut's direction, which turns on which side of the midpoint the true weight sits — precisely what an audit is for. So what the cut buys from inside is not a flip rate. It is the probability that a fresh sample's estimate would land on the other side of the midpoint from the number in hand: an instability of the auditor's own verdict, which equals the flip risk when that verdict is on the truth's side and its complement when it is not, and is alarming either way. It survives the missing direction by being a statement about the sign of one scalar, D = A − (G/2 − B/4): two replicates land on opposite sides of the midpoint exactly when their Ds differ in sign, and both carry the same unknown direction.

Scored as a forecast of that event, the statement needs two corrections and neither of them is the spread. The spread is solved: estimating the fourth moments from the same sample that produced A matches a resampled spread to 1.5% at the worst of nine rows and under 0.7% at the other eight, against an error in the form itself that reaches a factor of 5.4. What has to change is which spread — D reads all three samples and the member class's variance enters it and A with opposite signs, so the substitution runs 16% narrower at one cell and 11% wider at the other, with no fixed sign — and that the spread is predictive, the auditor centring on their own D rather than on the population's, which costs a factor √2.

Corrected, the statement is still compressed. Write μ for the standardized distance the auditor is estimating — |D| over the spread of D, evaluated at the population rather than at the sample in hand — and μ̂ for that same ratio computed from the sample. The stated number equals that form's own expectation to within 0.009 at every world row, so what follows is the form and not the arithmetic. As μ sweeps 0.37 to 2.47 the two-sided normal model of the event sweeps 0.459 down to 0.013 while the statement moves only 0.286 to 0.073 — and the model is not the measurement: the realized rate sits above it at most rows and below it at the largest μ, so the verdict holds on either reading. Which rows fall on which side moves with the sample, a per-row realized rate at this replicate count carrying about 0.03 of noise, so nothing below is argued from one row. The statement is right in the mean at μ ≈ 1.1 alone: below that it under-warns, above it it over-warns, and the ratio to the event diverges, the warning becoming unboundedly conservative exactly where the audit's number is most stable. The cause is exact rather than a tuning miss — a predictive spread is the optimal answer under a flat prior on D, where a fixed cell holds D at a point.

The compression is not a property of the corrected statement in particular: one formula prices every form the statement could have taken. Where D is normal about its population value at that spread, μ̂ is |μ + Z| with Z standard normal, so a statement Φ(−μ̂/s) at any scale s has mean E[Φ(−|μ + Z|/s)] — the folded convolution, one integral covering the whole family. It lands within 0.023 of measurement at four scales sweeping s = 0.90 to 1.72, at every cell and audit size. That includes the designed cell, one built to put the population vertex inside the box, where no replicate has a derivation behind it at all: what a statement says is the formula's business and whether what it says is right is the derivation's, and only a cell built to break the second separates them. Dropping the fold gives the closed form Φ(−μ/√(s² + 1)), which the folded reading converges to monotonically in μ — over the four scales the two differ by at most 0.158 at μ = 0.37 and at most 0.002 at 2.47, so the closed form's own miss against measurement reaches 0.147 at the smallest μ and under 0.010 at both largest-μ rows. The corrected statement's decay is that closed form at s = √2, giving Φ(−μ/√3); the exponent was the scale's and never the form's.

What earns the number its place anyway is that the case it cannot cover announces itself from inside. The whole derivation assumes the vertex lies outside the box, and the replicate's own G/2B needs no truth to compute. On the designed cell it fires on 95–100% of replicates; on the two world cells, where the derivation holds, it fires on 0–29% and stands down as the audit grows. The condition is one a design controls — the two class variances differing by no more than the squared gap between their means — and at equality the test reads excluded where the law still holds, so it errs conservative and never the other way. Where it does fire the estimate is not a number at all: two solutions sit in the box in 6–24% of replicates, both zeroing the objective exactly, and which one wins is decided by floating-point residue and the order the candidates are tried, the upper taking it 0.42–0.51 of the time.

Scope. The compression is a rule: derived, then confirmed against printed output at three cells and three audit sizes, 300 replicate pairs per row, the folded convolution read at four scales on each of those nine rows. The applicability test's firing rates and the two-solution figures beside it are observations at those same cells. A per-row rate carries about 0.03 of noise, which is why the law is argued from the form's own expectation and never from a single row. The two corrections have been measured one at a time: the predictive factor is the one that earns the gain, the change of spread is a smaller term whose sign follows the cell, and at one cell they pull against each other. What is measured is the per-row mean, which is the only sense of calibration available to a truth-free statement here: within a cell the rate conditional on the replicate in hand takes two values decided by D's sign, the statement reads |D| alone and is not independent of that sign, so a diagram binned on the statement must slope and does. A mixture over the two values is perfectly calibrated and needs the truth. Controls read before any of it: the marginal flip rate tracks the rates on record within 0.052 at every row, and the realized pair rate tracks its own pooled prediction within 0.039 at all nine. One audit tool, scalar features, one world family, one seed stream. Toy scale.

verifiers: explore_flip_risk.py, explore_flip_floor.py

The divergence is the family's, and not a flaw in one member that another member repairs. A statement whose tail decays like exp(−ax²/2) has, under the same noise, a mean decaying like exp(−aμ²/(2(1 + a))), and a/(1 + a) is below the event's own coefficient 1 for every finite a: estimating μ flattens every exponent in the family, so every member's ratio to the event diverges. What buys the exponent back is sharpening — driving the scale down — and that is a dial an auditor can turn without knowing the truth.

It is paid for at the other end. The sharpened statement, the pair form 2p(1 − p) read at p = Φ(−2μ̂), does not over-warn at any world row: its ratio to the event holds a band of 1.9× across the sweep, against 7.7× for the corrected statement. At the small-μ end the same sharpening states 0.205 against a realized 0.457, and the exponent is reached exactly only at scale 0, where the statement is 0 at every positive distance. No member is right at both ends, and the dial between the two ends is one number. The least-wrong member also fails toward confidence rather than away from it: all six of its ratios sit below 1, which is the shape and not the sweep — 2p(1 − p) is pinned at ½ where the event's own rate is ½, so sharpening can only push it down.

Scope. Rule: the exponent statement is derived, and the trade is read off the folded convolution, which is derived and confirmed above. The band figures beside it are observations at the six world rows — two cells at three audit sizes, 300 replicate pairs per row — and the designed cell extends nothing here, every ratio there being read against a model that cell breaks. One audit tool, scalar features, one world family, one seed stream. Toy scale.

verifier: explore_flip_floor.py

Every statement above reads a derivation. One instrument does not. Split-half cuts all three samples, runs the same estimator on each half, counts whether the two halves land on opposite sides of the midpoint, and pools over re-splits of the auditor's own samples — available to a single auditor the way a resampled spread is. Cutting only the audited sample would leave both halves sharing one G and one B, so they would stop being replicates of the audit and the count would stop estimating the event.

Three costs, and the third was not priced before the run. It answers at half the audit size, tracking the model at μ/√2 within 0.053 at every world row, so the bridge back to full size re-imports the parametric assumption the instrument was chosen to avoid. One split is a single yes-or-no, so only pooled rates read at all. And complementary halves are anticorrelated by construction — a high D in one half forces a low one in the other — which lifts the crossing rate above the half-size model by 0.027 to 0.053 at five of the six world rows and survives the pooling; the sixth sits 0.012 below, at the largest μ, where the rate itself is 0.066.

Bridged back, it beats the form the corrected statement had and does not beat the field. It is closer to the model than that statement at five of six world rows, with a ratio band of 0.85 → 2.71 against 0.66 → 5.11 — but it is the closest of the eight statements the run scored at only three of the six, losing the largest-μ row of each cell, and the sharpened statement's 0.42 → 0.80 holds the smallest worst case of anything measured. Its ratio still rises with μ at both world cells — the bridge is a smooth function of a noisy pooled rate, so the convolution reaches it too, and the divergence survives the move outside the family.

Where it wins is the designed cell, the one place no derivation holds: absolute error against the realized rate of 0.058, 0.000 and 0.016 across the three audit sizes, against 0.137, 0.090 and 0.043 for the corrected statement. Where the applicability test fires on nearly every replicate, the instrument that never read a derivation is the better one.

Scope. Observation: three cells at three audit sizes, 300 replicate pairs and 32 re-splits per row, one seed stream. The anticorrelation is a property of the construction; the claim that the divergence continues past the measured sweep is argued and not proved. The designed-cell verdict rests on the two smaller sizes, decided by gaps of 0.079 and 0.090 — at the largest the two errors differ by 0.027 against a realized rate whose own standard error is 0.015, and an independent measurement of that same cell and size reverses their order. The model column is not the truth at that cell, reading 0.001 against a realized 0.077 at the largest size, so a ratio there measures the derivation failing and not a statement over-warning. Halves with B exactly 0 are concentrated at the smallest audit size, where a half runs at 25, and are never filtered; dropping them moves the pooled rate by 0.001. One audit tool, scalar features, one world family. Toy scale.

verifier: explore_flip_floor.py

The per-example audit was run against the same world at the same target placements, the audit feature the same count of 2-moves, at shadow counts 8, 32 and 128 per class — the last its own published setting — with 400 replicates per cell. Its two outputs are the membership call, scored as balanced accuracy — accuracy averaged over equal-size member and non-member sets — and the stated membership probability, scored on calibration: its distance from the exact posterior, weighted by the audited fiber's own mass. They come apart.

The call is essentially exact. At shadow count 128 the fitted rule's balanced accuracy matches the exact best achievable accuracy to within 4·10⁻⁴ in every cell.

The stated probability is not, and more data does not cure it. The mean stated probability converges not to the exact posterior but to the hand-derivable limit of the Gaussian fits: overconfident at interior cells — 0.830 stated at the top shadow count where the posterior is 11/15 — and underconfident exactly where the truth is certain, 0.951 stated where the posterior is 1. The limit is deterministic, so the calibration error settles at a positive floor instead of vanishing. And the fraction of audited mass stated MORE extreme than its exact posterior rises with shadow data, 0.57 to 0.81 at one placement, because more data does not cure the miscalibrated limit — it concentrates every replicate onto it. The mechanism needs no worlds: on a designed two-atom feature the fit states 0.985 where the exact posterior is 0.9, a Gaussian tail pricing a far atom at its squared distance where the true distribution holds linear mass.

Scope. Observation: one audit tool of the per-example likelihood-ratio family, run as its own algorithm states — Gaussian fits to the two shadow-score populations — on scalar window features, one world family, one seed stream. A positive control with the Gaussian model well-specified calibrates and gates the run. The limit miscalibration is a fact about the estimator's model class on discrete features, reproduced ring-free on the two-atom cell.

verifier: explore_membership_ruler.py