Pinned statistics epochs

The score of a (campaign, article) pair is computed at read time from statistics-free features and a statistics epoch that every node folds locally from the catalog. Consequences:

• a new epoch every day, or every generation, costs no S3 writes;
• stored matches never go stale;
• incremental matching and full regeneration produce the same scores.

bm25f-pinned-v1 is BM25F with the corpus statistics taken from an epoch E instead of from the segment being read:

tf~_E(t, d) = Σ_f  w_f · tf_f(t, d) / (1 − b_f + b_f · l_f(d) / avgl_E(f))
idf_E(t)    = ln(1 + (N_E − df_E(t) + 0.5) / (df_E(t) + 0.5))
g(x)        = x · (k1 + 1) / (x + k1)                          k1 = 1.2
S_E(c, d)   = Σ_{t ∈ q_c}  ω_{c,t} · idf_E(t) · g(tf~_E(t, d))

Only N_E, avgl_E(f), and df_E(t) depend on the epoch. Everything that depends on the article, its field lengths l_f(d) and term frequencies tf_f(t, d), is an integer feature that can be stored once and never changes.

A match therefore stores features, never a score:

The reader computes the score: one logarithm per campaign term per epoch, cached, then a few multiply-adds per match.

E(G) = Σ_{s ∈ segments(G)} stats(s)

• Additive. E(G+1) = E(G) + stats(new segments). A node never re-reads old segments. A compaction output carries the exact difference of its statistics, so a node applies the diff instead of re-reading.
• Every row counts. The sum includes rows that are dead in G. Liveness never rewrites statistics.
• Deterministic. Every node computes the same epoch from the same generation, with no coordination and nothing written to S3.
• Path-independent digest. stats_epoch_digest(G) hashes the generation's segment IDs and their statistics digests in segment-ID order. Two nodes with the same digest hold the same epoch, however they got there.

Each node keeps its epoch in a statistics database on SSD, with history records so an older epoch can be reconstructed after the fold has moved on.

Storing every (campaign, article) pair is impossible, so a match is stored only above a floor: the lexical score S ≥ F_c or the vector score v ≥ f_{v,c}, evaluated at the epoch of the evaluation. The question is whether an article that fell below the floor at one epoch could reach the serving threshold at a later epoch, when idf and average lengths have moved.

The drift bound answers it. For each campaign column, the dial controller tracks the minimum idf of every campaign term and the minimum average field length over the epochs since the column's baseline. From those it bounds how far any unstored article's score can have risen. If the bound stays below the lowest score the campaign could be served at, every article that should be served is stored. If not, the campaign's column is recomputed, and the dial reports DRIFT_RECOMPUTING until it is complete.

The bound needs two things: campaign term weights are non-negative, and every evaluation uses an epoch the controller has already checked.

G_chk is the newest catalog generation the dial controller has checked for drift. It is published in gossip next to the catalog heads.

• A node answering MATCH.ARTICLE pins E(min(G_node, G_chk)), where G_node is the generation it has folded. It never serves at an epoch the controller has not checked.
• The checked window is [G_chk − stats.max_serving_lag, G_chk]. A node lagging more than max_serving_lag stops answering MATCH.ARTICLE, and a healthy node scores for it. If no node is inside the window, the request fails with the typed, retryable error EPOCH_LAG. It never scores at an unchecked epoch.
• Matchers evaluate cells at G_eval = min(current generation, G_chk), and provisional matching on the fresh tail uses the same rule.

τ_min(c), the lowest cutoff at which campaign c's served set is exact, is the maximum over the checked window. The controller never writes a dial below it. See matching and dials.

A floor decided on one node and a threshold applied on another must agree bit for bit, on amd64 and arm64. In Go that takes these rules, enforced in internal/ysmath and by an AST check in the repository's tools/dev:

• Logarithms come from ysmath, a pure-Go port, never math.Log, whose amd64 assembly and arm64 Go versions differ by an ulp.
• No fused multiply-add. Every product in scoring code is written float64(x*y), which the Go specification guarantees is rounded, so the compiler cannot fuse it on arm64.
• Fixed order. Sums run over terms in (field, term_hash) order and over fields in field-number order.
• Integer inputs. Features are integers. A matcher app emits integers, and the host computes its score.
• One cosine. The cosine is the dot product of two canonical fp16 vectors, summed in component order.

A golden test pins the score bits of bm25f-pinned-v1. Running it on both architectures needs the arm64 CI runner of YS1, which does not exist yet. A test checks that the pinned scorer's top-k is identical whether a corpus sits in one segment or fifty.