Segments and the catalog

A segment is the unit of immutability, publication, caching, and compaction. Physically it is one WavesDB checkpoint: each column family in its own table and blob files, so a query reads only the physical files it needs. A reader opens a checkpoint remotely, fetching the manifest once and leaving tables and blobs on S3, read by range requests and cached locally.

The inherited families are meta, terms, postings, positions, impact, filters, docvalues, vectors, and stored. ysearch adds families without changing the old ones, and old segments stay readable:

Format	Adds	Milestone
7	stats (per-term document frequencies and total term frequencies, the input of the epoch fold), forward (per document, its sorted (term_hash, tf) pairs and field lengths), hashterms (term hash back to field and term)	YS2, on main
8	exact vector lanes: canonical fp16 vectors, IVF centroids with radii, int8 codes with residual norms, for cosine fields	YS3, on main

The builder writes format 8 by default. Readers accept formats 1 through 8.

stats is what makes pinned statistics epochs cheap: a node folds a new segment's statistics without reading its postings. forward is what makes matching cheap: a matcher percolates every campaign against each article's term list without an inverted-index lookup per term.

A builder constructs a segment on local disk, uploads the checkpoint and its sidecars into a unique object prefix, verifies them, and writes segment.commit last. Object keys are never overwritten during normal publication, so a retried build either verifies the same bytes or fails as a collision. Catalog visibility is the cross-family transaction: a segment without a commit marker is invisible.

A segment never changes after commit. Compaction writes a new segment and a new generation, and retires its inputs.

A catalog generation is an immutable record naming the complete active segment set. One publisher writes a linear lineage: it uploads the generation object first and updates latest second. latest is a discovery hint that readers validate, not the authority.

A query pins one generation for its lifetime. The segment set it planned against cannot change underneath it, and the segments themselves are immutable, so a long result stream stays coherent.

Local disk is disposable. Losing a cache makes the next request cold; it cannot lose an indexed document or make an unpublished segment visible.

The specification keeps this model and extends it:

• Namespaces. Every namespace has its own catalog chain, statistics epoch, segments, campaign batches, and match tables under <prefix>/ns/<namespace>/.
• Fencing epochs in object names. A generation is written as catalog/gen/<G>-<F>.pb, where F is the publisher lease's fencing epoch, committed in the namespace's Raft group. Readers reject a lower epoch. Every object except catalog/latest is write-once, and readers never list a prefix.
• Liveness bitmaps. A generation names one liveness bitmap per segment. Liveness never rewrites statistics.
• Locators. For every document, a KV record names the segment, ordinal, and version that hold it. The KV plane's current version v(d) is the authority for which indexed version is current.
• Submit order. The publisher's steps are fixed: liveness, then the generation and latest, then locators, then the reply.
• Match cells and tables. Matchers write content-addressed cells, and the table writer builds by-article and by-campaign tables, all as WavesDB checkpoints on S3. See matching.
• Compaction carries statistics. A compaction output carries stats-diff.bin, the exact difference of its statistics from its inputs', so a node can apply the diff instead of re-reading.

Today's publisher lease is an object on the store renewed by compare-and-swap. In the design it moves to the control role's Raft groups (YS5), because some S3 providers' conditional writes are not safe as a lock. Object store says what the inherited engine needs from a bucket today.