One database for news search and ad matching

ysearch is one Go database system for a news and ad-matching workload. It keeps a large daily volume of articles, their traffic, and the advertising campaigns that want to run next to them. For every article, it answers one question fast: which campaigns match it right now?

It is a fork of the yolosearch search engine with a second storage plane added: a replicated, convergent key-value store whose design is ported from marekvs.

The segment plane is on S3. Immutable index segments, match results, and cold documents live in object storage. Local SSD holds only metadata and caches, so a query node owns nothing it cannot rebuild. This part runs today: it is yolosearch's engine. See segments and the catalog.

The KV plane is on SSD. Documents, per-article traffic, campaigns, reach dials, and the hot part of the match tables live in an eventually consistent store replicated three ways. Every record type merges as a CRDT, so replicas converge without coordination. Its building blocks are on main; the plane itself is milestone YS5. See the KV plane.

A match stores statistics-free features, such as term frequencies, field lengths, and cosines, never a finished score. Every node folds the same statistics epoch from the catalog. A reader pins one epoch and computes the score on the spot.

Epochs can therefore change every day, or with every new catalog generation, without rewriting a single stored match. A drift bound keeps the stored set exact as statistics move. See pinned statistics epochs.

Each campaign has a dial (α_c, τ_c): α_c weighs the lexical score against the vector score, and τ_c is the cutoff. The dial is read when the ad server asks, so widening or narrowing a campaign's reach rewrites nothing. A controller can set τ_c from a threshold, a count, an impression forecast, or a pacing goal, and never below the lowest cutoff at which the answer is still exact. See matching and dials.

New articles arrive from Kafka-protocol topics as protobuf records. The target is under 30 seconds from write to matched: the partition's owner indexes each new article in memory and matches it provisionally against every campaign, long before its segment reaches S3. See streams.

• gRPC and RESP only. There is no HTTP API and no SQL. Prometheus metrics and health probes are the only plain-HTTP surface.
• WASM apps, not Lua. Write scripts, pipeline processors, and custom matchers are WebAssembly components against ysearch-defined interfaces. A matcher app emits integer features, and the host computes the score. See extensions.
• Deterministic scoring. Scores are bit-identical on amd64 and arm64, so a floor decided on one node and a threshold applied on another always agree.

The inherited engine builds, ingests, and serves lexical and vector search over S3-compatible storage or a local directory:

just build
./bin/ysearch-local

The quick start takes it from a source checkout to a served query. The guides cover schemas, ingest, both query grammars, vectors, and the local development stack, which already includes Redpanda and the marekvs sidecar the matcher prototype uses.

• Status and roadmap: what exists, what is next, milestone by milestone
• Why ysearch: the workload and the design choices it forces
• Architecture: planes, roles, and data flows
• Quick start: build, load, and search locally