Streams

Every production stream arrives as protobuf records on a Kafka-protocol topic: articles, ad actions (impressions and clicks), and bid data. Any Kafka-protocol broker works; Redpanda is the tested one. JSON and raw bytes remain for other deployments and for tests.

• A protobuf decoder is compiled once per descriptor version and maps fields by number, so a producer can add fields without breaking a source.
• Unknown fields in a broker record are ignored, because protobuf evolution depends on it. Unknown fields in a request are still rejected.
• There is no file ingestion path for these feeds. A bulk load is a replay of a topic from an earlier offset.

Kind	Content	Lands in	Idempotent under replay by
documents	articles, or any namespace's documents	DOC.PUT and DOC.DEL through the write pipeline	versions derived from offsets
traffic	beacon events: article, metric, time, visitor hash, count	traffic slots	exactly-once counting per slot
campaigns	campaign records from an upstream ad system	the campaigns namespace	versions derived from offsets
dials	reach targets pushed by an upstream system	dial records	versions derived from offsets
ad_actions	pair-level ad events: campaign, article, placement, site, publisher, action, and the served score	campaign-keyed counters	exactly-once counting per slot

Topic names and partition counts are deployment configuration. Every topic is keyed by the entity it writes or counts: articles and beacons by article, ad actions by campaign.

In yolosearch a source committed offsets only after catalog publication, because segments were the source of truth. In ysearch the KV plane is the source of truth, so a source commits once its writes are replicated there, by default on two of three owners. It does not wait for indexing.

• A source commits last contiguously acknowledged + 1 per partition, and never past a gap.
• An offset that is out of range stops the partition visibly. It never skips to the latest offset.
• Broker retention must exceed the worst processing lag.

A document written from a topic gets an external version derived from its partition and offset. Replaying the topic writes the same versions, which the KV plane recognises as already applied. A record never goes backwards because of a replay.

Counters cannot use versions, because every event adds. A traffic source counts each partition into partition-scoped slots that merge by the larger offset. A zombie consumer that replays a range writes values the slot already holds, so every event is counted exactly once, including the pair-level counters that click calibration reads.

The owner's feeds are all streams. Keeping one ingestion path means one set of commit rules, one poison-record policy with dead-letter topics, and one place to measure lag.