Local development

There are two local setups, and the choice between them is about what you need to observe.

One binary. ysearch-local runs the complete data plane plus the admin UI in a single process, with no flags required. Use it for schema work, ingest, queries, and anything that does not depend on more than one node.

A Docker Swarm fleet. just swarm-up deploys a coordinator, two workers with separate caches, a writable server, an aggregator, a compactor, MinIO, the admin console, and — since milestone YS1 — a single-node Redpanda broker and a marekvs sidecar. Use it to exercise worker placement, remote execution, cross-node failures, and the stream and sidecar pieces of the matcher prototype.

Testing placement, remote workers, and cross-node failure requires separate node processes. Swarm and Kubernetes provide deployment setups for these tests.

A third set of tools builds the development corpus and its synthetic feeds, and checks the YS1 infrastructure; see the YS1 fixtures and corpus.

Go 1.27 or newer, Node.js and npm, just, and zsh. The Swarm path adds Docker Engine or Docker Desktop with a running Linux-container daemon. The disposable fixtures (Redpanda, marekvs, MinIO) run on Apple's container runtime by default, or on podman with YSEARCH_CONTAINER_RUNTIME=podman.

From the repository root:

just build
./bin/ysearch-local

Or compile and run in one step with just local. Once the ready block appears, open http://127.0.0.1:8787:

ysearch-local: ready
  data:  127.0.0.1:9500
  admin: http://127.0.0.1:8787
  root:  <cwd>/ysearch-local-data
  objects: <cwd>/ysearch-local-data/objects
  cache:   <cwd>/ysearch-local-data/cache
  ingest:  <cwd>/ysearch-local-data/ingest
  models:  <cwd>/ysearch-local-data/models

The native gRPC API is 127.0.0.1:9500, which is the address the ordinary ysearch client already dials. In another terminal:

./bin/ysearch schema apply deploy/swarm/articles.proto
./bin/ysearch push articles deploy/swarm/articles.jsonl
./bin/ysearch search articles search --fields title,url
./bin/ysearch stats articles

One OS process containing the production direct multi-namespace data path, the builder, the publisher, the follower and cache, the idle-aware compactor, the optional GC scheduler, the embedding runtime, the native gRPC server, and the admin HTTP/Connect server. The admin bridge still calls the public gRPC API; there is no local-only search adapter, so what the console does here is what it does everywhere.

The default persistent layout:

ysearch-local-data/
├── config.yaml   optional configuration
├── objects/      authoritative schemas, segments, commits, and catalogs
├── ingest/       durable spool and build scratch
├── cache/        disposable segment/block cache
└── models/       checksum-addressed internal embedding models

Data persists across restarts; restarting the same command reopens the same schemas and documents. Ctrl-C or SIGTERM drains both endpoints and joins every background goroutine. Logs go to the inherited stdout and stderr only; the process creates no hidden log file.

Settings work exactly as configuration describes — flags, YSEARCH_* variables, ysearch-local-data/config.yaml, or the runtime config API. Only server.data_dir has a local command default, so every explicit setting wins. Secrets remain environment-only.

./bin/ysearch-local --data-dir ./scratch/search \
  --listen '[::1]:9500' --admin-listen '[::1]:8787'

Local mode refuses S3 as authoritative storage. Internal model packages live under models/; external HTTP and gRPC embedders use the same embedding.* settings as ysearch serve, so vector search works here. A provider being unavailable affects the requests that need it and does not prevent lexical-only startup.

The idle-aware compaction scheduler is available in-process: set compaction.enabled=true when the local process should run it. Safe GC stays off unless gc.sweep_enabled=true, and enabling it retains the existing dry-run, quarantine, and fresh reachability proof.

Stop the process and run this guarded, path-resolved reset from the repository root:

local_root="$(realpath ./ysearch-local-data)"
test "$local_root" = "$(pwd -P)/ysearch-local-data" && rm -rf -- "$local_root"

--dir turns a directory into a self-contained installation and runs it:

./bin/ysearch-local --dir ~/search/articles
# ysearch-local: created installation /home/you/search/articles
# config: loaded /home/you/search/articles/ysearch.yml (--config)
# ysearch-local: ready

It creates the directory and objects/, cache/, ingest/, and models/ beneath it — the same names --data-dir derives — so the layout exists and is inspectable before anything starts. When the directory holds no configuration file yet (ysearch.yml, ysearch.yaml, or config.yaml), it writes ysearch.yml with the object backend, the four directories, the listen address, and the log level. A file that is already there is never rewritten: --dir means "run this installation", not "reset it".

The generated file spells its directories relative to itself, and a relative path in a configuration file resolves against that file's directory. So the installation moves:

mv ~/search/articles /media/stick/articles
./bin/ysearch-local --dir /media/stick/articles   # nothing to edit

--dir and --data-dir both name a root, so giving both is an error. An explicit --config still wins over the installation's own file, and every flag still outranks both.

--help does not print every catalog flag. It opens on an ESSENTIALS block — the dozen flags that decide where the data lives, what is listening, and what is switched on — followed by an index of configuration topics and how many flags each holds:

./bin/ysearch-local --help            # essentials and the topic index
./bin/ysearch-local --help cache      # one topic, every flag in it
./bin/ysearch-local --help all        # every topic

Each topic screen shows the flag, its first sentence of documentation, its constraint, its default, and the catalog key to write in ysearch.yml. The full paragraph for every setting is in ysearch manual and the CLI reference. The screen wraps to the terminal's width between 60 and 120 columns and is colored when it is going to one — NO_COLOR, TERM=dumb, a pipe, or --color never each turn that off, and --color always forces it on through a pager.

These flags are specific to this binary, beside the --admin-upload-* limits described in the admin console:

If port 8787 or 9500 is taken, the process reports the conflicting address and exits. Choose others with --admin-listen and --listen, using bracketed IPv6 literals where needed.

just swarm-up

That is the whole bootstrap. The first run downloads dependencies and container base images, builds the current checkout, initializes a single-manager Swarm if Docker is not already in Swarm mode, deploys the fleet, creates an articles namespace, indexes six sample documents, and verifies a distributed query. It also builds the marekvs sidecar image into the Docker daemon when no image with the pinned revision label is present, and reports ready only after a Kafka produce-and-read-back on articles and a RESP SET/GET succeed from the host.

When it returns:

./bin/ysearch --server 127.0.0.1:9600 \
  search articles 'title:search' --fields title,url

The MinIO login is ysearchdev / ysearch-dev-only-secret. These are public credentials for local development.

The single-process write path is on 9500 and the distributed read path is on 9600. All query workers read the same authoritative object catalog and do not own shards permanently. The aggregator improves placement but is not a correctness dependency — the coordinator can fall back to its explicit worker list. See architecture.

Application roots run read-only as UID/GID 65532. MinIO, Redpanda, and marekvs data use normal named volumes; builder scratch and query caches use bounded tmpfs-backed named volumes, so they may be cold after a stack stop or a Docker restart.

just swarm-status           # services, task placement, endpoints
just swarm-logs             # last 200 timestamped lines from every service
just swarm-logs coordinator # one allowlisted service
just swarm-down             # stop services; preserve MinIO object data
just swarm-up               # rebuild current source and resume or reseed

swarm-up is idempotent. Reapplying the same schema is a no-op and pushing the same six keys creates newer versions that normal searches collapse by key. Existing services are force-refreshed, so rebuilding a local image under the same development tag rolls the tasks.

To delete the local stack and its data:

just swarm-reset

swarm-reset removes the ysearch-dev stack and only its eight exact named volumes. It does not remove images, unrelated stacks, networks, or other volumes, and it never runs docker swarm leave. Run just swarm-up afterward for a fresh namespace and cold caches.

No lifecycle driver detaches a helper shell, log tail, or find process. Every readiness loop is finite and attached to the command. If startup fails, the stack is left in place for diagnosis instead of being erased.

./bin/ysearch --server 127.0.0.1:9500 schema apply deploy/swarm/articles.proto
./bin/ysearch --server 127.0.0.1:9500 push articles deploy/swarm/articles.jsonl

./bin/ysearch --server 127.0.0.1:9600 \
  search articles 'search AND body:cache' --fields title,site,url
./bin/ysearch --server 127.0.0.1:9600 \
  search articles '[word="cold"] []{0,3} [word="cache"]' --dialect cqp --fields title,url

The workers and coordinator follow the latest catalog generation every second, so a published write becomes searchable through port 9600 without restarting the fleet.

Corpus and fleet statistics:

./bin/ysearch --server 127.0.0.1:9600 stats articles
./bin/ysearch --server 127.0.0.1:9600 stats --node
./bin/ysearch --server 127.0.0.1:9600 stats --fleet

The sandbox's non-secret settings are in deploy/swarm/stack.yml. Change cache budgets, builder memory and concurrency, seal thresholds, follower cadence, storage mode, or compaction thresholds there, then rerun just swarm-up. The stack disables configuration discovery so a host's files cannot leak into containers, and its flags override shared environment settings.

Startup-scoped settings require editing the stack and rolling the service; node-runtime settings can also use the ephemeral overlay:

./bin/ysearch --server 127.0.0.1:9500 config get ingest.seal_age
./bin/ysearch --server 127.0.0.1:9500 config set ingest.seal_age 5s

For the direct-server split, compile and run serve on its own:

just build
./bin/ysearch serve --data-dir ./ys --listen 127.0.0.1:9500

Then, in another terminal, the same schema, push, and search commands. This form has no co-located admin UI. Prefer ysearch-local unless you specifically need to exercise ysearch serve alone. Stop it with Ctrl-C; nothing is detached. ./ys holds the filesystem object store, ingest scratch, cache, and derived config path.

Common contributor commands:

just test-unit       # deterministic Go tests
just ui-test         # React tests and TypeScript checking
just bench-short     # quick diagnostic microbenchmark
just ci              # infrastructure-free CI gate
just gate            # CI plus compatibility and integration proofs

Laptop benchmark numbers are diagnostic. Reportable resource and performance receipts run through the repository's remote benchmark commands.

Milestone YS1 added the infrastructure the matcher prototype needs before any ysearch feature is built. These pieces are development and test tooling. None of them is part of a deployment, and the prototype pieces are marked throwaway in the code: the KV plane (YS5) and the stream source (YS7) replace them. See status for where each milestone stands.

tools/dev starts and stops pinned containers by name. A leftover of the same name is removed before each start, and readiness is polled against a deadline:

go run ./tools/dev redpanda-up          # single-node Redpanda, Kafka on 127.0.0.1:19092
go run ./tools/dev redpanda-down
go run ./tools/dev marekvs-image        # build tcr.teixos.net/yannick/marekvs:5df6ff5
go run ./tools/dev marekvs-up --nodes 3 # a 1- or 3-node marekvs cluster
go run ./tools/dev marekvs-down
go run ./tools/dev minio-up
go run ./tools/dev minio-down

marekvs-image builds the sidecar from git archive exports of marekvs 5df6ff5 and the storage-engine revision it pins, into a FROM scratch image that carries a revision label. The fixtures refuse an image whose label is not 5df6ff5. A marekvs cluster counts as ready only when every node reports the full cluster membership.

fixtures-integration runs the fixture proofs under the integration build tag: Redpanda produce and consume, marekvs on one and three nodes, and a corpus replay into Redpanda. The tests skip unless YSEARCH_CONTAINER_FIXTURES=1 is set, because the CI runner has no container runtime.

go run ./tools/dev fixtures-integration

tools/corpus has four subcommands that build a reproducible corpus and the synthetic feeds derived from it. The records are ysearch.fixture.v1 protobuf messages (api/ysearch/fixture/v1), keyed by the entity they count. They are fixtures, not the production contract. Every output is deterministic for a given input and seed on every architecture.

The popularity and decay parameters of beacons, the campaign model, and the default partition count of 6 are assumptions. The manifests say so.

go run ./tools/corpus sample --input articles.jsonl --output ~/.ysearch/corpus/ys1-3m
go run ./tools/dev redpanda-up
go run ./tools/corpus replay --corpus ~/.ysearch/corpus/ys1-3m --brokers 127.0.0.1:19092
go run ./tools/corpus campaigns --corpus ~/.ysearch/corpus/ys1-3m --output ~/.ysearch/corpus/campaigns-10k
go run ./tools/corpus beacons --corpus ~/.ysearch/corpus/ys1-3m --brokers 127.0.0.1:19092

Without --brokers, replay and beacons are dry runs. A missing topic is created with --partitions partitions and a 16 MiB message limit. The benchmark article corpus that sample reads is not distributed with the repository; point --input at your own JSONL of articles.

go run ./tools/dev infra-check          # evidence gathered now
go run ./tools/dev infra-check --live   # also runs the fixture proofs
go run ./tools/dev infra-check --strict # fails unless every row passes

infra-check reports each row of the YS1 infrastructure table as pass, fail, or not configured. A row passes only on evidence gathered during the run. Rows that need a staging cluster, the target S3 provider, CI runners, bench hosts, or GPU hosts report "not configured" with the ticket that would configure them, and never pass. A failing row fails the verb; --strict also fails on a row that is not configured. --json PATH writes the report with host facts.

ysearch-prototype adapter is the matcher prototype's throwaway feed. It consumes ysearch.fixture.v1.Article records from a Kafka-protocol topic and writes each one twice: first to the marekvs sidecar as JSON with a stand-in document version, then to the ordinary ingest path of an index with the same version. It is its own binary so that the sidecar's RESP client never links into the server.

Every flag is a prototype.* key of the configuration catalog, also settable as a YSEARCH_PROTOTYPE_* variable or in a file. The defaults dial 127.0.0.1:9092 for the broker and 127.0.0.1:6379 for the sidecar, so against the Swarm stack name the published ports:

./bin/ysearch-prototype adapter \
  --prototype-adapter-brokers 127.0.0.1:19092 \
  --prototype-sidecar-address 127.0.0.1:16379 \
  --prototype-adapter-ingest-server 127.0.0.1:9500 \
  --prototype-adapter-index articles

The target index must declare at least one of the article fields (url, title, content, excerpt, author, lang, site, publisher_id, pub_time_ms, source_id). Its key field receives the article id, so the key must not itself be an article field: the Swarm sample schema, keyed by url, is refused, and a schema keyed by a field such as doc_key is accepted. Article fields the index does not declare go to the sidecar only.

The adapter commits a partition's offset only up to the last contiguously applied record, stops a partition on a record it cannot apply rather than skipping it, and retries sidecar and ingest failures. The sidecar password is read from YSEARCH_PROTOTYPE_SIDECAR_PASSWORD only.

failed to connect to the docker API. The CLI exists but the daemon is not running, or the active context is wrong:

docker context show
docker info

Swarm is locked, pending, or belongs to another cluster. swarm-up initializes only an inactive daemon and reuses only an active one. It will not repair or leave another Swarm:

docker info --format '{{.Swarm.LocalNodeState}}'
docker node ls

Neither swarm-down nor swarm-reset runs docker swarm leave.

A public port is in use. Ports 8787, 9001, 9500, 9600, 16379, and 19092 must be available to Swarm. Change the corresponding published value in deploy/swarm/stack.yml and use the new address manually — the lifecycle acceptance probes use the default ports, so changing them also means changing the constants in tools/dev/swarm.go.

A service keeps restarting.

just swarm-status
docker service ps --no-trunc ysearch-dev_server
just swarm-logs server
just swarm-logs minio-init

Catalog-bound workers, the coordinator, and the compactor can restart briefly before MinIO and the sample namespace exist. A persistent restart loop usually means the bucket initializer failed, object credentials disagree, a cache mount is not writable, or the local image architecture is wrong.

Search says the index or generation is unavailable. Confirm seed publication on the write endpoint, then allow one follower poll:

./bin/ysearch --server 127.0.0.1:9500 stats articles
./bin/ysearch --server 127.0.0.1:9600 stats articles
just swarm-logs coordinator

A rebuilt service still runs old code. Use just swarm-up, not a raw docker stack deploy. The lifecycle command force-refreshes existing application services after rebuilding the fixed local tags; confirm task creation times with just swarm-status.

Images fail on another Swarm node. Local image tags exist only in one Docker daemon. Push immutable multi-platform images to a registry, authenticate every node, set YSEARCH_SWARM_IMAGE and YSEARCH_SWARM_ADMIN_IMAGE, and deploy with --with-registry-auth. Also replace the node-local volume strategy before treating the deployment as durable — the default placement constrains tasks to manager nodes and every named volume uses node-local storage, which is what makes this a developer topology rather than a highly available one.

Full clean start.

just swarm-reset
just swarm-up

• Quick start — the shortest path to a served query.
• The admin console — what the UI on 8787 can do.
• Configuration — precedence, reload, and scope.
• Containers and Kubernetes — the deployment forms beyond a sandbox.