Object store

Object storage is the only authority. Every published segment, every catalog generation, and every schema version lives there. Local disk beneath cache.dir is disposable: losing it may make the next request cold, but it cannot lose an indexed document or make an unpublished segment visible.

Object storage must preserve published data. Ingest spool durability also matters before publication; see ingest for what an acknowledgment promises.

The default is fs. A fleet running under the Kubernetes operator uses s3, because the fs backend points at a pod-local path that is not a shared authority.

For an object root ROOT, index articles, segment SEGMENT, and generation GENERATION, publication writes this shape:

ROOT/indexes/articles/
├── schema/
│   ├── latest
│   └── <version>.binpb
├── segments/SEGMENT/
│   ├── segment.pb                 immutable metadata envelope
│   ├── segment.commit             visibility marker, published last
│   ├── ids.bloom                  public-ID negative-proof sidecar
│   ├── ids.winners                exact ordinal/ID/version sidecar
│   ├── ids.ordinals               format-v5 dense ordinal→public-ID table
│   ├── terms.bloom                per-field term-presence sidecar
│   ├── filters.postings           low-cardinality exact-filter sidecar
│   └── wavesdb/
│       ├── MANIFEST
│       └── ...                    WavesDB CF table/blob checkpoint objects
├── catalogs/
│   ├── latest                     current generation hint
│   ├── publisher.lease            the publisher's compare-and-swap lease
│   ├── generations/GENERATION.pb
│   └── liveness/GENERATION.bin
├── compactions/<job>.pb
├── retired/<segment>
└── gc/...

Three properties of this layout matter operationally:

• segment.commit is written last. It is written only after the checkpoint, metadata, and sidecars are durable. A partially uploaded segment has no commit marker and is therefore invisible, not corrupt.
• Catalog generations refer to immutable segment commits. A compaction writes a new segment and a new generation and retires its inputs; it never modifies a visible segment in place.
• catalogs/latest is a hint, not the authority. It is updated by compare-and-swap. The generation lineage under catalogs/generations/ is what a reader trusts.

Segments and catalog explains why publication is ordered this way.

The inherited publisher holds its lease as an object, catalogs/publisher.lease, renewed by compare-and-swap (publisher.lease_ttl, 30 s by default). Only one publisher per index announces generations at a time. That makes the store's conditional writes part of the correctness argument: a provider that grants two writers the same conditional put can grant two publishers the lease.

In the ysearch design, leases and fencing epochs come from embedded Raft in the control role instead (decision D1, milestone YS5), so they no longer depend on the provider's conditional writes. That is design, not code: the object lease above is what runs today. Until YS5 lands, check that your provider's conditional puts are safe before running more than one publisher candidate.

Three settings decide where in a bucket an index lands, and they compose in this order:

<bucket> / <object.s3.prefix> / <object.root> / indexes/<index>/...

A deployment that owns its bucket can leave prefix empty and root at its default. A deployment sharing a bucket with other tenants should set prefix and give each tenant its own IAM policy scoped to that prefix.

The S3 credential settings are read from the environment only. They have no flag, and they never appear in a rendered command line or in a custom-resource status.

Under Kubernetes, the Secret named by YSearchCluster.spec.objectStoreSecretRef is attached to every role pod with envFrom. Its keys are ysearch environment variable names, which is why the Secret looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: object-store
  namespace: search
type: Opaque
stringData:
  YSEARCH_OBJECT_BACKEND: s3
  YSEARCH_OBJECT_S3_BUCKET: search-production
  YSEARCH_OBJECT_S3_ENDPOINT: s3.eu-central-1.amazonaws.com:443
  YSEARCH_OBJECT_S3_REGION: eu-central-1
  YSEARCH_OBJECT_S3_USE_SSL: "true"
  YSEARCH_OBJECT_S3_ACCESS_KEY: ...
  YSEARCH_OBJECT_S3_SECRET_KEY: ...

Rotating that Secret rolls the Deployments: the operator annotates every pod template with search.ysearch.io/secret-revision.

Set object.s3.anonymous=true to make unsigned requests to a bucket that allows public reads. Configure this explicitly for public datasets.

Every one of these is a startup setting. Changing one needs a restart; none of them accepts config set.

MinIO is the S3 implementation the repository's own test harnesses use, both in the Docker Swarm developer environment and in the kind operator qualification. Two settings usually differ from AWS:

object:
  backend: s3
  s3:
    endpoint: minio.search.svc:9000
    bucket: ysearch
    use_ssl: false
    path_style: true

path_style: true because MinIO addresses buckets by path rather than by virtual host, and use_ssl: false for a plaintext in-cluster endpoint. Set a region only if your MinIO is configured with one.

The kind harness runs a single digest-pinned MinIO Deployment behind a ClusterIP Service on port 9000, with /minio/health/ready and /minio/health/live as its probes. deploy/operator/minio-kind.yaml is the manifest; its credentials are fixture values for that harness only.

object:
  backend: s3
  s3:
    endpoint: s3.eu-central-1.amazonaws.com:443
    region: eu-central-1
    bucket: search-production
    use_ssl: true

The endpoint is a host:port pair, not a URL. Leave path_style at false.

Publication verification is checksum-based by default (builder.publish_verification: checksum), which compares every object against the checksum the store reported when it accepted the write. S3 computes that server-side and refuses a mismatched write, so nothing is transferred back. The sample and readback modes exist for stores whose checksum reporting you do not trust; readback costs the whole index again on every build.

object.source attaches a published dataset read-only, deriving the backend, bucket, prefix, root, and the index when the URL names one:

ysearch serve s3://search-data/production/indexes/articles/

A whole root works too:

ysearch serve s3://search-data/production/

Attaching a source forces server.read_only=true. Writes remain disabled for that process.

Each derived value is applied only where nothing more specific was already set, so an explicit --s3-endpoint still wins.

Use per-role credentials when your store supports separate policies. The table below summarizes each role's operations. ysearch does not validate policy scope; test restricted credentials with the deployment that will use them.

The default is safe here: gc.sweep_enabled is false, so nothing deletes objects until you turn it on. ysearch catalog gc-dry-run proposes unreferenced objects and removes nothing.

Two histograms report whole-process object I/O. Their sample count is the request total and their sample sum is transferred bytes:

• ysearch_storage_process_object_read_bytes
• ysearch_storage_process_object_write_bytes

The counting wrapper encloses the complete object store, so these totals include ingest publication, catalog access, compaction, cache fills, and query reads — not only request-scoped search traffic. Per-operation detail is in ysearch_storage_object_requests_total{object_operation,outcome}. See metrics.

• Observability — alerting on publication and GC
• Segments and catalog — why the layout looks the way it does