Kubernetes

The operator turns two custom resources into a running fleet. A YSearchCluster holds what every index in a namespace shares — the image, the object-store Secret, the format compatibility window, monitoring, and cluster-wide per-role floors. A YSearchIndex holds one index: its schema, its replica counts per role, its own config, and its deletion policy.

Everything else — Deployments, Services, ConfigMaps, Jobs, PodDisruptionBudgets, HorizontalPodAutoscalers, NetworkPolicies, and optionally a ServiceMonitor — is rendered by the operator and owned by it.

kubectl apply -f deploy/operator/crds/
kubectl apply -f deploy/operator/operator.yaml
kubectl apply -f deploy/operator/sample.yaml

operator.yaml creates the ysearch-system namespace, a ServiceAccount, a ClusterRole and binding, a two-replica Deployment running /ysearch-operator with --leader-elect=true, a metrics Service on port 8080, and a PodDisruptionBudget. The operator container runs with a read-only root filesystem, all capabilities dropped, and an emptyDir at /tmp.

Replace ysearch:latest in operator.yaml and in your own cluster resource with an immutable image digest before applying either.

Both have a status subresource and print a Ready column; YSearchIndex also prints its Cluster.

The same override shape is accepted at cluster and index level, keyed by one of the eight role names — coordinator, worker, merger, aggregator, router, builder, publisher, compactor.

cacheVolume.sizeLimit is validated against the role's own resources.limits.ephemeral-storage. A cache larger than the pod may occupy is a potential cause of eviction before the emptyDir limit applies, so the operator rejects that configuration. A medium: Memory cache is charged against the memory limit instead.

1. It starts a temporary all-in-one bootstrap server and a content-addressed schema Job.
2. Only after that Job succeeds does it create the distributed role topology.
3. The bootstrap remains a query endpoint until every distributed role is available, then it is removed.

The Job's identity is derived from the schema ConfigMap's content, so an additive schema update is applied exactly once and an incompatible update fails in the Job with an error.

Both Services are dual-stack (ipFamilyPolicy: RequireDualStack) and expose gRPC on port 9500.

Every role pod is the same image with different arguments:

/ysearch node --roles <role> --listen :9500 --index <index>

Probes all use the gRPC health service on port 9500, with a 2-second timeout:

A startup budget of 60 failures at a 2-second period gives a cold process two minutes to resolve its catalog before the kubelet restarts it. Catalog-bound roles resolve their generation before the listener opens, so a node that cannot read its catalog fails at startup rather than accepting work it cannot perform.

Settings reach the container as environment variables. The object-store Secret is attached with envFrom, and the operator annotates the pod template with search.ysearch.io/secret-revision so a Secret rotation rolls the Deployments.

This is deploy/operator/sample.yaml, shortened (the repository copy also sets adminConsole and carries comments), with its placeholders intact. Replace every replace-me and the :latest tags before applying.

apiVersion: v1
kind: Secret
metadata:
  name: object-store
  namespace: search
type: Opaque
stringData:
  YSEARCH_OBJECT_BACKEND: s3
  YSEARCH_OBJECT_S3_BUCKET: replace-me
  YSEARCH_OBJECT_S3_ENDPOINT: replace-me:443
  YSEARCH_OBJECT_S3_USE_SSL: "true"
  YSEARCH_OBJECT_S3_ACCESS_KEY: replace-me
  YSEARCH_OBJECT_S3_SECRET_KEY: replace-me
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: articles-schema
  namespace: search
data:
  schema.proto: |
    syntax = "proto3";
    package example;
    import "ysearch/v1/schema.proto";
    message Article {
      option (ysearch.v1.document) = { index: "articles" };
      string url = 1 [(ysearch.v1.field) = { key: true }];
      string title = 2 [(ysearch.v1.field) = { indexed: true, stored: true }];
      string body = 3 [(ysearch.v1.field) = { indexed: true }];
    }
---
apiVersion: search.ysearch.io/v1alpha1
kind: YSearchCluster
metadata:
  name: production
  namespace: search
spec:
  image: ysearch:latest
  imagePullPolicy: IfNotPresent
  objectStoreSecretRef: {name: object-store}
  compatibility: {readFormatMin: 1, readFormatMax: 4, writeFormat: 4}
  defaultConfig:
    storage.mode: AUTO
    cache.full_bytes: 8GiB
    cache.block_bytes: 2GiB
  monitoring:
    serviceMonitor: Auto
    interval: 30s
    scrapeNamespaceSelector:
      matchLabels: {kubernetes.io/metadata.name: monitoring}
  roleOverrides:
    worker:
      resources:
        requests: {cpu: "2", memory: 8Gi}
        limits: {memory: 16Gi, ephemeral-storage: 48Gi}
      cacheVolume: {sizeLimit: 32Gi}
---
apiVersion: search.ysearch.io/v1alpha1
kind: YSearchIndex
metadata:
  name: articles
  namespace: search
spec:
  clusterRef: production
  schemaRef: {name: articles-schema, key: schema.proto}
  deletionPolicy: Retain
  replicas:
    coordinators: 2
    workers: 3
    routers: 2
    builders: 2
    publishers: 1
    compactors: 1
  roleOverrides:
    worker:
      nodeSelector: {node.kubernetes.io/instance-type: search-optimised}
      tolerations:
      - {key: dedicated, operator: Equal, value: search, effect: NoSchedule}
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            search.ysearch.io/index: articles
            search.ysearch.io/role: worker

Note the relationship the sample encodes in the worker override: cacheVolume.sizeLimit of 32Gi holds cache.full_bytes (8GiB) plus cache.block_bytes (2GiB) with room to spare, and stays under the 48Gi ephemeral-storage limit. The cache budgets and their defaults are in the configuration reference.

spec.compatibility is the upgrade safety mechanism. It declares the segment format range this cluster's readers accept and the format its writers emit:

compatibility: {readFormatMin: 1, readFormatMax: 4, writeFormat: 4}

writeFormat is pushed to builder.output_format_version on the writer roles, so it controls real builder and compactor output rather than being an annotation. A transition is refused when either side would break:

• the desired readers cannot read the format currently being published, or
• the desired writer format is outside the currently active reader range.

The practical consequence is that widening readers and advancing writers are two separate rollouts. Roll out readFormatMax: 5 first and wait for it to be everywhere; only then set writeFormat: 5. A resource that tries both at once reports UpgradeBlocked.

Retain is the default and never touches object data.

Delete is a sequence, and each step is a precondition for the next:

1. Remove the index's owned Deployments and Jobs.
2. Wait until every labelled Pod has terminated.
3. Delete, and then verify, only the index's validated publication prefix.

It supports the shared s3 backend and reads the endpoint, bucket, optional prefix, object root, and credentials directly from objectStoreSecretRef. Missing or invalid settings and object-store failures leave the finalizer in place and report DeletionBlocked rather than proceeding.

A cluster-protection finalizer also refuses to remove a YSearchCluster while any YSearchIndex still references it, so the Secret reference and controller configuration each index needs to finish its own deletion policy survive.

Set spec.adminConsole on the cluster:

adminConsole:
  image: ysearch-admin:latest
  imagePullPolicy: IfNotPresent
  indexes: [articles]

The operator creates a dedicated ServiceAccount, a Deployment with two replicas by default, a ClusterIP Service, a read-only target ConfigMap, and a restrictive NetworkPolicy. Console pods run as UID/GID 65532 from a scratch image with a read-only root filesystem and no automounted service-account token, and may dial only the <index>-admin:9500 upstream Services named in indexes. Omit indexes to select every index the cluster owns.

The console container is told --listen 0.0.0.0:8080, and its Service publishes port 8080. Its probes are HTTP rather than gRPC: /livez for startup and liveness, /readyz for readiness.

spec.monitoring.serviceMonitor takes three values:

scrapeNamespaceSelector names the namespaces allowed through the rendered NetworkPolicies to the metrics port, defaulting to the namespace literally named monitoring. It is matched on the namespace alone — combining it with a pod selector in one peer would AND the two and deny every scraper that does not also carry the pod labels.

deploy/operator/monitoring.yaml adds an optional ServiceMonitor and PrometheusRule for the operator process itself. It requires the Prometheus Operator CRDs. See observability for the data-plane rules.

just operator-manifests
just operator-manifests-check
just operator-envtest
just operator-kind

operator-envtest downloads a pinned Kubernetes 1.33 envtest control plane on first use, then exercises bootstrap, readiness, compatibility status, Retain, and fail-closed and retried Delete against a real API server.

operator-kind needs a running Docker daemon. It downloads pinned kind (v0.32.0) and kubectl (v1.36.1) into .cache/tools on first use. It builds and loads the local image, creates a disposable three-node cluster and a digest-pinned MinIO instance, then runs install, scale, worker-pod and worker-node loss and recovery, image upgrade and rollback, an incompatible format rejection, Retain, and a storage-outage-blocked then retried Delete. On failure it exports cluster logs under bench/receipts/kind-operator-logs; set YSEARCH_KIND_KEEP=true to keep the cluster for inspection.

• Object store — what goes in objectStoreSecretRef
• Observability — what to scrape and what to alert on
• Configuration reference — the cache budgets behind cacheVolume