Observability

Every ysearch process serves its own Prometheus exposition. Metric families use bounded cardinality: labels are closed enumerations, and index names, query text, document identities, object keys, and target addresses never become labels. A registry that would violate that is refused at construction — the process uses a pedantic registry and validates every gathered family before it serves.

That is a design constraint with an operational consequence. You will not find a per-index query rate in Prometheus. A host serving several indexes reports one series per role and sums its gauges; per-index detail lives in the admin console and in durable receipts.

It is served by ysearch serve, ysearch node, ysearchd, and ysearch-local. Unlike debug.pprof_listen, it is on by default and binds every interface. Profiles expose heap contents and goroutine stacks; an exposition of counters and histograms carries no document text, no query, and no key material. A scrape target that has to be switched on is one nobody switches on.

A bind failure is reported on stderr and survived. Telemetry is not a precondition for serving queries, so an occupied scrape port must not turn an observability problem into an availability one.

Every series carries role as a constant label, and its value is the role mix the process serves, not a single role. node --roles builder,publisher reports role="builder-publisher".

This is why the shipped rules match {role=~".*publisher.*"} rather than role="publisher". Under the Kubernetes operator each Deployment runs exactly one role, so the values are the plain role names there.

instance and pod are not emitted by the process and must come from the scrape target. The leader-lost alert aggregates with max without(instance, pod) (...), so a scrape config that does not supply them will not evaluate correctly.

Under the operator, set spec.monitoring on the YSearchCluster and let it render the ServiceMonitor and the NetworkPolicy peer together:

monitoring:
  serviceMonitor: Auto
  interval: 30s
  scrapeNamespaceSelector:
    matchLabels: {kubernetes.io/metadata.name: monitoring}
  labels: {release: kube-prometheus-stack}

scrapeNamespaceSelector is the same value in both places: it names the namespaces allowed through the rendered NetworkPolicies to the metrics port. If your Prometheus lives somewhere other than a namespace called monitoring, set it, or the scrape is denied at the network before it reaches a pod.

labels is attached to the rendered ServiceMonitor, for a Prometheus whose serviceMonitorSelector requires a release label.

Outside Kubernetes, scrape port 9550 on every ysearch process and make sure your job supplies instance. A static configuration is enough:

scrape_configs:
  - job_name: ysearch
    static_configs:
      - targets:
          - coordinator-1:9550
          - worker-1:9550
          - worker-2:9550
          - publisher-1:9550

The console's /metrics covers the console process only — six label-free gauges about its own bridge:

The engine's families — ysearch_query_*, ysearch_ingest_*, ysearch_publisher_*, ysearch_cache_*, and the rest — come from the data-plane processes. Scraping the console does not scrape the engine.

deploy/prometheus/ysearch.rules.yml is a single group, ysearch-m4, evaluated every 30 seconds. It is the M4 ingest and maintenance surface, not a complete alert set — nothing there covers query latency or cache behavior yet.

The two that matter most on a first deployment:

YsearchPublisherLeaseLost is the one that means documents stop becoming visible. Exactly one publisher holds a lease at a time; when none does, sealed segments accumulate without being announced.

expr: |
  max without(instance, pod) (ysearch_publisher_leader{role=~".*publisher.*"}) == 0
  or
  absent(ysearch_publisher_leader{role=~".*publisher.*"})
for: 2m

The absent() arm is what catches a publisher that vanished entirely rather than one that lost its lease. Both are the same outage.

YsearchPublicationLagHigh is the freshness alert. The histogram measures commit-marker durability to catalog-generation publication:

expr: |
  histogram_quantile(
    0.99,
    sum by (le, role) (
      rate(ysearch_publisher_publication_lag_seconds_bucket{role=~".*publisher.*"}[10m])
    )
  ) > 60
for: 5m

Each rule carries a runbook annotation pointing into the repository's milestone evidence, for example docs/milestones/m4-evidence.md#publisher-leadership.

Outside the Prometheus Operator, load the file directly:

rule_files:
  - /etc/prometheus/rules/ysearch.rules.yml

The repository keeps deploy/prometheus/rules_test.go alongside it, so the expressions are unit-tested rather than only linted.

Under the Prometheus Operator, wrap the same groups in a PrometheusRule. deploy/operator/monitoring.yaml shows the shape and adds one rule for the operator process itself:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ysearch-operator
  namespace: ysearch-system
spec:
  groups:
  - name: ysearch-operator
    rules:
    - alert: YSearchOperatorReconcileErrors
      expr: increase(controller_runtime_reconcile_errors_total{controller=~"ysearch.*"}[10m]) > 0
      for: 10m
      labels: {severity: warning}
      annotations:
        summary: YSearch operator reconciliation is failing

Data-plane and control-plane processes use different health mechanisms.

A data-plane process has no HTTP health endpoint. Use a gRPC probe:

startupProbe:
  grpc: {port: 9500}
  failureThreshold: 60
  periodSeconds: 2
  timeoutSeconds: 2
readinessProbe:
  grpc: {port: 9500}
  failureThreshold: 3
  periodSeconds: 5
  timeoutSeconds: 2
livenessProbe:
  grpc: {port: 9500}
  failureThreshold: 3
  periodSeconds: 10
  timeoutSeconds: 2

These are the operator's probe defaults. Catalog-bound roles resolve their generation before opening the listener, so startup probes must allow time for catalog reads and cache validation.

The admin console's /readyz becomes unavailable before graceful drain, which is what removes it from Service endpoints ahead of shutdown.

A second, smaller set of artifacts covers the console process rather than the engine: observability/alerts/admin.yaml, observability/recording-rules/admin.yaml, and a starter Grafana dashboard at observability/dashboards/admin.json.

They assume a scrape job named ysearch-admin and define one recorded ratio plus three alerts:

There is no equivalent shipped dashboard for the data plane yet.

If you are picking a handful of series to graph on day one:

Every one of these, with its labels and meaning, is in the metrics reference.

• Metrics reference — every exported family
• The admin console — per-index detail available through the console