Skip to content

Reference

Metrics reference

Every Prometheus family ysearch exports, with its type, labels, and meaning — plus the closed label vocabularies and the cardinality policy that keeps them bounded.


Engine server processes expose /metrics on observability.metrics_listen (:9550 by default) when metrics are enabled. The admin console exposes its own metrics on its HTTP listener. The tables below describe the application families; engine registries also include Go and process collectors.

Labels use closed enumerations; index names, query text, document identities, object keys, and node addresses are never labels. A registry that would break that rule is rejected at construction.

The cardinality policy

Two rules are enforced in code before a process serves anything.

Every label must be on the allowlist. The complete set:

role  lane  service  method  code  outcome  cache_tier  object_operation
exactness  cache  state  storage_mode  segment_format  strategy  event
source  operation  publication_operation  catalog_operation  gc_reason
lease_operation  cache_state  requested_mode  policy_reason  phase  reason
tier  ack_status  seal_trigger  mode  similarity  candidate_mode  tail_ordering
wait_stage  origin

Certain substrings are forbidden outright, so that a refusal says why:

query  document  segment_id  tenant  generation  object_key  error_message
url  model  fingerprint  text_digest  field  index  node  address  endpoint
peer  worker  aggregator  unit

Each of those would publish one series per thing that exists rather than per value in an enum.

The role label

Every series carries role as a constant label. Its value is the role mix the process serves, not a single role: node --roles builder,publisher reports role="builder-publisher". Under the Kubernetes operator each Deployment runs one role, so values are plain role names there.

instance and pod are not emitted by the process and must come from the scrape target. See observability for the consequences.

Label vocabularies

These are the closed enumerations the tables below refer to.

Label Values
outcome success, canceled, error, rejected
lane unspecified, interactive, streaming, background
exactness GLOBAL_EXACT, EXACT_WITHIN_CANDIDATES, APPROXIMATE
phase (query) parse, resolve, plan, place, execute, merge, encode, send
phase (seal) build, publish, announce, visible
ack_status accepted_ephemeral, rejected, throttled, published
seal_trigger age, bytes, documents, flush, schema, replay
cache_tier full, block
cache_state total, used, pinned, partial, quarantined, evictable
cache (WavesDB) block, table, file
state (WavesDB) resident, limit
storage_mode, requested_mode hydrate_full, remote_blocks, auto, unspecified
policy_reason full_cache_hit, absent, match_all, unknown_statistics, scan, hydrate_cost, remote_cost, capacity_rejection, availability_fallback, explicit, unspecified
lease_operation acquire, renew, loss
origin (I/O wait) demand, warm-prefetch, cache-writer
wait_stage range-fetch, full-fetch, metadata-round-trip, cache-hit, cache-miss, admission-wait, verify, persist, shared-load-wait
catalog_operation (publisher) reconcile, announce

The shared outcome label uses four values with a common mapping. A canceled context or a CANCELED/DEADLINE_EXCEEDED status is canceled; an explicit server rejection — INVALID_ARGUMENT, NOT_FOUND, FAILED_PRECONDITION, RESOURCE_EXHAUSTED, OUT_OF_RANGE, PERMISSION_DENIED, UNAUTHENTICATED, UNIMPLEMENTED — is rejected; anything else is error. What an operator branches on is whether the caller gave up, whether the server refused, or whether something broke. The error's own text says the rest, and is exactly what may not become a label.

Cache operations use their own outcome vocabulary rather than the shared one: hit, miss, admitted, evicted, quarantined, error, and — for the block tier — bypassed. The byte counters add restored for the full tier.

Every family

Query

Metric Type Labels Meaning
ysearch_query_candidates_total counter lane, exactness, strategy Total candidates scored by bounded execution strategy.
ysearch_query_completed_total counter lane, exactness, outcome Total search queries completed by outcome.
ysearch_query_duration_seconds histogram lane, exactness Search query execution duration in seconds.
ysearch_query_posting_frames_total counter storage_mode, outcome Total posting frames processed by storage mode and bounded outcome.
ysearch_query_result_bytes_total counter lane, exactness Total packed result payload bytes delivered to clients.
ysearch_query_result_cache_total counter outcome Generation-local result cache lookups by bounded outcome.
ysearch_query_result_frames_total counter lane, exactness Total packed result frames delivered to clients.
ysearch_query_send_blocked_seconds histogram lane, exactness Time spent in synchronous result stream sends, including flow-control blocking.
ysearch_query_started_total counter lane, exactness Total search queries admitted for execution.
ysearch_query_streamed_ids_total counter lane, exactness Total result document IDs delivered to clients.

Distributed execution

Metric Type Labels Meaning
ysearch_distributed_phase_duration_seconds histogram phase, outcome Distributed query phase duration by bounded phase and outcome.
ysearch_distributed_work_unit_failures_total counter reason Work streams that failed their query outright, by bounded reason.
ysearch_distributed_work_unit_retries_total counter reason Work streams reassigned to another target, by bounded reason.
ysearch_distributed_work_units_total counter lane, outcome Work units a worker served, by lane and outcome.

Merge tier

Metric Type Labels Meaning
ysearch_merge_bound_updates_total counter outcome Upward frames stamped with a bound, by whether it tightened a held frame or closed the run.
ysearch_merge_buffered_frames gauge — Decoded work pages this node holds at once across every live merge.
ysearch_merge_children_total counter outcome Direct children merged, by whether the child's run was observed complete. Divided by runs_total this is the fan-in.
ysearch_merge_parent_send_blocked_seconds histogram — Time one upward frame spent in the parent's send, including flow-control blocking.
ysearch_merge_runs_total counter outcome Merge runs this node served, by outcome.

Placement

Metric Type Labels Meaning
ysearch_placement_advances_total counter lane, reason Dispatches a candidate refused, advancing to the next, by lane and bounded reason.
ysearch_placement_advertisement_age_seconds histogram — Age of an advertisement when an aggregator folded it in, measured against the worker's own observation stamp.
ysearch_placement_advertisements_total counter outcome Advertisement observations, by what became of one: published, dropped, folded in, or expired.
ysearch_placement_candidates_total counter tier Candidates a lookup answered, by whether the node claimed the segment.
ysearch_placement_lookups_total counter outcome Aggregator lookups this node asked, by outcome.
ysearch_placement_selections_total counter lane, tier Dispatch targets chosen, by lane and the regime that produced the choice.

Storage and object I/O

Metric Type Labels Meaning
ysearch_storage_cache_requests_total counter cache_tier, outcome Total cache lookups by bounded tier and outcome.
ysearch_storage_decisions_total counter requested_mode, policy_reason Storage decisions by requested mode and bounded policy reason.
ysearch_storage_fallbacks_total counter outcome Remote reads answered by verified hydration instead, by outcome.
ysearch_storage_io_wait_operations_total counter origin, wait_stage Attributed I/O wait operations by origin and stage.
ysearch_storage_io_wait_seconds_total counter origin, wait_stage Attributed I/O wait by origin and stage, summed across concurrent operations: not a wall time.
ysearch_storage_object_bytes_total counter object_operation, outcome Total object-store bytes transferred by operation and outcome.
ysearch_storage_object_requests_total counter object_operation, outcome Total object-store operations by operation and outcome.
ysearch_storage_open_duration_seconds histogram storage_mode, outcome Segment open duration by explicit storage mode and outcome.
ysearch_storage_restore_duration_seconds histogram outcome Hydrated checkpoint restore duration by outcome.

Segment readers

Metric Type Labels Meaning
ysearch_segment_read_operations_total counter source, operation Segment object reads by byte source and operation.
ysearch_segment_reader_events_total counter event Immutable segment reader lifecycle events.

Disposable cache

Metric Type Labels Meaning
ysearch_cache_bytes gauge cache_tier, cache_state Cache occupancy in bytes by bounded tier and state.
ysearch_cache_entries gauge cache_tier, cache_state Cache occupancy in entries by bounded tier and state.
ysearch_cache_operation_bytes_total counter cache_tier, outcome Bytes moved by cache operations by bounded tier and outcome.
ysearch_cache_operations_total counter cache_tier, outcome Cache operations by bounded tier and outcome.
ysearch_cache_restore_duration_seconds histogram outcome Full-segment hydration (restore) duration by outcome.

Publication

Metric Type Labels Meaning
ysearch_publication_duration_seconds histogram publication_operation, outcome Segment publication phase duration by bounded operation and outcome.
ysearch_publication_object_bytes_total counter publication_operation, outcome Bytes published or verified by bounded operation and outcome.
ysearch_publication_operations_total counter publication_operation, outcome Segment publication phases by bounded operation and outcome.

Catalog

Metric Type Labels Meaning
ysearch_catalog_duration_seconds histogram catalog_operation, outcome Catalog operation duration by bounded operation and outcome.
ysearch_catalog_gc_proposals_total counter gc_reason Objects a dry run proposed, by bounded reason. A dry run never deletes.
ysearch_catalog_lineage_capacity_ratio gauge — Larger of generation-count and encoded-byte utilization observed by the most recent publisher lineage scan.
ysearch_catalog_lineage_repairs_total counter reason Degraded lineage conditions a reconciliation observed, by bounded reason: a repaired sibling conflict, or a reconciliation that never converged and repaired nothing.
ysearch_catalog_operations_total counter catalog_operation, outcome Catalog operations by bounded operation and outcome.

Catalog publisher

Metric Type Labels Meaning
ysearch_publisher_catalog_operations_total counter catalog_operation, outcome Publisher catalog cycles by bounded operation and outcome.
ysearch_publisher_leader gauge — One while this process holds and is renewing the catalog publisher lease.
ysearch_publisher_lease_operations_total counter lease_operation, outcome Publisher lease operations by bounded operation and outcome.
ysearch_publisher_publication_lag_seconds histogram — Commit-marker durability to catalog-generation publication latency.
ysearch_publisher_startup_bundle_active gauge — Active optional startup bundle jobs, including foreground-pressure waits.
ysearch_publisher_startup_bundle_duration_seconds histogram — Optional startup bundle job duration, including background-slot and foreground-pressure waits.
ysearch_publisher_startup_bundle_events_total counter outcome Optional startup bundle events: built, failed, canceled, dropped, coalesced or discarded.
ysearch_publisher_startup_bundle_pending gauge — Pending optional startup bundle jobs across process producers.

Catalog follower

Metric Type Labels Meaning
ysearch_follower_held_engine_references gauge — Live query and segment-execution references preventing engine reclamation.
ysearch_follower_lag_seconds gauge — Seconds a newer latest has been observed without being served; zero when current.
ysearch_follower_polls_total counter outcome Catalog latest polls by outcome.
ysearch_follower_retired_engines gauge — Superseded engines still held for in-flight queries or the retirement grace.
ysearch_follower_swaps_total counter — Generations installed by the follower.
ysearch_follower_window_engines gauge — Exact-generation engines held until their catalog-overlap expiry.
ysearch_follower_window_resolutions_total counter outcome Exact-generation window resolutions by bounded outcome.

Ingest router

Metric Type Labels Meaning
ysearch_ingest_admission_refusals_total counter reason Router admission refusals by bounded reason: queue or quota.
ysearch_ingest_batches_total counter ack_status Ingest batches by the acknowledgement they received.
ysearch_ingest_builder_selections_total counter outcome Builder routing decisions by bounded outcome.
ysearch_ingest_bytes_total counter ack_status Ingest request bytes accepted or rejected.
ysearch_ingest_documents_total counter ack_status Ingest documents accepted or rejected.
ysearch_ingest_eligible_builders gauge — Builders eligible in the router's most recent asynchronous health snapshot.
ysearch_ingest_pending_acks_dropped_total counter — Tracked batches dropped from a session's bounded PUBLISHED tracking.
ysearch_ingest_publish_failures_total counter — Seal pipelines that failed to build, publish, or announce; the sealed spool is kept and retried.
ysearch_ingest_queue_capacity_bytes gauge — Current global encoded-byte admission capacity of this router.
ysearch_ingest_queued_bytes gauge — Encoded batch bytes currently reserved by this router across all indexes.
ysearch_ingest_sessions gauge — Open Ingest streams.
ysearch_ingest_unavailable_targets_total counter reason Builder health observations excluded from routing by bounded reason.

Builder

Metric Type Labels Meaning
ysearch_builder_open_build_age_seconds gauge — Age of the oldest open build's first document; zero when nothing is open.
ysearch_builder_open_build_bytes gauge — Spooled bytes of open builds.
ysearch_builder_open_build_documents gauge — Spooled documents of open builds.
ysearch_builder_pending_builds gauge — Sealed builds not yet announced (queued, building, publishing, or retrying).
ysearch_builder_publication_lag_seconds histogram — First document spooled to the generation holding it being served.
ysearch_builder_seal_phase_seconds histogram phase, outcome Seal pipeline phase duration by bounded phase and outcome.
ysearch_builder_seals_total counter seal_trigger, outcome Builds sealed by bounded trigger and outcome.

Maintenance

Metric Type Labels Meaning
ysearch_maintenance_compaction_runs_total counter outcome Scheduled compaction cycles by bounded outcome: completed, noop, refused, or failed.
ysearch_maintenance_gc_sweeps_total counter outcome Destructive garbage-collection sweep cycles by bounded outcome: completed, noop, refused, or failed.

Embedding

Metric Type Labels Meaning
ysearch_embedding_duration_seconds histogram mode, outcome Embedding provider batch duration by mode and bounded outcome.
ysearch_embedding_query_cache_total counter outcome Completed query-vector cache lookups by bounded outcome.
ysearch_embedding_requests_total counter mode, outcome Embedding provider batches by schema-selected mode and bounded outcome.
ysearch_embedding_texts_total counter mode, outcome Texts submitted to embedding providers by mode and bounded outcome.
ysearch_embedding_truncated_texts_total counter — Texts whose gRPC embedding response reported truncation to the model's token window.

Vector queries

Metric Type Labels Meaning
ysearch_vector_queries_total counter similarity, candidate_mode, outcome Vector query executions by similarity, candidate mode, and bounded outcome.

Approximate tail

Metric Type Labels Meaning
ysearch_tail_executions_total counter tail_ordering, outcome Tail executions by actual ordering and bounded outcome.

Configuration

Metric Type Labels Meaning
ysearch_config_reload_failures_total counter — Configuration file reloads that were refused; the previous configuration stayed.
ysearch_config_reloads_total counter — Configuration file reloads that were applied.

Admin streams

Metric Type Labels Meaning
ysearch_admin_fleet_assemblies_total counter outcome Fleet frames assembled, by whether every configured node contributed.
ysearch_admin_fleet_nodes_total counter outcome Per-node contributions summed over fleet assemblies, by the node's condition.
ysearch_admin_frames_total counter outcome Admin stats frames delivered to clients, and frames a slow client was skipped past.

Custom collectors

The custom collectors below expose object I/O totals and WavesDB cache resources.

Whole-process object I/O

Metric Type Meaning
ysearch_storage_process_object_read_bytes histogram Whole-process object-store read request count and transferred-byte sum, including metadata requests
ysearch_storage_process_object_write_bytes histogram Whole-process object-store write request count and transferred-byte sum

These are constant histograms. Their sample count is the request total and their sample sum is transferred bytes — they have no meaningful bucket distribution. Read them as:

promql
rate(ysearch_storage_process_object_read_bytes_sum[5m])    # bytes per second
rate(ysearch_storage_process_object_read_bytes_count[5m])  # requests per second

The counting wrapper encloses the complete object store, so the totals include ingest publication, catalog access, compaction, cache fills, and query reads — not only request-scoped search traffic.

Reads count Stat, FullGet, RangeGet, List, Exists, and Size; writes count Put, Create, CompareAndSwap, and Delete.

WavesDB read resources

Metric Type Labels Meaning
ysearch_storage_wavesdb_read_cache_events_total counter cache, outcome Read-cache events separated by decoded-block and table-reader cache
ysearch_storage_wavesdb_read_cache_bytes gauge cache, state Resident and configured-limit bytes
ysearch_storage_wavesdb_read_cache_entries gauge cache, state Resident and configured-limit entry counts

cache is block, table, or file; outcome is hit, miss, or eviction; state is resident or limit.

The block-entry family has only state="resident": the decoded-block cache is bounded by bytes, not by an independent entry count. Rebinding an already registered process owner is idempotent, so overlapping generation readers do not multiply either limits or reported occupancy.

The admin console's families

The console process exposes six label-free gauges and nothing else. They describe the bridge, not any index.

Metric Type Meaning
ysearch_admin_build_info gauge Always 1; admin console process information
ysearch_admin_configured_targets gauge Allowlisted upstream targets
ysearch_admin_fleet_collectors gauge Shared upstream fleet streams currently active
ysearch_admin_fleet_subscribers gauge Browser fleet-stream subscribers currently active
ysearch_admin_search_in_flight gauge Query Lab searches currently occupying a bounded slot
ysearch_admin_search_capacity gauge Maximum concurrent Query Lab searches

Standard collectors

Every data-plane registry also registers the Prometheus Go client's process and Go collectors, so process_* and go_* families are present with their usual meanings.

Next

  • Observability — scrape configuration and the shipped alerts
  • Glossary — the vocabulary the label values use