Every Prometheus family ysearch exports, with its type, labels, and meaning — plus the closed label vocabularies and the cardinality policy that keeps them bounded.
Engine server processes expose /metrics on observability.metrics_listen
(:9550 by default) when metrics are enabled. The admin console exposes its
own metrics on its HTTP listener. The tables below describe the application
families; engine registries also include Go and process collectors.
Labels use closed enumerations; index names, query text, document identities, object keys,
and node addresses are never labels. A registry that would break that rule is
rejected at construction.
The cardinality policy
Two rules are enforced in code before a process serves anything.
Every label must be on the allowlist. The complete set:
role lane service method code outcome cache_tier object_operationexactness cache state storage_mode segment_format strategy eventsource operation publication_operation catalog_operation gc_reasonlease_operation cache_state requested_mode policy_reason phase reasontier ack_status seal_trigger mode similarity candidate_mode tail_orderingwait_stage origin
Certain substrings are forbidden outright, so that a refusal says why:
query document segment_id tenant generation object_key error_messageurl model fingerprint text_digest field index node address endpointpeer worker aggregator unit
Each of those would publish one series per thing that exists rather than per
value in an enum.
The role label
Every series carries role as a constant label. Its value is the role mix
the process serves, not a single role: node --roles builder,publisher reports
role="builder-publisher". Under the Kubernetes operator each Deployment runs
one role, so values are plain role names there.
instance and pod are not emitted by the process and must come from the
scrape target. See observability for the consequences.
Label vocabularies
These are the closed enumerations the tables below refer to.
The shared outcome label uses four values with a common mapping. A canceled
context or a CANCELED/DEADLINE_EXCEEDED
status is canceled; an explicit server rejection — INVALID_ARGUMENT,
NOT_FOUND, FAILED_PRECONDITION, RESOURCE_EXHAUSTED, OUT_OF_RANGE,
PERMISSION_DENIED, UNAUTHENTICATED, UNIMPLEMENTED — is rejected;
anything else is error. What an operator branches on is whether the caller
gave up, whether the server refused, or whether something broke. The error's own
text says the rest, and is exactly what may not become a label.
Cache operations use their own outcome vocabulary rather than the shared one:
hit, miss, admitted, evicted, quarantined, error, and — for the
block tier — bypassed. The byte counters add restored for the full tier.
Every family
Query
Metric
Type
Labels
Meaning
ysearch_query_candidates_total
counter
lane, exactness, strategy
Total candidates scored by bounded execution strategy.
ysearch_query_completed_total
counter
lane, exactness, outcome
Total search queries completed by outcome.
ysearch_query_duration_seconds
histogram
lane, exactness
Search query execution duration in seconds.
ysearch_query_posting_frames_total
counter
storage_mode, outcome
Total posting frames processed by storage mode and bounded outcome.
ysearch_query_result_bytes_total
counter
lane, exactness
Total packed result payload bytes delivered to clients.
ysearch_query_result_cache_total
counter
outcome
Generation-local result cache lookups by bounded outcome.
ysearch_query_result_frames_total
counter
lane, exactness
Total packed result frames delivered to clients.
ysearch_query_send_blocked_seconds
histogram
lane, exactness
Time spent in synchronous result stream sends, including flow-control blocking.
ysearch_query_started_total
counter
lane, exactness
Total search queries admitted for execution.
ysearch_query_streamed_ids_total
counter
lane, exactness
Total result document IDs delivered to clients.
Distributed execution
Metric
Type
Labels
Meaning
ysearch_distributed_phase_duration_seconds
histogram
phase, outcome
Distributed query phase duration by bounded phase and outcome.
ysearch_distributed_work_unit_failures_total
counter
reason
Work streams that failed their query outright, by bounded reason.
ysearch_distributed_work_unit_retries_total
counter
reason
Work streams reassigned to another target, by bounded reason.
ysearch_distributed_work_units_total
counter
lane, outcome
Work units a worker served, by lane and outcome.
Merge tier
Metric
Type
Labels
Meaning
ysearch_merge_bound_updates_total
counter
outcome
Upward frames stamped with a bound, by whether it tightened a held frame or closed the run.
ysearch_merge_buffered_frames
gauge
—
Decoded work pages this node holds at once across every live merge.
ysearch_merge_children_total
counter
outcome
Direct children merged, by whether the child's run was observed complete. Divided by runs_total this is the fan-in.
ysearch_merge_parent_send_blocked_seconds
histogram
—
Time one upward frame spent in the parent's send, including flow-control blocking.
ysearch_merge_runs_total
counter
outcome
Merge runs this node served, by outcome.
Placement
Metric
Type
Labels
Meaning
ysearch_placement_advances_total
counter
lane, reason
Dispatches a candidate refused, advancing to the next, by lane and bounded reason.
ysearch_placement_advertisement_age_seconds
histogram
—
Age of an advertisement when an aggregator folded it in, measured against the worker's own observation stamp.
ysearch_placement_advertisements_total
counter
outcome
Advertisement observations, by what became of one: published, dropped, folded in, or expired.
ysearch_placement_candidates_total
counter
tier
Candidates a lookup answered, by whether the node claimed the segment.
ysearch_placement_lookups_total
counter
outcome
Aggregator lookups this node asked, by outcome.
ysearch_placement_selections_total
counter
lane, tier
Dispatch targets chosen, by lane and the regime that produced the choice.
Storage and object I/O
Metric
Type
Labels
Meaning
ysearch_storage_cache_requests_total
counter
cache_tier, outcome
Total cache lookups by bounded tier and outcome.
ysearch_storage_decisions_total
counter
requested_mode, policy_reason
Storage decisions by requested mode and bounded policy reason.
ysearch_storage_fallbacks_total
counter
outcome
Remote reads answered by verified hydration instead, by outcome.
ysearch_storage_io_wait_operations_total
counter
origin, wait_stage
Attributed I/O wait operations by origin and stage.
ysearch_storage_io_wait_seconds_total
counter
origin, wait_stage
Attributed I/O wait by origin and stage, summed across concurrent operations: not a wall time.
ysearch_storage_object_bytes_total
counter
object_operation, outcome
Total object-store bytes transferred by operation and outcome.
ysearch_storage_object_requests_total
counter
object_operation, outcome
Total object-store operations by operation and outcome.
ysearch_storage_open_duration_seconds
histogram
storage_mode, outcome
Segment open duration by explicit storage mode and outcome.
ysearch_storage_restore_duration_seconds
histogram
outcome
Hydrated checkpoint restore duration by outcome.
Segment readers
Metric
Type
Labels
Meaning
ysearch_segment_read_operations_total
counter
source, operation
Segment object reads by byte source and operation.
ysearch_segment_reader_events_total
counter
event
Immutable segment reader lifecycle events.
Disposable cache
Metric
Type
Labels
Meaning
ysearch_cache_bytes
gauge
cache_tier, cache_state
Cache occupancy in bytes by bounded tier and state.
ysearch_cache_entries
gauge
cache_tier, cache_state
Cache occupancy in entries by bounded tier and state.
ysearch_cache_operation_bytes_total
counter
cache_tier, outcome
Bytes moved by cache operations by bounded tier and outcome.
ysearch_cache_operations_total
counter
cache_tier, outcome
Cache operations by bounded tier and outcome.
ysearch_cache_restore_duration_seconds
histogram
outcome
Full-segment hydration (restore) duration by outcome.
Publication
Metric
Type
Labels
Meaning
ysearch_publication_duration_seconds
histogram
publication_operation, outcome
Segment publication phase duration by bounded operation and outcome.
ysearch_publication_object_bytes_total
counter
publication_operation, outcome
Bytes published or verified by bounded operation and outcome.
ysearch_publication_operations_total
counter
publication_operation, outcome
Segment publication phases by bounded operation and outcome.
Catalog
Metric
Type
Labels
Meaning
ysearch_catalog_duration_seconds
histogram
catalog_operation, outcome
Catalog operation duration by bounded operation and outcome.
ysearch_catalog_gc_proposals_total
counter
gc_reason
Objects a dry run proposed, by bounded reason. A dry run never deletes.
ysearch_catalog_lineage_capacity_ratio
gauge
—
Larger of generation-count and encoded-byte utilization observed by the most recent publisher lineage scan.
ysearch_catalog_lineage_repairs_total
counter
reason
Degraded lineage conditions a reconciliation observed, by bounded reason: a repaired sibling conflict, or a reconciliation that never converged and repaired nothing.
ysearch_catalog_operations_total
counter
catalog_operation, outcome
Catalog operations by bounded operation and outcome.
Catalog publisher
Metric
Type
Labels
Meaning
ysearch_publisher_catalog_operations_total
counter
catalog_operation, outcome
Publisher catalog cycles by bounded operation and outcome.
ysearch_publisher_leader
gauge
—
One while this process holds and is renewing the catalog publisher lease.
ysearch_publisher_lease_operations_total
counter
lease_operation, outcome
Publisher lease operations by bounded operation and outcome.
ysearch_publisher_publication_lag_seconds
histogram
—
Commit-marker durability to catalog-generation publication latency.
ysearch_publisher_startup_bundle_active
gauge
—
Active optional startup bundle jobs, including foreground-pressure waits.
ysearch_publisher_startup_bundle_duration_seconds
histogram
—
Optional startup bundle job duration, including background-slot and foreground-pressure waits.
Pending optional startup bundle jobs across process producers.
Catalog follower
Metric
Type
Labels
Meaning
ysearch_follower_held_engine_references
gauge
—
Live query and segment-execution references preventing engine reclamation.
ysearch_follower_lag_seconds
gauge
—
Seconds a newer latest has been observed without being served; zero when current.
ysearch_follower_polls_total
counter
outcome
Catalog latest polls by outcome.
ysearch_follower_retired_engines
gauge
—
Superseded engines still held for in-flight queries or the retirement grace.
ysearch_follower_swaps_total
counter
—
Generations installed by the follower.
ysearch_follower_window_engines
gauge
—
Exact-generation engines held until their catalog-overlap expiry.
ysearch_follower_window_resolutions_total
counter
outcome
Exact-generation window resolutions by bounded outcome.
Ingest router
Metric
Type
Labels
Meaning
ysearch_ingest_admission_refusals_total
counter
reason
Router admission refusals by bounded reason: queue or quota.
ysearch_ingest_batches_total
counter
ack_status
Ingest batches by the acknowledgement they received.
ysearch_ingest_builder_selections_total
counter
outcome
Builder routing decisions by bounded outcome.
ysearch_ingest_bytes_total
counter
ack_status
Ingest request bytes accepted or rejected.
ysearch_ingest_documents_total
counter
ack_status
Ingest documents accepted or rejected.
ysearch_ingest_eligible_builders
gauge
—
Builders eligible in the router's most recent asynchronous health snapshot.
ysearch_ingest_pending_acks_dropped_total
counter
—
Tracked batches dropped from a session's bounded PUBLISHED tracking.
ysearch_ingest_publish_failures_total
counter
—
Seal pipelines that failed to build, publish, or announce; the sealed spool is kept and retried.
ysearch_ingest_queue_capacity_bytes
gauge
—
Current global encoded-byte admission capacity of this router.
ysearch_ingest_queued_bytes
gauge
—
Encoded batch bytes currently reserved by this router across all indexes.
ysearch_ingest_sessions
gauge
—
Open Ingest streams.
ysearch_ingest_unavailable_targets_total
counter
reason
Builder health observations excluded from routing by bounded reason.
Builder
Metric
Type
Labels
Meaning
ysearch_builder_open_build_age_seconds
gauge
—
Age of the oldest open build's first document; zero when nothing is open.
ysearch_builder_open_build_bytes
gauge
—
Spooled bytes of open builds.
ysearch_builder_open_build_documents
gauge
—
Spooled documents of open builds.
ysearch_builder_pending_builds
gauge
—
Sealed builds not yet announced (queued, building, publishing, or retrying).
ysearch_builder_publication_lag_seconds
histogram
—
First document spooled to the generation holding it being served.
ysearch_builder_seal_phase_seconds
histogram
phase, outcome
Seal pipeline phase duration by bounded phase and outcome.
ysearch_builder_seals_total
counter
seal_trigger, outcome
Builds sealed by bounded trigger and outcome.
Maintenance
Metric
Type
Labels
Meaning
ysearch_maintenance_compaction_runs_total
counter
outcome
Scheduled compaction cycles by bounded outcome: completed, noop, refused, or failed.
ysearch_maintenance_gc_sweeps_total
counter
outcome
Destructive garbage-collection sweep cycles by bounded outcome: completed, noop, refused, or failed.
Embedding
Metric
Type
Labels
Meaning
ysearch_embedding_duration_seconds
histogram
mode, outcome
Embedding provider batch duration by mode and bounded outcome.
ysearch_embedding_query_cache_total
counter
outcome
Completed query-vector cache lookups by bounded outcome.
ysearch_embedding_requests_total
counter
mode, outcome
Embedding provider batches by schema-selected mode and bounded outcome.
ysearch_embedding_texts_total
counter
mode, outcome
Texts submitted to embedding providers by mode and bounded outcome.
ysearch_embedding_truncated_texts_total
counter
—
Texts whose gRPC embedding response reported truncation to the model's token window.
Vector queries
Metric
Type
Labels
Meaning
ysearch_vector_queries_total
counter
similarity, candidate_mode, outcome
Vector query executions by similarity, candidate mode, and bounded outcome.
Approximate tail
Metric
Type
Labels
Meaning
ysearch_tail_executions_total
counter
tail_ordering, outcome
Tail executions by actual ordering and bounded outcome.
Configuration
Metric
Type
Labels
Meaning
ysearch_config_reload_failures_total
counter
—
Configuration file reloads that were refused; the previous configuration stayed.
ysearch_config_reloads_total
counter
—
Configuration file reloads that were applied.
Admin streams
Metric
Type
Labels
Meaning
ysearch_admin_fleet_assemblies_total
counter
outcome
Fleet frames assembled, by whether every configured node contributed.
ysearch_admin_fleet_nodes_total
counter
outcome
Per-node contributions summed over fleet assemblies, by the node's condition.
ysearch_admin_frames_total
counter
outcome
Admin stats frames delivered to clients, and frames a slow client was skipped past.
Custom collectors
The custom collectors below expose object I/O totals and WavesDB cache
resources.
Whole-process object I/O
Metric
Type
Meaning
ysearch_storage_process_object_read_bytes
histogram
Whole-process object-store read request count and transferred-byte sum, including metadata requests
ysearch_storage_process_object_write_bytes
histogram
Whole-process object-store write request count and transferred-byte sum
These are constant histograms. Their sample count is the request total and
their sample sum is transferred bytes — they have no meaningful bucket
distribution. Read them as:
promql
rate(ysearch_storage_process_object_read_bytes_sum[5m]) # bytes per secondrate(ysearch_storage_process_object_read_bytes_count[5m]) # requests per second
The counting wrapper encloses the complete object store, so the totals include
ingest publication, catalog access, compaction, cache fills, and query reads —
not only request-scoped search traffic.
Reads count Stat, FullGet, RangeGet, List, Exists, and Size; writes
count Put, Create, CompareAndSwap, and Delete.
WavesDB read resources
Metric
Type
Labels
Meaning
ysearch_storage_wavesdb_read_cache_events_total
counter
cache, outcome
Read-cache events separated by decoded-block and table-reader cache
ysearch_storage_wavesdb_read_cache_bytes
gauge
cache, state
Resident and configured-limit bytes
ysearch_storage_wavesdb_read_cache_entries
gauge
cache, state
Resident and configured-limit entry counts
cache is block, table, or file; outcome is hit, miss, or
eviction; state is resident or limit.
The block-entry family has only state="resident": the decoded-block cache is
bounded by bytes, not by an independent entry count. Rebinding an already
registered process owner is idempotent, so overlapping generation readers do not
multiply either limits or reported occupancy.
The admin console's families
The console process exposes six label-free gauges and nothing else. They
describe the bridge, not any index.
Metric
Type
Meaning
ysearch_admin_build_info
gauge
Always 1; admin console process information
ysearch_admin_configured_targets
gauge
Allowlisted upstream targets
ysearch_admin_fleet_collectors
gauge
Shared upstream fleet streams currently active
ysearch_admin_fleet_subscribers
gauge
Browser fleet-stream subscribers currently active
ysearch_admin_search_in_flight
gauge
Query Lab searches currently occupying a bounded slot
ysearch_admin_search_capacity
gauge
Maximum concurrent Query Lab searches
Standard collectors
Every data-plane registry also registers the Prometheus Go client's process and
Go collectors, so process_* and go_* families are present with their usual
meanings.
Next
Observability — scrape configuration and the shipped alerts