Vector and hybrid search

An index schema may declare several vector fields. Each field pins its own dimension, similarity, normalization rule, source-field recipe, and embedding profile, so a 384-dimensional semantic field and a 768-dimensional visual field can coexist in one index without sharing an embedding space. A query selects exactly one field.

A vector field is a repeated float with a vector option, and the embedding profile it names is declared on the message:

syntax = "proto3";
package cargo;

import "ysearch/v1/schema.proto";
import "google/protobuf/timestamp.proto";

message Article {
  option (ysearch.v1.document) = {
    index: "articles"
    embedding_profiles: {
      name: "semantic-gte-small"
      mode: EMBEDDING_MODE_EXTERNAL
      model: "gte-small"
      revision: "2026-08-31"
      dimensions: 384
      pooling: "mean"
      normalize: true
      maximum_tokens: 512
      endpoint: "http://embeddings.internal:8000"
      credential_environment: "YSEARCH_EMBEDDING_CREDENTIAL"
    }
  };

  string url_hash = 1 [(ysearch.v1.field) = { key: true }];
  string title    = 2 [(ysearch.v1.field) = { indexed: true, stored: true, weight: 2.0 }];
  string headline = 3 [(ysearch.v1.field) = { indexed: true, stored: true, weight: 2.0 }];
  string excerpt  = 4;

  repeated float semantic_vector = 5 [(ysearch.v1.field) = { vector: {
    dimensions: 384
    similarity: VECTOR_SIMILARITY_COSINE
    embedding_profile: "semantic-gte-small"
    source_fields: "title"
    source_fields: "headline"
    source_fields: "excerpt"
    normalize: true
  }}];
}

The vector field's own options:

A document may supply the field explicitly in JSON as an array of numbers. Supplied vectors override automatic embedding. When the field is absent, the builder embeds the ordered source_fields before atomically accepting the batch — so the acknowledgment you get back already accounts for the embedding work.

Apply and load exactly as for any other schema; see schemas and ingest.

Text embedding has three provider modes, plus a mode for vectors you compute yourself:

Calls the profile's OpenAI-compatible endpoint. Plain HTTP is refused unless embedding.external_allow_http=true, which exists for a trusted development LAN and not for anything else. The profile stores only the name of a credential environment variable — credential_environment above. The token itself is never in the schema, the command line, the logs, or a receipt.

export YSEARCH_EMBEDDING_CREDENTIAL='…'
ysearch serve --data-dir ./ys --embedding-external-allow-http

Uses the pure-Go runtime when embedding.internal_enabled=true. A profile pins a manifest_uri and its SHA-256, and every artifact is size- and digest-verified before installation into the separate model cache at embedding.model_cache_dir. No Python, shared library, repository code, custom operator, or tokenizer plugin is executed. Setting embedding.model_cache_dir to the empty string disables internal embedding.

Calls a generic gRPC model service; the runtime name is grpc-embedder-v1, and it is part of the model fingerprint. The schema pins the logical model name, the revision, and a 32-byte model_fingerprint_sha256, while the endpoint stays routing metadata. The fingerprint is part of the vector-space identity: changing the model package cannot silently reuse old vectors or cached query embeddings.

Because model_fingerprint_sha256 is a protobuf bytes field, it is base64 in protobuf JSON. Queries are sent as interactive work and indexing batches as bulk work when embedding.grpc_work_class=auto. Plaintext cluster endpoints require embedding.grpc_allow_insecure=true; production endpoints default to TLS. A dns:///host:port endpoint lets the process-wide connection use gRPC round-robin balancing across all service addresses.

A query names one vector field and gives either text to embed or a vector to use directly.

ysearch search articles --vector-field semantic_vector \
  --vector-text 'Roger Federer' --top-k 30 --ids

The coordinator embeds the text once, then sends the canonical vector to the segment executors and workers. That matters on a fleet: the embedding cost is paid once per query, not once per segment.

ysearch search articles 'title:federer' \
  --vector-field semantic_vector --vector-text 'Roger Federer tennis' \
  --probes 16 --candidates 1000 \
  --lexical-weight 0.5 --vector-weight 1.0 --fusion weighted

A hybrid query is an ordinary lexical query plus a vector clause. The lexical query is lowered by whichever grammar --dialect selects — see Lucene and CQP — and filters apply as usual, so filters and projection still holds.

ysearch search articles --vector-field semantic_vector \
  --vector-file query-vector.json --exhaustive --top-k 20

The file is a JSON float array, bounded to 16 MiB before decoding. Use --vector-file - to read it from standard input.

--fusion rrf selects reciprocal-rank fusion, which combines the two rankings by position rather than by score and so needs no weight calibration between two differently-scaled scores.

--probes and --candidates are the two knobs that trade recall against work. --exhaustive removes the candidate-generation step entirely and is the form to use when establishing a recall baseline for a tuned configuration.

--tail banded opts into a bounded approximate lexical tail. --maximum-ordering-error states the largest score inversion the caller will accept, and --score-ranges asks for conservative ranges alongside the scores.

An unsupported or too-tight request falls back to exact ordering, and the response header and trailer report the resulting ordering. Check those fields to determine which ordering the response uses.

Vector search on main is the inherited index: IVF lists with PQ codes for candidate generation, followed by an exact rerank of --candidates candidates. --probes, --candidates, and --exhaustive control it.

Since segment format 8 (milestone YS3), the builder also writes exact vector lanes for every populated cosine field: the canonical fp16 vectors, IVF centroids with their radii, and int8 codes. Those lanes serve the exact threshold scan that the planned matcher needs. The search flags above do not use them yet, and dot_product and Euclidean fields get no lanes.

The ysearch design keeps a single vector index, ivf_rabitq (IVF with RaBitQ 1-bit codes), in milestone YS16, and removes the IVF-PQ build and read paths then. Segments of formats 1 to 6 with a PQ vector section will still open for non-vector reads, but a vector query on one will fail with VECTOR_FORMAT_UNSUPPORTED (rebuild required) rather than return a partial result. See status.

The process owns one provider manager and one query-vector cache across every hosted index. A repeated text query is keyed by embedding fingerprint plus a SHA-256 of the text; the raw query text is not retained in the cache key.

The cache is bounded by three settings:

Including the embedding fingerprint in the key is what makes a model change safe: a new fingerprint cannot hit an entry produced by the old model.

--ids remains the cheapest large-result output on a vector query for the same reason it is on a lexical one: it requests neither scores nor stored documents.

ysearch search articles --vector-field semantic_vector \
  --vector-text 'cold cache hydration' --top-k 500 --ids

• Schemas — where vector fields and profiles are declared.
• Filters and projection — narrowing and hydrating a hybrid result.
• Configuration — every embedding.* setting and where it may be set.
• The CLI reference — the search flag table.