Schemas

A ysearch schema is a protobuf message. The field numbers in that message are the stable field IDs in the index. Options on the message and on each field say what the index does with the value: index it as text, store it for projection, make it filterable, or treat it as the document's identity.

The binary embeds the annotation file, available through ysearch schema options, and compiles the .proto in process. Applying a schema requires no separate protoc or buf installation.

syntax = "proto3";
package acme;
import "ysearch/v1/schema.proto";
import "google/protobuf/timestamp.proto";

message Article {
  option (ysearch.v1.document) = { index: "articles" };
  string url   = 1 [(ysearch.v1.field) = { key: true }];
  string title = 2 [(ysearch.v1.field) = { indexed: true, stored: true, weight: 2.0, b: 0.6 }];
  string body  = 3 [(ysearch.v1.field) = { indexed: true }];
  string team  = 4;                              // stored + filterable by default
  google.protobuf.Timestamp published = 5;       // stored + filterable, chronological ranges
}

Apply it against a running server:

ysearch schema apply articles.proto
# schema articles: created v1 (message acme.Article, 5 fields: 2 indexed, 4 stored, 3 filterable, key: url)

The index the schema lands in is the index in the message option, or --index when the file declares none. The document message is the one carrying the (ysearch.v1.document) option, else --message, else the file's only message.

Field options determine the derived defaults below.

The key field is exactly one non-repeated string. It occupies no field ID — it lives in the stored record's header — which is why a message's fields may start at 1 without reserving anything.

Accepted field types are string, int64, double, bool, bytes, and google.protobuf.Timestamp, each optionally repeated. Vector fields use repeated float with the vector annotation described in vector search. Other ordinary field types — int32, uint64, float, an enum, a map, a nested message, a oneof, a proto2 required field — is refused with the field named and the reason stated. A bytes field is stored but can never be filterable.

A schema can also be authored directly as an IndexSchema in YAML, JSON, or text protobuf, or supplied as a compiled descriptor set (.binpb). The file extension chooses the form, and --schema names the file when the extension is not enough:

ysearch schema apply --schema articles.yaml --index articles

An authored schema gets no derivation defaults. What is written is what is meant; only weight and b are filled in. That is the form to use when a generator produces schemas, because it removes the option-default table above from the equation.

Every applied schema is canonicalized — fields sorted by ID, analyzer unicode-simple-v1 — hashed with SHA-256 over the canonical encoding, and stored immutably at indexes/<index>/schema/<n>.binpb with a schema/latest pointer.

An index exists exactly when it has a schema/latest. There is no create verb. The first apply is the creation, and index list lists what has one.

The validation refusals come back as INVALID_ARGUMENT naming the field: no indexed field at all, an indexed non-string, two keys, an indexed key, weight on a non-indexed field, a filterable bytes field.

Changes are additive and auto-versioned. Re-applying unchanged bytes is a no-op.

A refusal is FAILED_PRECONDITION and carries the diff:

schema-1 -> schema-2: 2 change(s) (refused: not additive)
  ok       team                     FIELD_ADDED    "" -> "STRING filterable stored"
  refused  body                     INDEXED        "true" -> "false"

Every segment keeps the schema it was built under, and the engine resolves field names per segment. A change the old segments cannot honor would silently reinterpret them. One generation may therefore mix schema-1 and schema-2 segments; a stored field that a segment predates is unavailable from that segment and is reported as such, rather than failing the query. See segments and catalog for why a segment is never rewritten in place.

schema diff renders exactly the table the server puts in a refusal, writes nothing, and exits 1 when the change is not additive:

ysearch schema diff articles.proto || echo 'not additive; needs a new index'

schema apply --dry-run does the same but appends the would-be version.

ysearch schema apply [file] [--proto f|--schema f] [--index i] [--message m] [--dry-run]
ysearch schema diff  [file] [--proto f|--schema f] [--index i] [--message m]
ysearch schema show --index i [--version n]
ysearch schema history --index i
ysearch schema options
ysearch index list [--prefix p]
ysearch index describe <index>

show prints one version as a table, with weight and b shown only on indexed fields; history prints every version with its full digest; index list shows each index's schema version, short digest, and active generation; index describe prints an index's generation and scorer followed by its schema. All of them accept --json.

schema propose runs offline and recursively analyzes .json, .jsonl, .ndjson, .xml, .txt, and .md files. It writes an editable schema and a normalized JSONL file, then prints the commands that would create the index and ingest that JSONL. It never applies anything:

ysearch schema propose ./documents --index articles --output ./articles.schema.json
ysearch schema apply --index articles --schema ./articles.schema.json
ysearch push articles ./articles.schema.json.documents.jsonl --format jsonl

Existing output files are refused, so write the output outside the input directory and a second proposal will not analyze its own artifacts. Limits are 8 MiB per file, 32 MiB of total input, and 10,000 documents; normalized JSONL is capped at 64 MiB total with each record under 16 MiB.

Review the inferred types before applying the proposal. Dates stay strings. Nested objects and incompatible field types become JSON strings with a diagnostic. Numeric arrays are ordinary repeated values, not vectors. A nonempty key or id present on every document supplies the document key; otherwise a content hash does, which means identical documents share a key and deduplicate on ingestion. Names that ingestion reserves — id, key, fields, mutation_version — are renamed. Use the normalized JSONL rather than the original files when field names or value shapes were changed.

The admin console runs the same proposal from an uploaded example file, up to 1 MiB, and previews the change before applying it.

• Ingest — pushing documents against the schema you applied.
• Filters and projection — what stored and filterable enable at query time.
• Vector and hybrid search — declaring vector fields and embedding profiles in the same message.
• Pinned epochs — how BM25F scores are computed from weight, b, and the index statistics.