Filters and projection

Two schema options decide what a field can do at query time. indexed makes a field searchable text. filterable makes it comparable as a whole value. stored makes it readable back from a hit. A field can be all three, but as schemas explains, writing indexed: true turns the other two off unless you write them out.

This page covers the query side of filterable and stored.

There is no separate filter syntax to learn. A filterable-but-not-indexed field used as a term in either grammar becomes an exact filter:

ysearch search articles 'body:"object storage" AND category:engineering'

explain --index shows the split between the lowered query and its filters, which is the way to confirm a clause landed where you expected:

ysearch explain 'body:"object storage" AND category:engineering' --index articles

The output's filters: block reads none or one field = value line each.

Ranges apply to numeric and timestamp fields only.

A timestamp bound is RFC 3339, YYYY-MM-DD, or unix milliseconds. Bracket style decides inclusivity: [1 TO 20] includes both bounds, {1 TO 20} excludes both, [1 TO 20} mixes, and * is unbounded on that side.

ysearch search articles 'title:incident AND published:[2026-01-01 TO *]'
ysearch search articles 'price:[1 TO 20}'

--filter field=value adds an equality filter directly, without going through the query string. It is repeatable and it is equality only — there is no range form of the flag. The value is everything after the first =, so a value containing = needs no escaping:

ysearch search articles 'body:cache' \
  --filter category=engineering --filter site=example.test

The value is typed by the field, and a value the field's type cannot read is refused before anything is sent:

value "x" is not an integer
value "x" is not a boolean (true|false|1|0)
value "x" is not a timestamp (RFC 3339, YYYY-MM-DD, or unix millis)

A malformed flag is an invocation error and exits 2: --filter must be field=value, got "x".

A filter value carries the field's type, encoded as the attribute lane stores it:

This is why a filter compares whole values rather than analyzed tokens, and why a regular expression on a filterable field is refused: there is no dictionary of terms to expand against.

Every filter on a request is required. A filter in an optional (Should) or negated (MustNot) position is refused, and the refusal names the spelling that works:

A query of only filters and exclusions is a match-all with those applied, which is a legitimate way to page through a slice of an index:

ysearch search articles 'category:engineering' --keys --top-k 1000

Exclusion is a term, not a filter, and contributes nothing to scoring:

ysearch search articles 'status -beta'

A hit is an ID and a score. --fields asks the server for stored values, returned as one document frame per result frame and joined by position:

ysearch search articles 'body:cache' --fields title,site,url
ysearch search articles 'body:cache' --fields '*'

* requests every stored field, increasing projection work and response size.

Three rules govern projection:

1. A name that is not a stored field is INVALID_ARGUMENT listing the stored fields. An index with no stored fields refuses projection outright.
2. search always projects the key field alongside what you asked for, so every hit carries its key and mutation version — then hides the key from the columns that already show it.
3. A document a particular segment cannot answer — a format-v1 segment, or one built before the field became stored — comes back unavailable rather than failing the search. Missing stored values do not change the retrieval path or its reported exactness class. The response header's projection_unavailable_segments names the segment.

Rule 3 is the one that shows up after a schema evolution. Turning stored on mints a new schema version, but the segments built under the old version do not gain the lane retroactively; they answer unavailable for values that were not stored. Re-ingest the source documents under the updated schema to make those values available.

The table output prints KEY, SCORE to four decimals, VERSION, then one column per projected field. A repeated value is joined with , and an absent one is —.

Collapse groups hits by key in first-appearance order and shows the one with the greatest mutation_version; the version cell then reads … (n versions). --no-collapse prints every hit in rank order instead.

ysearch search articles 'body:cache' --fields title,url --no-collapse

A keyless document — the form inherited from yolosearch's first milestone, with no key field in the schema — has no key to collapse on. It renders as id:<high><low> in hex and stands alone.

Three flags choose what the server has to produce, and the difference is large on a big result set:

ysearch search articles cache --fields title,url --top-k 100
ysearch search articles cache --keys --top-k 100
ysearch search articles cache --ids --top-k 100

--keys and --ids are line-oriented streaming modes built for large result sets; use them when a downstream process is going to fetch the documents itself.

--json prints {"header", "hits":[…], "trailer"} — the protojson header and trailer, and every hit, never collapsed:

{"id":{"high":"…","low":"…"},"score":0.0,"key":"…","mutation_version":"…","fields":{}}

The id halves are decimal strings in JSON where the table shows them in hex. Because JSON output is never collapsed, a client consuming it has to do its own collapse by key if it wants one live version per key — or rely on the generation's liveness object, which already excluded superseded ordinals before top-K. See ingest for what liveness means.

--request FILE reads a protojson SearchRequest instead of a query string. --top-k and --fields still override what the file says, and --top-k 0 keeps the file's value. That is the way to a PrefixTermQuery, which neither grammar spells:

ysearch search articles --request prefix-query.json --fields title,url

A positional query beside --request is an invocation error; so is neither.

Filters need the attribute lane, and phrases need the positions lane. Both arrived with segment format 2; a format-v1 segment refuses each with FAILED_PRECONDITION naming the segment.

• The Lucene grammar — ranges in query syntax.
• The CQP grammar — predicates that lower to filters.
• Schemas — deciding which fields are stored and filterable in the first place.
• Pinned epochs — how a score is computed, and which statistics it reads.