The Lucene grammar

The Lucene-style dialect is the default for search strings. It supports terms, phrases, Boolean clauses, and ranges. Use CQP for token gaps and regular expressions. Both grammars lower through a shared abstract syntax tree, but they expose different syntax and capabilities.

ysearch search articles 'title:http AND body:"status code"'
ysearch explain 'title:http AND body:"status code"' --dialect lucene

Use explain to inspect parsing and lowering locally. With --index, it fetches the schema from the server to resolve field names and types.

query    := or_expr
or_expr  := and_expr { ("OR" | "||") and_expr }
and_expr := unary { ("AND" | "&&") unary }
unary    := [ "+" | "-" | "NOT" ] primary
primary  := "(" query ")" | field ":" ( range | phrase | term ) | phrase | term
range    := ("[" | "{") bound "TO" bound ("]" | "}")
bound    := term | "*"

A term ends at whitespace or at any of ( ) : [ ] { } " ~ ^. The keywords AND, OR, NOT, and TO are keywords only when followed by whitespace, a parenthesis, or the end of the query, so ANDroid is an ordinary term.

In both grammars the default field is the pseudo-attribute word, which means every indexed field. A bare status lowers to an OR of one term per indexed field — rendered field=* in explain output — which is the exact form under BM25F. When only one field is indexed it collapses to a single term.

That is why a bare query behaves like a search box without anyone configuring a default field, and why adding an indexed field to a schema widens every bare query in the index. See pinned epochs for how BM25F scores are computed and which statistics they read.

Clause occurrence follows these rules:

1. An explicit +, -, or NOT always wins.
2. An infix AND promotes both of its neighbors to required.
3. OR is parsed and has no effect. A clause is optional unless something promoted it.
4. Juxtaposition — a space — is OR, not AND.

ysearch has no scoring-only clause. A boolean is either all-required or all-optional, so a query that mixes required and optional clauses is refused:

+quick brown is refused with optional clauses beside required ones is not supported, and the hint tells you to write +a +b for all required or a b for any.

a AND b OR c is refused for the same reason: AND promoted a and b, OR did nothing to c, and c is left optional beside two required clauses. Write +a +b +c or a b c depending on what you meant.

A query containing only exclusions, such as NOT b, is rejected. Include a positive clause to define the candidate set.

A field that is filterable but not indexed becomes a filter rather than a term, and the grammar spells that with ordinary syntax:

Ranges apply to numeric and timestamp fields only. A range on an indexed field reads field "price" is indexed; ranges apply to filterable fields; a range on a string field reads a range over a string field is not supported. A timestamp bound is RFC 3339, YYYY-MM-DD, or unix milliseconds.

Filters are a conjunction, so a filter in an optional or negated position is refused. Filters and projection covers the typed values and the --filter flag in full.

Recognized-but-unsupported syntax is refused by name, never reinterpreted. Treating foo~2 as a term spelled foo~2 would quietly answer a different question, so the parser refuses and names the spelling that works:

Every refusal carries a hint naming the spelling that works. When you need one of the CQP forms, switch the whole query to --dialect cqp — the grammars are not mixed inside one string.

Each of these was lowered with explain. The notes say what the lowering does, including the parts that are not obvious from the source.

ysearch search articles 'body:cache'

TermQuery{body, cache}. One clause, optional, which for a single-clause boolean is the same result set as required.

ysearch search articles 'title:search AND body:cache'

AND promotes both clauses to required. +title:search +body:cache produces the same tree and therefore the same digest.

ysearch search articles 'cache buffer pool'

Three optional clauses, minimum_should_match: 1, each expanded over every indexed field. This is the shape a search box sends by default, and the one BM25F scores.

ysearch search articles 'body:"object storage"' \
  --filter category=engineering --fields title,category,url --top-k 20

The phrase is the scoring clause; the filter narrows the candidate set and contributes nothing to the score. Writing the filter as +category:engineering inside the query is equivalent when category is filterable.

ysearch search articles 'title:incident AND published:[2026-01-01 TO *]'

published is a filterable timestamp, so the range lowers to a TimeRangeFilter with an open upper bound and the AND keeps title:incident required.

ysearch search articles 'status -beta'

status is the scoring clause; beta is excluded. An exclusion is a term, not a filter, and contributes nothing to scoring.

ysearch search articles '(cache buffer) AND body:eviction'

The group is one optional-set clause; AND promotes the group and body:eviction to required. Note that the inner cache buffer stays an any-of, because promotion applies to the group as a whole.

ysearch explain 'title:new\ york' --index articles

Refused. A literal term value must analyze to exactly one token, and the escape produced the two-token value new york. The hint says to write a phrase — title:"new york" — which is what a two-word value means anyway.

The digest is a SHA-256 hash of the canonical AST. Queries that normalize to the same tree share a digest. It is independent of schema resolution; explain prints it:

ysearch explain 'foo'
ysearch explain '[word="foo"]' --dialect cqp

Both print the same 64-hex digest, because a metacharacter-free, case-sensitive CQP value is normalized to a literal. By contrast +foo and foo have different digests: a lone required clause is a different tree from a lone optional one, even though they select the same documents.

This is the mechanism for confirming that a query rewriter, a client library, and a hand-typed query all mean the same thing.

A parse or lowering refusal prints a caret rendering to stderr and exits 1: the message, the hint on its own line in parentheses, the offending source line, and ^ carets under the bytes at fault with their byte offset, then ysearch: query refused. An invocation error — a missing argument, a bad flag value — exits 2 without touching the query.

Server-side refusals arrive as gRPC status codes: INVALID_ARGUMENT for a query the engine cannot compile, FAILED_PRECONDITION for a segment lacking the lane a query needs, such as positions for a phrase.

The limits that produce a refusal, each a query.* setting:

An unknown field names the schema's fields: unknown field "nofield"; known fields: body, title, url.

A client that would rather not embed a grammar can send the string to the server instead. SearchRequest.query may be a TypedDSLQuery{language, expression, version} where language is lucene or cqp, and the server lowers it against the index schema exactly as the client would. A refusal comes back as INVALID_ARGUMENT whose message is the same caret rendering.

The CLI lowers client-side so that explain and search agree byte for byte.

• The CQP grammar — positional queries, regex, and slop.
• Filters and projection — typed filter values and --fields.
• Vector and hybrid search — adding a vector clause to a lexical query.
• Architecture — the roles a query passes through after lowering.