The CQP grammar

CQP expresses patterns over token sequences, including literal tokens, regular expressions, and gaps. Select it with --dialect cqp:

ysearch search articles '[word="status"] [word="code"]' --dialect cqp
ysearch explain '[word="status"] [word="code"]' --dialect cqp

CQP and Lucene lower through a shared abstract syntax tree. Queries with the same canonical tree share a digest. CQP provides regular expressions, case-insensitive matching, token gaps, and slop syntax that the Lucene-style parser does not accept.

query            := sequence
sequence         := element { element }
element          := atom [ quantifier ]
atom             := token_constraint | phrase | "(" query ")"
token_constraint := "[" [ predicate { "&" predicate } ] "]"
predicate        := ident ( "=" | "!=" ) string [ "%c" ]
quantifier       := "{" number [ "," number ] "}"

An attribute is a schema field name. word is the only magic name: it means every indexed field, so [word="a"] lowers to an OR of one term per indexed field — the exact form under BM25F — and collapses to a single term when only one field is indexed.

Every predicate value is compiled at parse time as the anchored regular expression ^(?:value)$, with (?i) prepended under %c. Anchoring is what makes [word="cat"] match the token cat and not the token catalog.

A case-sensitive value made only of letters, digits, _, and - is then normalized to a literal. That normalization is what gives [word="foo"] the same digest as the Lucene query foo, and it means the common case costs no regex machinery at all.

%c always forces the regex path, even on a value that would otherwise normalize.

Regular-expression expansion is per segment: the pattern is expanded into the dictionary terms it matches. That expansion is capped by query.regex_max_expansions (default 256, read once at startup by every engine the process serves). Over the cap, the query is refused with INVALID_ARGUMENT:

regex "t.*" matches more than 256 terms in field 2, cap 256

The sweep stops at the cap, so the message names the cap rather than the true count, and names the field by its schema ID rather than its name. A pattern that matches nothing is a proven-empty result, not an error.

A regular expression may stand inside a phrase: [word="colou?r"] []{0,3} [word="blind"]. A pattern on a filterable (non-indexed) field is refused: filterable fields compare whole values, and there is no dictionary to expand against.

A quantifier is legal only on []. [word="a"]{1,5} is refused, and so is the unbounded {2,}. A gap may not open or close a phrase, so []{1} [word="a"] is refused as well.

Given that, quantifiers lower by three rules.

1. {n} expands to n copies. [word="a"] []{2} [word="b"] is the exact phrase with two tokens between a and b.

2. When every variable gap in a sequence has minimum 0, the sequence becomes one phrase whose slop is the sum of the maxima. For one gap that is exact: [word="a"] []{0,3} [word="b"] is a phrase with slop 3. For two or more gaps it is a bound on the total span, not on each gap individually — [word="a"] []{0,2} [word="b"] []{0,2} [word="c"] is one phrase with slop 4, so a match may spend all four tokens of slack in the first gap.

3. When some gap has a minimum above 0, the sequence expands into an OR of every exact combination. {1,3} is three alternatives; {1,3} … {1,3} is nine. The expansion is capped by query.phrase_gap_expansion_limit (default 64) and a sequence over the cap is refused:

quantifier {m,n} is not expressible: … exceeds the alternative cap (64)

Rule 3 can expand into many alternatives. Use {0,n} when a zero minimum gap is acceptable; it lowers to one slop phrase.

• A term matches one analyzed token. A literal value must analyze to exactly one token, and a pattern may not contain whitespace, because no token does: [word="new york"] is refused rather than lowered to a regular expression that could never match. Write a phrase.
• A phrase is adjacent tokens in order. Every element of a phrase constrains the same field; a phrase across fields, or a compound element inside one, is refused.
• A gap is an exact number of tokens between two elements. [word="color"] []{1} [word="blind"] matches color X blind.
• Slop is a bounded window: the ordered positions must fit within the slack.

Phrases, gaps, and slop all need the positions lane, so they run only on segments of format 2 or later. A format-1 segment refuses a phrase with FAILED_PRECONDITION:

segment <id> has no positions lane for a phrase

The builder writes format 8 by default, so every segment it writes today has the positions lane. See segments and catalog for how segments are stored.

These forms are recognized so that the error can name them, and refused because the index does not hold what they need:

ysearch search articles '[word="cache"]' --dialect cqp

Identical in every respect to the Lucene query cache, digest included.

ysearch search articles '[title="cache"]' --dialect cqp

TermQuery{title, cache} — the same tree as Lucene's title:cache.

ysearch search articles '[word="cache" & word!="cold"]' --dialect cqp

cache required, cold excluded. The exclusion is a term, so it contributes nothing to the score.

ysearch search articles '[word="incident" & team="ops"]' --dialect cqp

team is filterable and not indexed, so the second predicate lowers to an EqualityFilter. The same constraint can be supplied with --filter team=ops.

ysearch search articles '[word="status"] [word="code"]' --dialect cqp

PhraseQuery{status, code}, ordered and exact — the same tree Lucene's "status code" produces.

ysearch search articles '[word="color"] []{1} [word="blind"]' --dialect cqp

Matches color X blind for any single token X, and matches neither color blind nor color X Y blind.

ysearch search articles '[word="cold"] []{0,3} [word="cache"]' --dialect cqp \
  --fields title,url

One phrase with slop 3. cold cache, cold read cache, and cold path to the cache all match; the order is still fixed.

ysearch explain '[word="a"] []{1,3} [word="b"]' --dialect cqp

Three exact phrases under an OR — gaps of 1, 2, and 3. Count the alternatives before adding a second such gap: two of them is nine.

ysearch search articles '[word="colou?r"]' --dialect cqp

RegexTermQuery, anchored, expanded per segment into the dictionary terms it matches. The anchoring means colour and color match while colorimetry does not.

ysearch search articles '[word="colou?r"] []{0,3} [word="blind"]' --dialect cqp

The pattern expands per segment and the result stands as the first element of a slop-3 phrase.

ysearch search articles '[title="Federer" %c]' --dialect cqp

%c compiles (?i)^(?:Federer)$. Without it the query would be a case-sensitive literal against a case-folded dictionary and could not match.

ysearch explain '([word="status"] [word="code"])' --dialect cqp

A group lowers to its contents, so this is exactly the two-element phrase. Parentheses are for readability in a longer sequence.

ysearch explain 'foo'
ysearch explain '[word="foo"]' --dialect cqp

Both print the same 64-hex digest, because the digest is the SHA-256 over the canonical AST and the CQP value normalized to a literal. explain --json prints that canonical AST alongside the digest, which is what to diff when two digests differ and you want to know why.

Under the assumed schema — explain without --index — every named field is treated as indexed, so no filter can be lowered and every range is refused; and there is no analyzer, so the one-token check on a literal is skipped. Explain with --index to see exactly what search would send.

A parse or lowering refusal prints a caret rendering to stderr and exits 1: the message, the hint in parentheses on its own line, the offending source line, and ^ carets under the bytes at fault with their byte offset. An invocation error exits 2 before anything is sent.

The query.* limits that produce a refusal:

All three are query.* settings and, as configuration explains, query.* keys are read once at startup: changing one with config set is visible to config get and takes effect at the next restart.

• The Lucene grammar — the default dialect and its occurrence rules.
• Filters and projection — what happens when a predicate names a filterable field.
• Pinned epochs — how a score is computed.
• The CLI reference — search and explain flags.