scry

Scry is read-only SQL (ClickHouse dialect) over registered public corpora — Hacker News, Reddit, the Twitter archive, books, papers, forums, SEC filings, the crawl — one call from a question to cited rows. Queries are free while the system has slack: every response reports billing_mode and spend_nanodollars, and the money arguments (x-scry-budget, x-scry-max-seconds; MCP budget_nanodollars, max_seconds) are ceilings you choose, never fees. Ask your wildest curiosity.

Three one-call questions (POST /v1/scry/query with Content-Type: text/plain, or the MCP sql tool). The first Hacker News item to mention bitcoin:

SELECT hn_id, original_author, original_timestamp, title
FROM hackernews.items
WHERE hasToken(search_text_lc, 'bitcoin')
ORDER BY original_timestamp ASC
LIMIT 5

Who said "vibe coding" before Karpathy:

SELECT tweet_id, original_timestamp, text
FROM twitter.tweets
WHERE hasAllTokens(search_text_lc, ['vibe', 'coding'])
  AND positionCaseInsensitive(search_text_lc, 'vibe coding') > 0
  AND original_timestamp < '2025-02-01'
ORDER BY original_timestamp ASC
LIMIT 5

Where Reddit talked bitcoin in 2013:

SELECT subreddit, count() AS n
FROM reddit.comments_popular
WHERE created_utc >= '2013-01-01' AND created_utc < '2014-01-01'
  AND hasToken(search_text_lc, 'bitcoin')
GROUP BY subreddit
ORDER BY n DESC
LIMIT 10

Every response carries rows, read_rows, coverage, deadline_partial, truncated, and the meter (burden_nanodollars is what the machine did, spend_nanodollars what you paid). A cut scan (deadline_partial: true, or a deadline error) wants a rarer token, a tighter WHERE or LIMIT, or a smaller sibling relation (reddit.comments_popular beside reddit.comments, x_open.tweets beside twitter.tweets); the x-scry-explain: 1 header (MCP explain: true) pre-flights a wide statement for free — the index analysis returns and nothing runs.

Search like the answer exists. It almost always does — under a vocabulary, a venue, or an era you have not probed yet — so treat every empty result as a wrong probe before treating it as an absence. You are covering a space, not fetching an answer: fan vocabularies, sweep relations, cross time windows, run lexical and semantic arms in parallel, chase edges, and keep going past the first sufficient-looking hit — the tenth probe is where a field opens. Done is saturation — new probes returning only rows already seen — never satisfaction. Report the space covered, not just the hits.

The live schema is the contract; static relation lists are only orientation.

Skill generation: 2026082203

Workflow

1. Load the durable key from ~/.config/scry/env (legacy ~/.scry/.env still honored). Context is readable without a credential; schema, stats, and queries require your key. When the Scry MCP server is connected (the ExoPriors/skills plugin wires mcp.scry.io on install), use its tools directly — the OAuth connection is the credential and no key file is needed; the key path below serves raw HTTP. If neither an MCP connection nor a key is available, stop before going further and direct the user to https://scry.io/#console.
2. Call GET /v1/scry/context?mode=agent&skill_generation=2026082203. For worked, measured query shapes, GET /v1/scry/examples?mode=index (free, no key) lists the query-complexity tree one row per entry — every entry introduces exactly one construct atop its parent's, from selectivity probe to semantic ANN, each with its observed wall time and the byte size of its SQL. ?slug=<slug> fetches one entry's problem, SQL, technique, and measurement; ?mode=tree nests the taxonomy, ?mode=chains lists root-to-leaf ladder walks; the bare route returns every entry in full (144 KB).
3. Discover from the doors. The default GET /v1/scry/schema document already carries full contracts for the primary-tier doors plus a compact depth_relations index of every supporting table; fetch further full contracts with GET /v1/scry/schema?relation=<name>[,<name>], or ?mode=index for the whole catalog as one relation | tier | extent | lag | purpose line per relation (both also exposed as the MCP schema tool's mode and relation arguments; the MCP default is the index and mode="contract" carries the product contract, census, and live statistics). Schema discovery is also one SQL call: scry.relations and scry.columns are the same catalog served as relations you can filter and join, e.g. SELECT relation FROM scry.columns WHERE name = 'author_id' LIMIT 100. Use only relations and helper functions returned there, and read each relation's query_guidance block — filter_columns_first, indexed_predicates, coverage_note — before writing the first predicate: it names the indexed access paths. Never guess column names from memory of similar sources — a wrong column returns the relation's real column roster in the error, so one failed query self-corrects in one step; an unknown relation returns the nearest registered names.
4. Send one SQL statement to POST /v1/scry/query with Content-Type: text/plain.
5. Semantic search: mint a named query vector with POST /v1/scry/embed {text, name}, then use it as the unquoted @name inside scry_vector_topk_distance; full patterns are in references.md § Scry query patterns. Query text craft dominates every other parameter: embed answer-shaped, exuberant passages — the paragraph you hope to find — never keyword stubs, and fan out registers (references.md § Writing the query text). The same endpoint takes {expression, name} to compose stored handles (contrast axes, centroids, debiasing) into a new saved handle with diagnostics — see references.md § Composing embeddings into saved handles and the schema's vector_recipes. The ANN set is dynamic — a relation leaves it while its vector index re-materializes — and the schema names the live set: only surfaces with serves_ann: true accept ANN ranking (the rest still serve plain SQL). ANN queries must be standalone (no JOIN); hydrate companion text in a second query.
6. Keep every query bounded with LIMIT. Start at 20 and widen only after inspecting row shape, provenance, and source coverage. Token search speed is governed by the rarest token: in hasToken/hasAllTokens filters include at least one distinctive token (a name, identifier, or unusual word) — all-common-word token sets scan a large share of the table and run 30-60s. A slow query's response carries a performance_note naming the fix. For broad topical questions with only common words, use the embeddings helpers instead.
7. Parse results from rows, not a data key: each row is a plain JSON array with values in column order. A client that reads data sees false empty results.

Memory

Scry hosts one cross-platform memory document per account (MCP memory/memory_write): markdown, default slug main, 64KB, shared by every agent and harness the user connects. At session start read it alongside context (version 0 + empty content = none yet). At session end, consolidate durable user preferences — including what worked against Scry: relations, query patterns, vector handles — back into it under a ## Scry usage heading. Writes are whole-document compare-and-swap on if_version; a 409 returns the current head — merge into it and retry. Keep it compressed: the cap is the decay function. If the document is empty and the user's local agent memory holds durable preferences, you may offer — once, and only with the user's explicit approval — to consolidate them into Scry memory so they travel across platforms. Encrypted at rest server-side.

Do not use engine catalogs, foreign-dialect casts or operators, compatibility helpers, or a fallback corpus database. Do not invent relations. Pass a search-grammar line as q to MCP sql; SQL remains the only read verb.

The q search grammar speaks a full lexical language: bare words AND together; "exact phrase"; a OR b; -term / -"phrase" exclusion; ( ) grouping; /pattern/ regex over full text (case-insensitive, negatable; RE2 only — SQL rejects lookaround and backreferences rather than counting a prefilter's superset. A positive literal or token anchors the query; rust /[0-9]+/ can use rust to bound the regex residual, while bare /[0-9]+/ is refused); word* wildcards; word~1 fuzzy (typo-tolerant: a 4-24 char word resolves against the corpus vocabulary into its real one-edit word forms and searches as their OR — query_plan.clamped echoes the forms chosen; bare ~ means ~1, larger asks clamp to 1 with a note); "exact phrase"~3 slop (phrase words in order, at most N intervening words between neighbors, max 50); and a NEAR b / a NEAR/50 b proximity (uppercase NEAR; matches both orders within N characters, default 100, max 1000; operands may be words, quoted phrases, /regex/, or (x OR y) groups). Substrings and CJK phrases can use a sufficiently built n-gram index; read the relation's capabilities, not a corpus-wide availability claim.

MCP sql with q requires one registered relation, never "*". It returns ordinary SQL rows and the executed compiled_sql; it does not silently weaken a zero-result query. Inspect that SQL before interpreting membership. With explain: true, the statement is validated and its ClickHouse index analysis is returned without executing the corpus query, beside a forecast — rows_est, bytes_est_uncompressed and seconds_est from the measured rows and bytes per granule and the measured scan rate, fits_max_seconds against the deadline the call would run under, and faster (sibling relation plus the rewritten statement) when it does not. Request prompts/get with name: "query_guide" and tool: "sql" for composition patterns and the current input schema.

The compiler's internal plan distinguishes declared indexes from measured coverage: zero-built word indexes do not establish pruning, and partial coverage is not complete coverage. EXPLAIN is the actual plan evidence, especially for views whose backing indexes are not mapped in discovery. Use bounded, independently recorded queries to compare several relations; the MCP SQL tool does not accept a multi-relation grammar sweep.

The grammar is also a first-class SQL operand: inside any POST /v1/scry/query statement, scry_lex('<line>') expands server-side into exactly the predicate sql with explain would return for the statement's one registered relation — so WHERE scry_lex('"scaling laws" -toy'), countIf(scry_lex('/GPT-[0-9]/')) AS hits, and GROUP-BY histograms over a lexical cohort are plain SQL. An optional second argument pins the text expression (scry_lex('rust', title)); an operator the relation cannot express is a hard error, never a silent drop. At most 8 calls per statement; one registered relation per statement.

Lexical recipes

Reuse shared term instruments with scry_recipe('<slug>'[, text]) for membership and scry_recipe_score('<slug>'[, text]) for token-weighted score. Use scry_recipe_density('<slug>'[, text]) for weighted term occurrences per 1,000 characters across token, phrase, and regex members. Discover them with MCP recipes; publish a complete measured version with recipe_write and the returned head version as if_version. Derive candidates read-only with recipe_derive, then curate noise, measure the instrument, and publish through recipe_write. Write a recipe when you derived at least five surface forms, or when a polarity instrument survives reading 20 matches per cohort. Read those matches before publishing, keep provenance and measurements with the terms, and treat the stance as part of the recipe's identity. The seeded shelf and choosing guidance live in references.md § The recipe shelf; the author/thread/time/graph quantifier shapes that recipes plug into are references.md § The quantifier chain; the full plane-by-plane operator map — quorum and frequency gates, named quantifiers, Allen span relations, life-history regex, epistemic operator families — is references.md § The operator space.

Composing recipes has an operand: scry_recipe('a - b') difference, scry_recipe('a & b') intersection, scry_recipe('a ^ b') exclusive-or — whitespace around the operator, one operator kind per call (chains like a - b - c fine, mixing refused), ^ takes exactly two operands, and score/density each measure one slug at a time. The expansion keeps a positive index-engaging leaf in front by construction, so the NOT inside -/^ rides the residual. The same booleans remain writable by hand (scry_recipe('hedging') AND NOT scry_recipe('certainty')), and the contrast ratio countIf(scry_recipe('a')) / countIf(scry_recipe('b')) per cohort cancels base rates. A composition worth reusing gets published as its own recipe (derived_from naming the algebra) — that also makes it scoreable. Terms may carry form: "regex" (RE2, compiled to match()): give a regex-bearing recipe token or phrase recall leaves beside the patterns or it evaluates as a scan. Disjointness of two instruments is a property to measure, not assume: countIf( scry_recipe('a & b')) beside each count says how much they overlap on the relation you quantify over, and a stance pair that overlaps heavily is one recipe with a missing stance.

Guiding knobs beyond the query text: snippet_chars (64-1200, default 240) widens each result's served context window; max_per_source (>=1) caps any one source's share of the page; limit, sources, kinds, from/to bound the pool. In-query, NEAR/50 sets the proximity window in characters, "phrase"~3 the slop window in words, and word~1 the edit-distance window for typo tolerance.

Any community- or venue-scoped question starts from an enumerated source set: run the inexpensive partition-enumeration query on the candidate relations (e.g. SELECT source, count() FROM forums.posts GROUP BY source; subreddit and list catalogs likewise) and report which sources were consulted and which excluded. Missing a source that was one GROUP BY away is the corpus's most common research failure.

For multi-step research — several hypotheses, several sources, or any ask where missing vocabulary would silently distort the answer — follow references.md § Deep research operations: fan out lexical probes, keep a probe ledger, verify the written report against the ledger, and end in a durable artifact. Surface selection starts with schema: the compact catalog plus per-relation stats is the shortlist; enumerate partition values yourself rather than delegating the plan.

For any study that compares cohorts or tests a hypothesis (who does X more, does trait A predict behavior B), follow references.md § Comparative study design before writing the first query: pre-state the refuter, audit selection–outcome independence, and climb no higher on the interpretation ladder than the instrument licenses.

For academic work — finding papers, tracing citation neighborhoods, and above all reviewer discovery — follow references.md § Academic papers and reviewer discovery. Reviewer discovery is a coverage problem: enumerate every candidate pool with its denominator, keep a candidate ledger, screen conflicts, rank on explicit axes, and stop on pool exhaustion, never on "enough names."

Conduct

Every claim ships with its source row or it does not ship. Prefer the denominator: report what was searched — relations, sources, probe terms — not only what was found. When sources conflict, resolve the conflict or report it; never average it away. Small bounded probes cast wide before expensive queries close. Done means the written answer is checked against the queries that actually ran.

Fixpoint programs (recursive graph search)

WITH RECURSIVE is served on /v1/scry/query (body must be anchor UNION ALL step; read the CTE only in the step's FROM/JOIN, never in a subquery) — but every iteration rescans the joined relation (~1.8 s per step on openalex.works), so declare x-scry-max-seconds. For frontier-pruned walks — citation closures, filtered multi-hop expansions, walked sets ranked semantically — send a program instead of SQL: POST /v1/scry/query with a JSON body {"program": {...}} (MCP datalog). A sql atom is one statement (LIMIT <= 50000, the relation cap): alone in its body it seeds a set from column id; after a rel it hydrates that relation — the rows it returns keep their parent/depth and gain the other columns as attrs (the statement must read the relation: WHERE <key> IN {name} — the keys are hn_id, post_key, tweet_id, and the OpenAlex id URL). Inside a sql atom, {name} binds an already-evaluated relation as a query-scoped table of its ids (OpenAlex ids retain their full URLs), bounded by the 50k relation cap.

Walk then hydrate:

{
  "relations": {
    "seed": {"bodies": [[{"sql": "SELECT hn_id AS id FROM hackernews.items WHERE scry_lex('claude code') AND hn_type = 'story' ORDER BY original_timestamp DESC LIMIT 100"}]]},
    "thread": {"bodies": [[{"rel": "seed"}], [{"rel": "thread"}, {"edge": "hackernews.children"}]]},
    "final": {"bodies": [[{"rel": "thread"}, {"sql": "SELECT hn_id AS id, original_author, left(payload, 200) AS text FROM hackernews.items WHERE hn_id IN {thread} LIMIT 500"}]]}
  },
  "out": ["final"],
  "depth": 2
}

Aggregate the same thread by replacing final with the following definition (ids are handles, with kind absent unless an edge consumes them):

{"bodies": [[{"sql": "SELECT original_author AS id, count() AS replies FROM hackernews.items WHERE hn_id IN {thread} GROUP BY id ORDER BY replies DESC LIMIT 50"}]]}

A sql seed runs as your own statement, so seed from keyed reads; for an account's tweets, use twitter.tweets_of from its account id instead of filtering twitter.tweets by author_id. A program is named relations (sets of node ids) built from a closed atom vocabulary — ids seeds, ann (top-k probe from an embed handle), rel (a body naming its own relation recurses), edge (graph steps: OpenAlex references/cited_by; twitter twitter.replies/twitter.quotes + inverses; hackernews.children/ parent/story_items; forums.children/parent/thread — and pivots that change what a node is: openalex.authors/institutions/works_of, twitter.by/following/followers/tweets_of, hackernews.by/items_of, forums.by/posts_of, github.repos_of, bluesky.by/posts_of, youtube.uploader/commenters, tiktok.videos_of, instagram.posts_of, crawl.urls_of; rows carry kind; an unknown edge name returns the catalog with measured costs), filter (in-walk attribute prune — changes what gets expanded and billed), in (intersection), not_in (stratified negation; on a recursive body it prunes the walk itself) — plus an optional per-relation "rank": {handle, k} ordering final rows by exact distance to a handle (OpenAlex only); a relation left out of out ships only its per-depth counts, zero egress (out: [] is the census). Every evaluation step is one ordinary metered statement under your own key; depth (default 3) and 50k-row caps bound the walk; the envelope returns {id, kind, parent, depth} provenance rows (a sql atom's other columns ride in attrs), counts for every relation (an empty seed set shows counts.seed.rows = 0), a meter with per_statement, and truncations[] (empty = fixpoint over the graph the index holds). Prefer rank over intersecting a walk with a global ANN top-k — measured near-empty overlap at corpus scale. Rank is terminal: it orders a relation's final rows after the walk, so put it on the last relation (the hydrating one), not on a set another relation reads.

Bound bodies give a relation tuples and variables: declare "vars": ["S", "W"] and every body opens with a driving {"rel": {"name": "seed", "vars": ["S"]}} (naming its own relation recurses), then up to four {"edge": {"name": "references", "vars": ["S", "W"]}} joins whose source var is already bound, {"rel": {name, vars}} joins and {"not": {name, vars}} anti-joins against evaluated relations, and filters either on the var an edge produces ({"filter": {"on": "W", "col": "publication_year", "op": ">=", "val": 2020}}) or between two vars ({"filter": {"on": "B", "op": "!=", "var": "A"}}). Every head/negated/filtered var needs an earlier positive binding; kinds come from edges, not sql; cited_by goes last; legacy atoms consume only unary bound relations. Rows return as {tuple, parent, depth} plus an envelope schemas map. A k-edge chain nests its prefilters (three HN edges in one body read ~88M rows), so keep bodies to one or two edges when intermediate sets are large. Coauthors in one step:

{"relations": {"a": {"bodies": [[{"ids": ["A5000000036"]}]]},
  "co": {"vars": ["B"], "bodies": [[{"rel": {"name": "a", "vars": ["A"]}}, {"edge": {"name": "openalex.works_of", "vars": ["A", "W"]}}, {"edge": {"name": "openalex.authors", "vars": ["W", "B"]}}, {"filter": {"on": "B", "op": "!=", "var": "A"}}]]}},
 "out": ["co"]}

The MCP tool contract carries ten worked templates, including a seed-keyed citation closure and an anti-join.

First-contact audit rail (datalog)

The datalog door improves by rotation: outside models meet it docs-only, attempt real research tasks, and report frictions; the lead folds the real ones (executor fix over wording), builds on colo2, lands, deploy-applies, and live-verifies. Rotations 1–13 (2026-09-09..11) ran this way.

• Brief template bin/datalog-audit-brief.md (fill the numbered tasks; keep the docs-only rule, the colo2 /tmp/hunt-client/mcp.py mechanics, the ≤25-call cap, and "the report file is the deliverable — final chat text is not seen"). Launcher bin/datalog-audit.sh <model> <brief.md> <log> runs headless OMP from /tmp/xp-dl-probes; launch as (nohup bin/datalog-audit.sh <model> brief.md log >/dev/null 2>&1 &). Models that have worked: gemini-3.8-flash (fast, fabricates quotes and misreads figures), openrouter/x-ai/grok-4.6 (careful, best cost tables), openrouter/deepseek/deepseek-v4-flash-0731 (slow, hours). Liveness pgrep -f "auditN-"; progress ls auditN-*.py | wc -l.
• An auditor's figure or quote is a lead, never a measurement: re-measure before it lands (a quoted guide passage did not exist; an "11.5M rows" figure was a LIMIT-10 early exit of a 510M-row read; a "16M rows" was the hit range). Two auditors agreeing is still a lead.
• Burden (nanodollars) swings >2x between identical runs while read_rows holds; guide prose carries read-row magnitudes as the pre-pay lesson, never dated nanodollar or seconds figures.

Lexical range

Embeddings are for missing vocabulary. When you know the words — names, handles, idioms, error strings, catchphrases — token search composed with plain SQL is sharper and faster, and it composes further: GROUP BY, joins, and window functions turn retrieval into measurement. The corpus is a programmable instrument; the searches worth running are the ones only you would think to compose. Shapes that reward that creativity:

• Earliest attestation: hasToken(search_text_lc, 'term') on internet.text ordered by original_timestamp ASC — when and where a phrase first appeared.
• An author's written history:

<!-- Content truncated for initial SEO render. Open the source file tab for the full file. -->