squirreling

1 What it does

squirreling parses and runs read-only SQL against in-memory tables or your own data sources, and returns results as a stream. A query gives you back the column names and a rows() generator; iterate it and each row hands you a bag of cells, where every cell is a function you call to get its value. Nothing in a row is computed until you ask for it, and a row that the query plan discards is never computed at all.

The reason to work this way is latency and cost. When a column is the result of a model call or a remote lookup, the difference between evaluating it eagerly and evaluating it lazily is the difference between paying for the whole table and paying for the handful of rows you keep. The engine pushes WHERE, LIMIT, and OFFSET down into the source when the source can apply them, so the expensive work happens as late and as rarely as possible.

2 Quickstart

If you do not need streaming, collect drains the generator and resolves every cell for you, returning plain row objects. It is the right call for aggregations and small result sets where you want the whole answer at once.

import { collect, executeSql } from 'squirreling'

const rows = await collect(executeSql({
  tables: { users },
  query: 'SELECT active, count(*) AS cnt FROM users GROUP BY active',
}))

executeSql takes tables (a map of name to array of row objects), the query string, and an optional functions map of user-defined functions. It returns { columns, numRows?, rows() }, where rows() is the AsyncGenerator of rows and each row exposes cells as named async thunks.

3 Async UDFs and custom sources

A user-defined function is an object with an apply that can be async and an arguments arity. Because the engine awaits it per cell and only for surviving rows, the function can do real work: call a completions endpoint, hit an API, read from a cache. The query below scores each product description with a model, and only the descriptions that reach the result are ever sent.

const rows = await collect(executeSql({
  tables: { products },
  query: 'SELECT name, AI_SCORE(description) AS score FROM products',
  functions: {
    AI_SCORE: {
      apply: async (text) => completions(`Rate the product description: ${text}`),
      arguments: { min: 1, max: 1 },
    },
  },
}))

Tables do not have to be arrays. Implement AsyncDataSource with a scan method and squirreling will hand it a ScanOptions describing the columns, the where expression node, the limit, the offset, and an AbortSignal. Your source applies what it can and reports back which predicates it handled via appliedWhere and appliedLimitOffset, so the engine knows what is left to do. That is the hook for reading Parquet or Iceberg over HTTP range requests instead of loading a table into memory.

4 Features

• Streaming output. Rows are native AsyncGenerators and cells are async thunks, so results flow as they resolve and nothing is materialized before you read it.
• Lazy per-cell evaluation. A cell runs only when called, and only for rows that survive the plan, so a model call or remote lookup costs nothing for rows you filter out.
• Async UDFs. User-defined functions can be async and call out to models or APIs mid-query, with a declared argument arity.
• Pluggable data sources. Implement scan and the engine pushes down columns, WHERE, LIMIT, OFFSET, and an abort signal, then runs whatever the source could not apply itself.
• Broad SQL. CTEs, correlated subqueries, every join kind including LATERAL VIEW EXPLODE, GROUP BY / HAVING, window functions, and set operations with their ALL variants.
• Tiny and dependency-free. 13 kb minified, zero dependencies, instant startup, built to run in a browser tab.

5 Supported SQL

The grammar covers SELECT with DISTINCT, WHERE, ORDER BY, LIMIT, and OFFSET; WITH clauses and correlated subqueries; INNER, LEFT, RIGHT, FULL, CROSS, POSITIONAL, and LATERAL VIEW EXPLODE joins; GROUP BY and HAVING; UNION, INTERSECT, and EXCEPT; and expression forms like CASE, CAST, BETWEEN, IN, LIKE, and null checks.

The built-in function library is wide. Aggregates run from COUNT and SUM through MEDIAN, PERCENTILE_CONT, APPROX_QUANTILE, and ARRAY_AGG. There are window functions (ROW_NUMBER, LAG, LEAD), string and math and trig functions, date and interval handling, a full JSON_* set, array functions, regex (REGEXP_EXTRACT, REGEXP_REPLACE, REGEXP_MATCHES), table functions (UNNEST, EXPLODE, JSON_EACH), and a spatial set built on ST_GeomFromText, ST_Intersects, ST_Within, and friends.

6 Benchmarks

The point of lazy per-cell evaluation is that an expensive cell, a model call or a remote lookup, fires only when a downstream operator demands it. Against DuckDB-wasm, the closest browser-native baseline, that shows up two ways. On a filter-bounded query (WHERE llm(...) = 1 LIMIT N) squirreling short-circuits at the LIMIT and stops calling the model, finishing about 315× faster. On a sort-bounded query (ORDER BY llm(...) LIMIT N) every row must be scored before the sort, so call counts match, but squirreling’s async operators overlap up to 256 inflight calls while DuckDB-wasm’s synchronous UDF boundary pins concurrency at one, so it still lands about 192× faster.

Numbers from A Query Engine for the Agents↗ (Table 2, N=50, 10,000-row input, 5 ms per-call latency). squirreling supplies the SQL; pair it with hyparquet to read Parquet over HTTP range requests, and together they stay under 70 KB and need no backend service between your agent and object storage.