hypgrep

1 What it does

hypgrep constructs a compact n-gram index for a Parquet file, so you can run case-insensitive, grep-style substring search over a large dataset without a search server. You store the dataset on S3 or locally, generate one index file beside it, and query through HTTP range requests that download only the pages a given search needs.

Because the index is itself a Parquet file, it travels with your data and needs no separate service to host it. The same query path runs in the browser, on Node, or in a serverless function: the client reads index pages to find candidate rows, then reads those rows out of the source file. This is the full-text piece of HypStack, the search counterpart to hypvector for vectors and squirreling for SQL.

2 Quickstart

Build an index once from a Parquet file. hypgrep reads the source with hyparquet and writes the index with hyparquet-writer, both pure JavaScript.

// build the index once
import { createIndex } from 'hypgrep'
import { asyncBufferFromFile } from 'hyparquet'
import { fileWriter } from 'hyparquet-writer'

const sourceFile = await asyncBufferFromFile('dataset.parquet')
const indexFile = fileWriter('dataset.index.parquet')
await createIndex({ sourceFile, indexFile })

With the index uploaded next to the data, query it straight from a URL. parquetFind streams rows that contain the substring, reading only the bytes each match requires:

// grep-style substring search from the client
import { parquetFind } from 'hypgrep'

for await (const row of parquetFind({
  query: 'serverless',
  url: 'https://s3.hyperparam.app/hypgrep/wiki_en.parquet',
})) {
  console.log(row)
}

For ranked results, parquetSearch orders matches by occurrence count and treats whitespace-separated terms as AND. From the command line, the same queries run through npx hypgrep search, including regular expressions:

$ npx hypgrep search dataset.parquet 'rhythm' --limit 5
$ npx hypgrep search dataset.parquet '/eigen.+value/i'

3 Features

• No server. The index is a Parquet file in object storage; clients read it with HTTP range requests, so there is nothing to keep warm between queries.
• Grep semantics. Case-insensitive substring matching over the whole dataset, the way you would expect from grep.
• Regex queries. Pass a pattern like /eigen.+value/i and hypgrep extracts the literal parts to narrow the search before matching.
• Ranked search. parquetSearch ranks rows by occurrence count and ANDs whitespace-separated terms for multi-word queries.
• Runs anywhere. Browser, Node, or a serverless function. Point at a URL for S3, or use asyncBufferFactory for local files.
• Custom filtering. A rowFilter callback lets you drop or keep candidate rows during a search.
• Pure JavaScript, MIT. Built on hyparquet and hyparquet-writer, no native binaries, part of HypStack.

4 Benchmarks

Full-text search over 3,199,860 real LLM conversations (WildChat-4.8M), the JSON conversation stored verbatim (14.7 GB of Parquet), searched with 109 queries across 15 shapes (tokens, phrases, JSON structure, code, Unicode, and regex) run against the same data on every engine and measured as time-to-first-100 rows from one client. Two things separate the engines: whether they can answer a query at all, and what they cost.

Can it answer the query? Only hypgrep and Athena handle all 109; Quickwit, a tokenized engine, cannot do literal-punctuation substrings or regex and fails nearly half.

Speed, footprint, and cost. The always-on engines answer fast but bill a box 24/7; the client and serverless engines cost per query, where an index is the difference between pennies and dollars.

^† Postgres supports the backreference regex but times out (>60 s), since a trigram index cannot accelerate it. ^‡ Warm median time-to-first-100 rows over the 58 queries every engine can answer (hypgrep, the always-on engines, and Athena); DuckDB over S3 across all 109 (no warm cache, re-scans each query). hypgrep warm shown; first-visit cold ≈ 1.6 s. Costs are all-in us-east-1 on-demand (see cost-model.mjs): fixed = S3 storage, or the cheapest instance that held the measured latency plus disk; per query = S3 GET+egress, or Athena’s $5/TB scan.

Where each engine wins and breaks down. Warm latency by query shape: green is fast, red is slow, and ✗ marks a shape the engine cannot answer at all.

^§ DuckDB has no index and no warm cache, so over S3 it re-scans the column every query (median 30 s, up to 127 s to full-scan for a selective term). * = the engine cannot answer one query in that shape (median is of the rest).