icebird

1 What it does

icebird reads and writes Iceberg tables directly from JavaScript, on Node or in the browser, with no native binaries and no JVM. Reads cover v1, v2, and v3 tables; writes cover v2 and v3, including v3 deletion vectors. Parquet and Avro storage are both supported, and Parquet I/O runs through hyparquet and hyparquet-writer, so every Parquet type and compression codec a table might use comes along for free.

The read path is the point. icebird starts from the table’s metadata JSON, walks the manifest list to the manifests, and uses partition and column statistics to skip any data file a query cannot match. What remains is read with range requests, so the bytes you pay for scale with the answer rather than the table. You can pin a specific snapshot for time travel by passing a metadata file name, and you can hand in your own metadata object to avoid re-reading it across calls.

2 Quickstart

The minimal read takes a table URL and a row range. To avoid resolving metadata on every call, read it once with icebergMetadata and pass it back in:

import { icebergMetadata, icebergRead } from 'icebird'

const metadata = await icebergMetadata({ tableUrl })
const data = await icebergRead({ tableUrl, metadata, rowStart: 0, rowEnd: 10 })

For private buckets, icebird does not hardcode an auth scheme. You supply a resolver and a lister: the resolver turns a path into a fetchable URL, the lister enumerates files. There are built-ins for bearer-token headers and for SigV4 signed S3 requests, and the same hooks point at Cloudflare R2 or MinIO by passing an endpoint and pathStyle: true:

import { icebergRead, s3SignedResolver } from 'icebird'

const resolver = s3SignedResolver({ accessKeyId, secretAccessKey, region: 'us-east-1' })
const data = await icebergRead({ tableUrl: 's3://my-bucket/warehouse/orders', resolver })

When you want to ask a question rather than scan a range, icebird ships a SQL engine built on squirreling that streams rows lazily across one or more tables. Identifiers with spaces and dotted multi-segment namespaces are quoted:

import { collect, icebergQuery, restCatalogConnect } from 'icebird'

const catalog = await restCatalogConnect({ url: 'https://catalog.example.com' })
const result = await icebergQuery({
  catalog,
  query: 'SELECT "Breed Name", "Popularity Rank" FROM "java.bunnies" ORDER BY "Popularity Rank"',
})
const rows = await collect(result)

3 Catalogs and versions

A table URL is enough on its own, but most deployments name tables through a catalog. icebird supports two: a file-based catalog that reads and writes metadata pointers directly in the bucket, and a REST catalog that speaks the Iceberg REST protocol. Connect once, load a table by namespace and name, and read from the location the catalog reports:

import { icebergRead, restCatalogConnect, restCatalogLoadTable } from 'icebird'

const ctx = await restCatalogConnect({ url: 'https://catalog.example.com' })
const { metadata } = await restCatalogLoadTable(ctx, { namespace: 'analytics', table: 'orders' })
const data = await icebergRead({ tableUrl: metadata.location, metadata })

Write support is newer and covers v2 and v3 tables. You can create a table from a schema, append records, delete rows by position (as v3 deletion vectors or v2 Parquet delete files), manage refs and snapshots, and compact small files into sorted, non-overlapping ones with icebergRewrite. The same calls work against either catalog:

import { fileCatalog, icebergCreateTable, icebergAppend } from 'icebird'

const catalog = fileCatalog({ resolver })
const schema = {
  type: 'struct', 'schema-id': 0,
  fields: [
    { id: 1, name: 'id', required: true, type: 'long' },
    { id: 2, name: 'name', required: false, type: 'string' },
  ],
}

await icebergCreateTable({ catalog, tableUrl, schema })
await icebergAppend({ catalog, tableUrl, records: [{ id: 1n, name: 'alice' }] })

4 Features

• Reads v1, v2, and v3 tables. One client across every Iceberg spec version in use, with Parquet and Avro data and manifest files both handled.
• Scan pruning over range requests. Partition and column statistics narrow a query to the files it can match, and only those Parquet files are read, byte-range by byte-range, through hyparquet.
• SQL over one or more tables. A streaming engine built on squirreling evaluates queries lazily across a catalog, so large results never have to materialize at once.
• File and REST catalogs. Point at a bucket directly or connect to an Iceberg REST catalog; the read and write calls are the same either way.
• Writes for v2 and v3. Create tables, append records, run position and equality deletes including binary deletion vectors, manage snapshots and refs, expire old snapshots, and compact files with a global sort.
• Time travel. Pin any prior table state by naming its metadata file, reading the exact snapshot you ask for.
• Bring your own transport. Resolver and lister hooks supply auth and file listing, with built-ins for bearer tokens and SigV4, and support for R2 and MinIO via path-style endpoints.
• No runtime to install. Pure JavaScript on Node or in the browser, no JVM, no native addons, built on hyparquet and hyparquet-writer.

Source on GitHub↗ README↗