hyparquet-writer

1 What it does

hyparquet-writer turns columnar JavaScript data into Apache Parquet, the columnar file format that tools like DuckDB, Spark, Pandas, and the rest of the data ecosystem already read. You describe each column by name, data, and optional type; the library builds the schema, encodes and compresses the values into row groups, writes the page indexes and footer, and hands back the bytes.

Because it is the writer half of hyparquet, the two share a model of the format and round-trip cleanly: what one writes, the other reads. Inside HypStack it is how telemetry becomes files you can query, and on its own it is a small way to emit Parquet from any JavaScript program without leaving the language. The whole library is one dependency, hyparquet itself, and no native binaries.

2 Quickstart

There are three entry points, all taking the same columnData shape. Pick the one that matches where the output should go.

// 1. to an ArrayBuffer in memory
import { parquetWriteBuffer } from 'hyparquet-writer'
const buffer = parquetWriteBuffer({ columnData })

// 2. to a local file (Node.js)
import { parquetWriteFile } from 'hyparquet-writer'
parquetWriteFile({ filename: 'example.parquet', columnData })

// 3. low-level: stream into a ByteWriter you control
import { ByteWriter, parquetWrite } from 'hyparquet-writer'
const writer = new ByteWriter()
parquetWrite({
  writer,
  columnData,
  codec: 'SNAPPY',
  rowGroupSize: 1000000,
})
const buffer = writer.getBuffer()

parquetWriteBuffer and parquetWriteFile are thin wrappers over parquetWrite, which writes into any Writer. ByteWriter is the in-memory implementation; supply your own to send bytes somewhere else.

3 Schema and types

If you leave type off a column, the writer infers it from the data. When you need control, set the type explicitly or pass overrides through schemaFromColumnData, for example to mark an integer column as unsigned:

import { schemaFromColumnData } from 'hyparquet-writer'

const schema = schemaFromColumnData({
  columnData,
  schemaOverrides: {
    unsigned_int: { name: 'unsigned_int', type: 'INT32', converted_type: 'UINT_32' },
  },
})

Supported types cover the common primitives, BOOLEAN, INT32, INT64, FLOAT, DOUBLE, BYTE_ARRAY, STRING, JSON, and TIMESTAMP, alongside UUID, FLOAT16, GEOMETRY, GEOGRAPHY, and the VARIANT type for semi-structured JSON-like values. VARIANT can shred chosen fields out into typed sub-columns, so values you query often are stored as real columns while the rest stay flexible:

parquetWriteBuffer({
  columnData: [{
    name: 'event',
    data: [{ type: 'login', user: 'alice' }, { type: 'click', x: 10 }],
    type: 'VARIANT',
    shredding: { type: 'STRING', user: 'STRING' },
  }],
})

4 Features

• No native binaries. Plain JavaScript with a single dependency on hyparquet, so the same writer runs in the browser, on Node, and in workers, with nothing to compile per platform.
• Buffer or file output. parquetWriteBuffer returns an ArrayBuffer, parquetWriteFile writes to disk on Node, and parquetWrite targets any Writer you pass in.
• Inferred or explicit schema. Types are detected from the data and can be overridden through schemaFromColumnData, including converted types like unsigned integers.
• Compression built in. Snappy by default, with a codec option and a compressors hook to supply your own codec implementations.
• Statistics and page indexes. Column statistics and offset and column indexes are written by default, so readers can skip row groups and pages they don’t need.
• Tunable layout. rowGroupSize, pageSize, per-column encoding (PLAIN, RLE, DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT, and more), nullability, and custom kvMetadata are all exposed.
• Round-trips with hyparquet. Built to the same model of the format as the reader, so files written here read back cleanly in hyparquet and any other Parquet tool.