lakehouse

Go to file

root 40305da654 500K scale test: 2.9M rows, sub-120ms SQL, architecture holds

Bumped upload limit to 512MB for large CSV ingests. Generated and
ingested 500K staffing worker profiles (346MB CSV → 75MB Parquet
in 5.9s).

SQL at 500K: COUNT=35ms, filter+state=67ms, aggregation=80ms,
complex filter=117ms, 10 concurrent=84ms total (10/10 pass).

HNSW memory projection: 500K vectors = 1.5GB RAM (comfortable on
128GB server). Ceiling at ~5M vectors (14.6GB) — Lance IVF_PQ
takes over beyond that as designed in ADR-019.

Hybrid search 500K SQL → 10K vector: 131ms with 6,289 SQL matches
narrowed to 5 vector-ranked results.

Total scale: 2.9M rows across all datasets (500K workers + 2.47M
staffing data).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-17 01:00:21 -05:00

crates

500K scale test: 2.9M rows, sub-120ms SQL, architecture holds

2026-04-17 01:00:21 -05:00

data

Ingest Ethereal 10K worker profiles — domain data in the substrate

2026-04-16 22:26:19 -05:00

docs

PRD: hybrid search is operational, Ethereal data integrated

2026-04-16 23:10:56 -05:00

inbox/processed

Scheduled ingest: file watcher auto-ingests from ./inbox