lakehouse

Go to file

root 6df904a03c Phase 8: Hot cache + incremental delta updates

- MemCache: LRU in-memory cache for hot datasets (configurable max, default 16GB)
  Pin/evict/stats endpoints: POST /query/cache/pin, /cache/evict, GET /cache/stats
- Delta store: append-only delta Parquet files for row-level updates
  Write deltas without rewriting base files, merge at query time
- Compaction: POST /query/compact merges deltas into base Parquet
- Query engine: checks cache first, falls back to Parquet, merges deltas
- Benchmarked on 2.47M rows:
  1M row JOIN: 854ms cold → 96ms hot (8.9x speedup)
  100K filter: 62ms cold → 21ms hot (3x speedup)
  1.1M rows cached in 408MB RAM

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 08:37:28 -05:00

crates

Phase 8: Hot cache + incremental delta updates

2026-03-27 08:37:28 -05:00

data

Phase 8: Hot cache + incremental delta updates

2026-03-27 08:37:28 -05:00

docs

PRD v2: production roadmap with ingest, vector search, hot cache phases

2026-03-27 07:54:24 -05:00

proto

Phase 5: hardening — gRPC, observability, auth, config