root 6a532cb248 Background job system for embedding — fixes 100K timeout ...

- JobTracker: create/update/complete/fail jobs with progress tracking
- POST /vectors/index now returns immediately with job_id (HTTP 202)
- Embedding runs in tokio::spawn background task
- GET /vectors/jobs/{id} returns live progress (chunks embedded, rate, ETA)
- GET /vectors/jobs lists all jobs
- Progress logged every 100 batches with chunks/sec and ETA
- 100K embedding job running successfully at 44 chunks/sec
- System stays responsive during embedding (queries in 23ms)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 09:03:07 -05:00

crates

Background job system for embedding — fixes 100K timeout

2026-03-27 09:03:07 -05:00

data

Background job system for embedding — fixes 100K timeout

2026-03-27 09:03:07 -05:00

docs

PRD v3: future-proofing roadmap — event journal, rich catalog, tool registry

2026-03-27 08:57:29 -05:00

proto

Phase 5: hardening — gRPC, observability, auth, config

2026-03-27 06:37:07 -05:00

scripts

Scale test: 2.47M rows + 10K vector index benchmarked

2026-03-27 08:31:37 -05:00

sidecar

Add __pycache__ to gitignore

2026-03-27 05:54:09 -05:00

tests

Phase 2: DataFusion query engine over Parquet

2026-03-27 05:48:20 -05:00

.gitignore

Add __pycache__ to gitignore

2026-03-27 05:54:09 -05:00

Cargo.lock

Background job system for embedding — fixes 100K timeout

2026-03-27 09:03:07 -05:00

Cargo.toml

Phase 7: Vector index + RAG pipeline

2026-03-27 08:12:28 -05:00

justfile

Phase 4: Dioxus frontend with dataset browser and SQL query editor

2026-03-27 06:24:15 -05:00

lakehouse.toml

Phase 5: hardening — gRPC, observability, auth, config

2026-03-27 06:37:07 -05:00

Description

Rust-first object storage system

6.3 GiB

Languages

TypeScript 38.4%

Rust 35.8%

HTML 13.9%

Python 7.8%

Shell 2.1%

Other 2%