lakehouse

Go to file

root 1565f536eb Fix: job tracker field name mismatch — the overnight killer

ROOT CAUSE: Python scripts polled status.get("processed", 0) but the
Rust Job struct serialized as "embedded_chunks". Scripts always saw 0,
looped forever printing "unknown: 0/50000" for 8+ hours.

Fix (both sides):
- Rust: added "processed" alias field + "total" field to Job struct,
  kept in sync on every update_progress() and complete() call
- Python: fixed autonomous_agent.py and overnight_proof.sh to read
  "embedded_chunks" as primary key

The actual embedding pipeline was working the whole time — 673K real
chunks embedded overnight. Only the monitoring was blind.

One-word bug, 8 hours of zombie output. This is why you test the
monitoring, not just the pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-17 10:41:32 -05:00

crates

Fix: job tracker field name mismatch — the overnight killer

2026-04-17 10:41:32 -05:00

data

Ingest Ethereal 10K worker profiles — domain data in the substrate

2026-04-16 22:26:19 -05:00

docs

PRD: hybrid search is operational, Ethereal data integrated

2026-04-16 23:10:56 -05:00

inbox/processed

Scheduled ingest: file watcher auto-ingests from ./inbox