lakehouse

History

root 1565f536eb Fix: job tracker field name mismatch — the overnight killer

ROOT CAUSE: Python scripts polled status.get("processed", 0) but the
Rust Job struct serialized as "embedded_chunks". Scripts always saw 0,
looped forever printing "unknown: 0/50000" for 8+ hours.

Fix (both sides):
- Rust: added "processed" alias field + "total" field to Job struct,
  kept in sync on every update_progress() and complete() call
- Python: fixed autonomous_agent.py and overnight_proof.sh to read
  "embedded_chunks" as primary key

The actual embedding pipeline was working the whole time — 673K real
chunks embedded overnight. Only the monitoring was blind.

One-word bug, 8 hours of zombie output. This is why you test the
monitoring, not just the pipeline.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-17 10:41:32 -05:00

autonomous_agent.py

Fix: job tracker field name mismatch — the overnight killer

2026-04-17 10:41:32 -05:00

copilot.py

Staffing Co-Pilot — the anticipation layer that changes everything

2026-04-17 00:19:07 -05:00

generate_demo.py

Phase 6: Ingest pipeline — CSV, JSON, PDF, text file support

2026-03-27 08:07:31 -05:00

generate_workers.py

MCP server (Bun) + 100K worker generator + lakehouse integration