10,000 staffing worker profiles from profit/ethereal repo. Flattened
JSON → CSV → Parquet. Indexed on HNSW (9.5s) + Lance IVF_PQ (7.2s).
SQL hybrid verified: forklift operators in IL with reliability > 0.8
returned exact matches. Vector search alone missed the state filter —
confirms the hybrid SQL+vector routing need from quality eval.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three-tier evaluation:
1. NL→SQL with verifiable ground truth (10 questions): 7/10 (70%)
2. RAG with LLM reranker (5 questions): 4/5 (80%)
3. Self-assessment calibration: 2.8/5 avg, NOT calibrated
Real problems surfaced:
- qwen2.5 generates `WHERE vertical = 'Java'` instead of
`WHERE skills LIKE '%Java%'` without few-shot schema examples
- DataFusion-specific SQL quirks (must SELECT the COUNT in
GROUP BY queries) trip the model without explicit instruction
- Vector search can't do structured filtering (city, status) —
needs hybrid SQL+vector routing
- Self-assessment is uncalibrated: wrong answers score higher
than correct ones (3.0 vs 2.8)
Fixes validated:
- Few-shot examples fix NL→SQL accuracy from 70% → ~90%
- Reranker stage works but needs more diversity in results
Also includes lance_tune.py IVF_PQ parameter sweep script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python agent that exercises the full Lakehouse substrate as a real
consumer would: ingests 10 Postgres tables (1,356 rows), embeds 5,415
chunks into 2 vector indexes, creates hot-swap profiles (Parquet+HNSW
with qwen2.5 vs Lance IVF_PQ with mistral), runs stress queries
across SQL + vector search + RAG, reads its own error pipeline to
generate recursive test scenarios, and iterates.
50/50 tests pass across 2 iterations with zero errors. Error pipeline
flushes failures back to the lakehouse as a queryable dataset so the
next iteration can target weak spots.
The agent IS the proof that the substrate works end-to-end: ingest →
embed → index → search → generate → profile swap → iterate. Every
capability we built today gets exercised in one script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- lakehouse.service: release gateway on :3100, auto-restart
- lakehouse-sidecar.service: Python FastAPI on :3200, auto-restart
- lakehouse-ui.service: WASM file server on :3300, auto-restart
- All enabled at boot (multi-user.target)
- scripts/serve_ui.py for systemd-compatible file serving
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
