lakehouse

profit/lakehouse

Fork 0

Commit Graph

Author	SHA1	Message	Date
root	468798c9ac	/spec: technical specification — 11-chapter README-equivalent J's ask: explain the full architecture so someone reading a README can dispute it or recreate it. The repo isn't public yet; this page IS the spec until it is. Ch1 Repository layout — 13 crates + tests/multi-agent + docs + data, with owned responsibility and file path per crate. Ch2 Data ingest pipeline (8 steps) — sources (file/inbox/DB/cron), parse+normalize with ADR-010 conservative typing, PII auto-tag, dedup, Parquet write, catalog register with fingerprint gate, mark embeddings stale, queryable immediately. Ch3 Measurement & indexing — row count / fingerprint / owner / sensitivity / freshness / lineage per dataset. HNSW vs Lance tradeoff table with measured numbers (ADR-019). Autotune loop. Per-profile scoping (Phase 17). Ch4 Contract inference from external signal — Chicago permit feed → role mapping → worker count heuristic → timeline → hybrid search with boost → pattern discovery → rendered card. All pre-computed before staffer opens UI. Ch5 What a CRM can't do — 11-row comparison table of capabilities. Ch6 How it gets better over time — three paths: - Phase 19 playbook boost (full math) - Pattern discovery meta-index - Autotune agent Ch7 Scale story: 20 staffers, 300 contracts, midday +20/+1M surge - Async gateway + per-staffer profile isolation + client blacklists - 7-step surge handling flow (ingest, stale-mark, incremental refresh, degradation, hot-swap, autotune re-enter) - Known pain points: Ollama inference serial, RAM ceiling ~5M on HNSW (mitigated by Lance), VRAM 1-2 models sequential, playbook_memory unbounded. Ch8 Error surfaces & recovery — 10-row table covering ingest schema conflicts, bucket failures, ghost names, dual-agent drift, empty searches, Ollama down, gateway restart, schema fingerprint divergence. Every failure has a named surface and recovery path. Ch9 Per-staffer context — active profile, workspace, client blacklist, audit trail, daily summary. How 20 staffers don't see the same UI. Ch10 Day in the life — 07:00 housekeeping → 07:30 refresh → 08:00 staffer opens → 08:15 drill down → 08:30 Call click → 09:00 second staffer shares memory → 12:30 surge → 14:00 no-show → 15:00 new embeddings live → 17:00 retrospective → 22:00 overnight trials. Ch11 Known limits & non-goals — deferred (rate/margin, push, confidence calibration, neural re-ranker, pm compaction, call_log cross-ref) and explicitly out-of-scope (cloud, ACID, streaming, CRM replace, proprietary formats, hard multi-tenant). Also: nav updated on /dashboard, /console, /proof to link /spec. Every architectural claim in the spec cites either a code path, an ADR number, or a phase reference so someone skeptical can target the specific artifact.	2026-04-20 17:56:18 -05:00
root	76bfa2c8d7	/proof: explain the dual-agent recursive architecture with citations Previous page was numeric claims without explanations — 'sub-100ms SQL', '500K vectors in 341ms' etc. Accurate but undefendable without math, code paths, and ADR references. Expanded to 8 chapters: Ch1 — Live receipts (unchanged: real gateway tests, pass/fail, timing) Ch2 — Architecture. 13-crate diagram with per-crate responsibility table and file paths. gateway → catalogd/queryd/vectord/ingestd + aibridge → object_store. References ADRs 1-20. Ch3 — Dual-agent recursive consensus loop (NEW) - Role specialization (executor=optimist, reviewer=pessimist) - Parallel orchestration via Promise.all - Recursive: sealed playbooks feed playbook_memory → next query - Termination math: sealed \| tool-error abort \| drift abort \| turn-cap abort — every path dumps forensic log - File refs: tests/multi-agent/agent.ts, orchestrator.ts, scenario.ts, run_e2e_rated.ts Ch4 — Playbook memory feedback loop (NEW) - PlaybookEntry shape with embedding - Full boost math: similarity * base_weight * decay * penalty / n_workers, capped at MAX_BOOST_PER_WORKER - Temporal decay (e^-age/30, 30d half-life) - Negative signal (0.5^failures) - Why k=200: narrow cosine discrimination in nomic-embed-text - Evidence: compounding test 0 → 0.250 cap in 3 seeds - persist_sql write-through - Pattern discovery (Path 2 meta-index) - File: crates/vectord/src/playbook_memory.rs Ch5 — ADR citations for each key choice ADR-001, 008, 012, 015, 019, 020 + Phase 19 design note Ch6 — Live scale data (unchanged: pulled from /proof.json) Ch7 — Reproduction recipes: curl for health, sql, hybrid with boost, patterns, pm stats, and the full dual-agent scenario run Ch8 — Honest limits (unchanged: synthetic workers_500k, 1K candidates misaligned to call_log, 7B model imperfection, no rate/margin) Every architectural claim now cites either the code path (crates/.../src/file.rs::fn_name) or the ADR (docs/DECISIONS.md). Someone disputing the system has specific targets to attack. Mechanism unchanged: /proof serves mcp-server/proof.html via Bun.file. /proof.json still returns the live test data the page consumes client-side.	2026-04-20 17:49:08 -05:00

Author

SHA1

Message

Date

root

468798c9ac

/spec: technical specification — 11-chapter README-equivalent

J's ask: explain the full architecture so someone reading a README
can dispute it or recreate it. The repo isn't public yet; this page
IS the spec until it is.

Ch1 Repository layout — 13 crates + tests/multi-agent + docs + data,
    with owned responsibility and file path per crate.

Ch2 Data ingest pipeline (8 steps) — sources (file/inbox/DB/cron),
    parse+normalize with ADR-010 conservative typing, PII auto-tag,
    dedup, Parquet write, catalog register with fingerprint gate,
    mark embeddings stale, queryable immediately.

Ch3 Measurement & indexing — row count / fingerprint / owner /
    sensitivity / freshness / lineage per dataset. HNSW vs Lance
    tradeoff table with measured numbers (ADR-019). Autotune loop.
    Per-profile scoping (Phase 17).

Ch4 Contract inference from external signal — Chicago permit feed
    → role mapping → worker count heuristic → timeline → hybrid
    search with boost → pattern discovery → rendered card. All
    pre-computed before staffer opens UI.

Ch5 What a CRM can't do — 11-row comparison table of capabilities.

Ch6 How it gets better over time — three paths:
    - Phase 19 playbook boost (full math)
    - Pattern discovery meta-index
    - Autotune agent

Ch7 Scale story: 20 staffers, 300 contracts, midday +20/+1M surge
    - Async gateway + per-staffer profile isolation + client blacklists
    - 7-step surge handling flow (ingest, stale-mark, incremental refresh,
      degradation, hot-swap, autotune re-enter)
    - Known pain points: Ollama inference serial, RAM ceiling ~5M on
      HNSW (mitigated by Lance), VRAM 1-2 models sequential,
      playbook_memory unbounded.

Ch8 Error surfaces & recovery — 10-row table covering ingest schema
    conflicts, bucket failures, ghost names, dual-agent drift,
    empty searches, Ollama down, gateway restart, schema fingerprint
    divergence. Every failure has a named surface and recovery path.

Ch9 Per-staffer context — active profile, workspace, client blacklist,
    audit trail, daily summary. How 20 staffers don't see the same UI.

Ch10 Day in the life — 07:00 housekeeping → 07:30 refresh → 08:00
     staffer opens → 08:15 drill down → 08:30 Call click → 09:00
     second staffer shares memory → 12:30 surge → 14:00 no-show →
     15:00 new embeddings live → 17:00 retrospective → 22:00
     overnight trials.

Ch11 Known limits & non-goals — deferred (rate/margin, push, confidence
     calibration, neural re-ranker, pm compaction, call_log cross-ref)
     and explicitly out-of-scope (cloud, ACID, streaming, CRM replace,
     proprietary formats, hard multi-tenant).

Also: nav updated on /dashboard, /console, /proof to link /spec.
Every architectural claim in the spec cites either a code path, an
ADR number, or a phase reference so someone skeptical can target
the specific artifact.

2026-04-20 17:56:18 -05:00

root

76bfa2c8d7

/proof: explain the dual-agent recursive architecture with citations

Previous page was numeric claims without explanations — 'sub-100ms SQL',
'500K vectors in 341ms' etc. Accurate but undefendable without math,
code paths, and ADR references. Expanded to 8 chapters:

Ch1 — Live receipts (unchanged: real gateway tests, pass/fail, timing)

Ch2 — Architecture. 13-crate diagram with per-crate responsibility
      table and file paths. gateway → catalogd/queryd/vectord/ingestd
      + aibridge → object_store. References ADRs 1-20.

Ch3 — Dual-agent recursive consensus loop (NEW)
      - Role specialization (executor=optimist, reviewer=pessimist)
      - Parallel orchestration via Promise.all
      - Recursive: sealed playbooks feed playbook_memory → next query
      - Termination math: sealed | tool-error abort | drift abort |
        turn-cap abort — every path dumps forensic log
      - File refs: tests/multi-agent/agent.ts, orchestrator.ts,
        scenario.ts, run_e2e_rated.ts

Ch4 — Playbook memory feedback loop (NEW)
      - PlaybookEntry shape with embedding
      - Full boost math: similarity * base_weight * decay * penalty
        / n_workers, capped at MAX_BOOST_PER_WORKER
      - Temporal decay (e^-age/30, 30d half-life)
      - Negative signal (0.5^failures)
      - Why k=200: narrow cosine discrimination in nomic-embed-text
      - Evidence: compounding test 0 → 0.250 cap in 3 seeds
      - persist_sql write-through
      - Pattern discovery (Path 2 meta-index)
      - File: crates/vectord/src/playbook_memory.rs

Ch5 — ADR citations for each key choice
      ADR-001, 008, 012, 015, 019, 020 + Phase 19 design note

Ch6 — Live scale data (unchanged: pulled from /proof.json)

Ch7 — Reproduction recipes: curl for health, sql, hybrid with boost,
      patterns, pm stats, and the full dual-agent scenario run

Ch8 — Honest limits (unchanged: synthetic workers_500k, 1K candidates
      misaligned to call_log, 7B model imperfection, no rate/margin)

Every architectural claim now cites either the code path
(crates/.../src/file.rs::fn_name) or the ADR (docs/DECISIONS.md).
Someone disputing the system has specific targets to attack.

Mechanism unchanged: /proof serves mcp-server/proof.html via
Bun.file. /proof.json still returns the live test data the page
consumes client-side.

2026-04-20 17:49:08 -05:00

2 Commits