ac01fffd9a
Architectural snapshot of the lakehouse codebase at the point where the
full matrix-driven agent loop with Mem0 versioning + deletion was
validated end-to-end.
WHAT THIS REPO IS
A clean single-commit snapshot of the lakehouse code. Heavy test data
(.parquet datasets, vector indexes) excluded — see REPLICATION.md for
regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse.
WHAT WAS PROVEN
- Vector retrieval across multi-corpora matrix (chicago_permits + entity
briefs + sec_tickers + distilled procedural + llm_team runs)
- Observer hand-review (cloud + heuristic fallback) gating each candidate
- Local-model agent loop (qwen3.5:latest) with tool use + scratchpad
- Playbook seal on success → next-iter retrieval surfaces it as preamble
- Mem0 versioning + deletion in pathway_memory:
* UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical
* REVISE: chains versions, parent.superseded_at + superseded_by stamped
* RETIRE: marks specific trace retired with reason, excluded from retrieval
* HISTORY: walks chain root→tip, cycle-safe
KEY DIRECTORIES
- crates/vectord/src/pathway_memory.rs — Mem0 ops live here
- crates/vectord/src/playbook_memory.rs — original Mem0 reference
- tests/agent_test/ — local-model agent harness + PRD + session archives
- scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus)
- scripts/vectorize_raw_corpus.ts — corpus → vector indexes
- scripts/analyze_chicago_contracts.ts — real inference pipeline
- scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces
Replication: see REPLICATION.md for Debian 13 clean install + cloud-only
adaptation (no local Ollama).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
70 lines
3.3 KiB
Markdown
70 lines
3.3 KiB
Markdown
# Auditor Context — project preamble for fact extraction
|
|
|
|
This file is read by `auditor/fact_extractor.ts` and prepended to the
|
|
extract-facts prompt sent to llm_team. The goal: give the extractor +
|
|
verifier enough grounding to ground domain-specific facts instead of
|
|
marking them UNVERIFIABLE by default.
|
|
|
|
Keep this short (< 400 words). Verifier only reads the first ~4KB of
|
|
the prompt alongside the facts. Longer = noise, not signal.
|
|
|
|
Update when: a new Phase lands, a crate is added/removed, the project's
|
|
primary domain shifts (e.g. staffing → DevOps).
|
|
|
|
---
|
|
|
|
## What Lakehouse is
|
|
|
|
Lakehouse is a Rust-first data platform over S3-compatible object
|
|
storage. Primary use: a staffing company ingesting legacy CRM data for
|
|
AI-powered worker matching, contract fulfillment, and playbook-driven
|
|
coordination.
|
|
|
|
Architecture: 13 Rust crates + a Python sidecar (Ollama) + TypeScript
|
|
sub-agents (auditor, scrum_master, bot). Runs on a single server
|
|
(Nvidia A4000, 128GB RAM). All services on localhost: gateway :3100,
|
|
sidecar :3200, UI :3300, MCP :3700, observer :3800, MinIO :9000.
|
|
|
|
## Key crates (each maps to a responsibility)
|
|
|
|
- **shared** — types, Arrow helpers, PII utilities, SecretsProvider
|
|
- **proto** — gRPC definitions
|
|
- **storaged** — S3/MinIO I/O, AppendLog, ErrorJournal
|
|
- **catalogd** — metadata authority (manifests, views, tombstones)
|
|
- **queryd** — DataFusion SQL, MemTable cache, compaction
|
|
- **ingestd** — CSV/JSON/PDF/Postgres/MySQL ingest
|
|
- **vectord** — embeddings, HNSW index, **playbook_memory meta-index** (Phase 19+)
|
|
- **vectord-lance** — Lance 4.0 firewall crate (separate Arrow version)
|
|
- **journald** — append-only mutation event log
|
|
- **aibridge** — Rust↔Python sidecar bridge, context budget + continuation
|
|
- **gateway** — Axum HTTP :3100 + gRPC :3101 (Phase 38+ adds /v1/chat)
|
|
- **ui** — Dioxus WASM (stale, pre-Phase-9)
|
|
- **lance-bench** — standalone benchmark
|
|
|
|
## Current architectural direction (Phase 38-44)
|
|
|
|
Universal AI Control Plane: a `/v1/chat` OpenAI-compatible API that
|
|
routes all LLM traffic through one layer for token accounting + provider
|
|
fallback. Truth Layer + Validation Pipeline enforce staffing-domain
|
|
invariants (worker eligibility, PII, contract rules). The Auditor
|
|
(Phase A of cohesion plan) hard-blocks PR merges on placeholder code.
|
|
|
|
## Auditor sub-agent role
|
|
|
|
`auditor/` (TypeScript, Bun runtime) polls Gitea every 90s for open PRs.
|
|
For each fresh head SHA it runs 4 checks in parallel: static (grep-style
|
|
placeholder detection), dynamic (runs the hybrid fixture), inference
|
|
(gpt-oss:120b cloud review with N=3 consensus + qwen3-coder:480b
|
|
tie-breaker), and kb_query (reads `data/_kb/*.jsonl` for prior evidence).
|
|
Verdicts post to Gitea as commit status + review comment. Findings
|
|
append to `data/_kb/audit_lessons.jsonl` (path-agnostic signatures for
|
|
dedup). Curated scratchpads from tree-split get routed through this
|
|
extract-facts pipeline to populate `audit_facts.jsonl` — which is what
|
|
you (the extractor) are currently producing.
|
|
|
|
## Things that are NOT the auditor
|
|
|
|
- The LLM Team UI at `/root/llm_team_ui.py` (devop.live:5000) — a separate product for human-facing multi-model experimentation
|
|
- The scrum_master pipeline at `tests/real-world/scrum_master_pipeline.ts` — reviews files, not claims
|
|
- The bot at `bot/` — will apply fixes, doesn't audit
|