profit ac01fffd9a checkpoint: matrix-agent-validated (2026-04-25)

Architectural snapshot of the lakehouse codebase at the point where the
full matrix-driven agent loop with Mem0 versioning + deletion was
validated end-to-end.

WHAT THIS REPO IS
A clean single-commit snapshot of the lakehouse code. Heavy test data
(.parquet datasets, vector indexes) excluded — see REPLICATION.md for
regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse.

WHAT WAS PROVEN
- Vector retrieval across multi-corpora matrix (chicago_permits + entity
  briefs + sec_tickers + distilled procedural + llm_team runs)
- Observer hand-review (cloud + heuristic fallback) gating each candidate
- Local-model agent loop (qwen3.5:latest) with tool use + scratchpad
- Playbook seal on success → next-iter retrieval surfaces it as preamble
- Mem0 versioning + deletion in pathway_memory:
    * UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical
    * REVISE: chains versions, parent.superseded_at + superseded_by stamped
    * RETIRE: marks specific trace retired with reason, excluded from retrieval
    * HISTORY: walks chain root→tip, cycle-safe

KEY DIRECTORIES
- crates/vectord/src/pathway_memory.rs — Mem0 ops live here
- crates/vectord/src/playbook_memory.rs — original Mem0 reference
- tests/agent_test/ — local-model agent harness + PRD + session archives
- scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus)
- scripts/vectorize_raw_corpus.ts — corpus → vector indexes
- scripts/analyze_chicago_contracts.ts — real inference pipeline
- scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces

Replication: see REPLICATION.md for Debian 13 clean install + cloud-only
adaptation (no local Ollama).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-25 19:43:27 -05:00

3.3 KiB

Raw Blame History

Auditor Context — project preamble for fact extraction

This file is read by auditor/fact_extractor.ts and prepended to the extract-facts prompt sent to llm_team. The goal: give the extractor + verifier enough grounding to ground domain-specific facts instead of marking them UNVERIFIABLE by default.

Keep this short (< 400 words). Verifier only reads the first ~4KB of the prompt alongside the facts. Longer = noise, not signal.

Update when: a new Phase lands, a crate is added/removed, the project's primary domain shifts (e.g. staffing → DevOps).

What Lakehouse is

Lakehouse is a Rust-first data platform over S3-compatible object storage. Primary use: a staffing company ingesting legacy CRM data for AI-powered worker matching, contract fulfillment, and playbook-driven coordination.

Architecture: 13 Rust crates + a Python sidecar (Ollama) + TypeScript sub-agents (auditor, scrum_master, bot). Runs on a single server (Nvidia A4000, 128GB RAM). All services on localhost: gateway :3100, sidecar :3200, UI :3300, MCP :3700, observer :3800, MinIO :9000.

Key crates (each maps to a responsibility)

shared — types, Arrow helpers, PII utilities, SecretsProvider
proto — gRPC definitions
storaged — S3/MinIO I/O, AppendLog, ErrorJournal
catalogd — metadata authority (manifests, views, tombstones)
queryd — DataFusion SQL, MemTable cache, compaction
ingestd — CSV/JSON/PDF/Postgres/MySQL ingest
vectord — embeddings, HNSW index, playbook_memory meta-index (Phase 19+)
vectord-lance — Lance 4.0 firewall crate (separate Arrow version)
journald — append-only mutation event log
aibridge — Rust↔Python sidecar bridge, context budget + continuation
gateway — Axum HTTP :3100 + gRPC :3101 (Phase 38+ adds /v1/chat)
ui — Dioxus WASM (stale, pre-Phase-9)
lance-bench — standalone benchmark

Current architectural direction (Phase 38-44)

Universal AI Control Plane: a /v1/chat OpenAI-compatible API that routes all LLM traffic through one layer for token accounting + provider fallback. Truth Layer + Validation Pipeline enforce staffing-domain invariants (worker eligibility, PII, contract rules). The Auditor (Phase A of cohesion plan) hard-blocks PR merges on placeholder code.

Auditor sub-agent role

auditor/ (TypeScript, Bun runtime) polls Gitea every 90s for open PRs. For each fresh head SHA it runs 4 checks in parallel: static (grep-style placeholder detection), dynamic (runs the hybrid fixture), inference (gpt-oss:120b cloud review with N=3 consensus + qwen3-coder:480b tie-breaker), and kb_query (reads data/_kb/*.jsonl for prior evidence). Verdicts post to Gitea as commit status + review comment. Findings append to data/_kb/audit_lessons.jsonl (path-agnostic signatures for dedup). Curated scratchpads from tree-split get routed through this extract-facts pipeline to populate audit_facts.jsonl — which is what you (the extractor) are currently producing.

Things that are NOT the auditor

The LLM Team UI at /root/llm_team_ui.py (devop.live:5000) — a separate product for human-facing multi-model experimentation
The scrum_master pipeline at tests/real-world/scrum_master_pipeline.ts — reviews files, not claims
The bot at bot/ — will apply fixes, doesn't audit

3.3 KiB Raw Blame History