profit ac01fffd9a checkpoint: matrix-agent-validated (2026-04-25)

Architectural snapshot of the lakehouse codebase at the point where the
full matrix-driven agent loop with Mem0 versioning + deletion was
validated end-to-end.

WHAT THIS REPO IS
A clean single-commit snapshot of the lakehouse code. Heavy test data
(.parquet datasets, vector indexes) excluded — see REPLICATION.md for
regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse.

WHAT WAS PROVEN
- Vector retrieval across multi-corpora matrix (chicago_permits + entity
  briefs + sec_tickers + distilled procedural + llm_team runs)
- Observer hand-review (cloud + heuristic fallback) gating each candidate
- Local-model agent loop (qwen3.5:latest) with tool use + scratchpad
- Playbook seal on success → next-iter retrieval surfaces it as preamble
- Mem0 versioning + deletion in pathway_memory:
    * UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical
    * REVISE: chains versions, parent.superseded_at + superseded_by stamped
    * RETIRE: marks specific trace retired with reason, excluded from retrieval
    * HISTORY: walks chain root→tip, cycle-safe

KEY DIRECTORIES
- crates/vectord/src/pathway_memory.rs — Mem0 ops live here
- crates/vectord/src/playbook_memory.rs — original Mem0 reference
- tests/agent_test/ — local-model agent harness + PRD + session archives
- scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus)
- scripts/vectorize_raw_corpus.ts — corpus → vector indexes
- scripts/analyze_chicago_contracts.ts — real inference pipeline
- scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces

Replication: see REPLICATION.md for Debian 13 clean install + cloud-only
adaptation (no local Ollama).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-25 19:43:27 -05:00

3.5 KiB

Raw Permalink Blame History

ADR-021: Sparse Data Trust Path — start with nothing, earn everything

Status: Proposed — 2026-04-17 Triggered by: Legacy staffing company pushback: "we don't have that data" Owner: J

The Problem

We demonstrated a system with rich worker profiles (18 fields, behavioral scores, certifications, communication history). The staffing company said: "We don't have any of that. We have a name, a phone number, and a contract."

They're right. Our demo assumed data that doesn't exist in their world. Showing AI that only works with perfect data is worse than useless — it builds distrust.

Their Reality

Day 1 data for a typical worker:

Name
Phone number
Maybe: city, role, one or two skills mentioned on a call

Day 1 data for a contract:

Client name
Role needed
Headcount
Start date
Maybe: required certs, location

That's it. No reliability scores. No availability metrics. No communication history. No certifications in a database. The staffing coordinator carries that knowledge in their HEAD.

The Trust Path

Phase 1: Work with what they have (Day 1)

Accept sparse profiles: name + phone + role. That's enough.
Match contracts to workers by role + location only. No scores.
The system is useful immediately: "here are the 12 people you have tagged as Forklift Operators in IL." That's already faster than scrolling a spreadsheet.
Don't show empty fields. Don't show 0% bars. Don't show what's missing — show what's THERE.

Phase 2: Earn data through use (Week 1-4)

Every placement generates a timesheet → reliability starts building
Every call logged → communication history accumulates
Every check-in → availability becomes real
Every cert verified → certification database grows
The staffer doesn't "enter data" — they just do their job, and the system learns from it.

Phase 3: AI starts helping (Month 2+)

Enough data to actually score workers → show reliability
Enough history to predict availability → surface it
Enough placements to know client preferences → suggest matches
The enrichment happened BECAUSE they used the system, not as a prerequisite TO use it.

What This Means for the UI

Worker cards must gracefully degrade. Name only? Show name only. Name + role? Show name + role. Full profile? Show everything.
Never show empty metrics. No "Reliability: 0%" — that looks like the worker is unreliable. Just don't show it until there's data.
Lead with what the staffer KNOWS: "you placed this worker at Company X last month" — that's information they trust because they lived it.

What This Means for the Architecture

The vector index works on whatever text exists. A name + role is 200 characters. That's enough for an embedding. As more data arrives, the embeddings get richer and search gets better.
The hybrid search works with sparse SQL filters too. "role = 'Forklift Operator'" is a valid filter even without reliability.
The playbook system captures the staffer's decisions: "you picked this worker for this contract" → that IS the training data for future AI recommendations.

Implementation

Sparse profile ingest: accept CSV with as few as 2 columns (name, phone). Everything else optional.
Graceful UI degradation: worker cards only show fields that exist
Progressive enrichment: timesheet ingest → auto-calculate reliability; check-in ingest → auto-calculate availability
Trust indicators: "based on 3 placements" not "Reliability: 87%" — show WHERE the number comes from

3.5 KiB Raw Permalink Blame History