Lakehouse — Technical Specification

Repository layout

Data ingest pipeline

Measurement & indexing

What gets measured per dataset

• Row count (from parquet footer, not a SELECT COUNT). O(1).
• Schema fingerprint — SHA-256 over (column_name, type, nullability, sort) tuples. Drives ADR-020 idempotent register.
• Owner / sensitivity / freshness SLA — catalog v2 metadata. PII auto-detected; owner assigned on ingest.
• Lineage — source_system → ingest_job → dataset. Who put this here, when, from what.
• Last embedded at — when the vector index covering this dataset was last refreshed. Drives stale-detection.

How vector indexes are built

Two backends, chosen per profile (ADR-019):

Autotune

The vectord::agent background task runs continuously. Per index, it proposes HNSW configurations (ef_construction × ef_search), executes a trial against a stored eval set, journals the result as JSONL, and — if recall beats the min_recall gate (0.9) and latency wins the Pareto test — promotes the new config atomically via promotion_registry. No downtime. Rollback in milliseconds.

Per-profile / per-staffer indexing

Model profiles (Phase 17) are not routing strings — they are named scopes. Each profile has bound_datasets[], hnsw_config, vector_backend, and bucket. When a staffer activates a profile:

• EmbeddingCache warms for bound indexes only
• HNSW is rebuilt with the profile's config (if different from current)
• Search via POST /vectors/profile/<id>/search rejects out-of-scope queries with 403 + list of allowed bindings
• Ollama swaps to the profile's model via keep_alive=0; only one model in VRAM at a time

Provider fleet — 5 active, 40+ frontier models reachable

Declared in config/providers.toml + config/modes.toml. Gateway is an OpenAI-compatible drop-in middleware: any consumer that speaks POST /v1/chat/completions gets routing, audit, cost telemetry, and the full memory substrate behind every call.

The 9-rung cloud-first ladder

Defined in tests/real-world/scrum_master_pipeline.ts as const LADDER. Each attempt is evaluated by isAcceptable() = chars ≥ 3800 ∧ not malformed JSON-only. On reject, the next rung sees a learning preamble carrying the prior rejection reason.

1  ollama_cloud / kimi-k2:1t            1T params · flagship
2  ollama_cloud / qwen3-coder:480b      coding specialist
3  ollama_cloud / deepseek-v3.1:671b    reasoning
4  ollama_cloud / mistral-large-3:675b  deep analysis
5  ollama_cloud / gpt-oss:120b          reliable workhorse
6  ollama_cloud / qwen3.5:397b          dense final thinker
7  openrouter   / openai/gpt-oss-120b:free  rescue tier
8  openrouter   / google/gemma-3-27b-it:free fastest rescue
9  ollama       / qwen3.5:latest        last-resort local

N=3 consensus + cross-architecture tie-breaker

Every audit and every consensus-required call fires the primary reviewer N=3 times in parallel (Promise.all — wall-clock = single call). Aggregate votes per claim_idx, majority wins. On a 1-1-1 split, a tie-breaker model with different architecture (qwen3-coder:480b vs primary gpt-oss/kimi) is invoked. Every disagreement, even when majority resolves, writes to data/_kb/audit_discrepancies.jsonl. Closes the cloud-non-determinism gap: temp=0 isn't actually deterministic in practice across hours; consensus + cross-architecture tie-break stabilizes verdicts.

Auditor cross-lineage (Kimi ↔ Haiku ↔ Opus)

Every push to PR #11 triggers auditor/audit.ts within ~90s. To prevent a single model lineage's blind spots from becoming the system's blind spots, audits alternate between Kimi K2.6 (Moonshot lineage) and Haiku 4.5 (Anthropic lineage) by head SHA. Diffs over 100k chars auto-promote to Claude Opus 4.7 (Anthropic frontier). Per-PR cap of 3 audits with auto-reset on each new head SHA prevents infinite-loop spend. Latest verdict on c3c9c21: Haiku 4.5, 24.6s, 100% grounding-verified across 10 findings.

Distillation v1.0.0 — the frozen substrate

The substrate the auditor and mode runner sit on is tagged at distillation-v1.0.0 / commit e7636f2. 145 unit tests pass · 22/22 acceptance invariants · 16/16 audit-full checks · bit-identical reproducibility verified. The distillation phase exports clean SFT / RAG / preference samples with a multi-layer contamination firewall (SFT_NEVER constant + scorer category mapping + acceptance fixtures); the auditor consumes the substrate. The frozen tag means: any future "the system regressed" question has a baseline to bisect against, byte-for-byte.

Continuation primitive (Phase 21)

generateContinuable() handles output-overflow without max_tokens tourniquets — empty response → geometric backoff retry; truncated-JSON → continue with partial as scratchpad. generateTreeSplit() handles input-overflow via map-reduce with running scratchpad. Both respect assertContextBudget() so silent truncation can't happen. Now Rust-native in crates/aibridge/src/continuation.rs (Phase 44).

Per-staffer tool_level (Phase 23)

Scenarios can be scoped to a specific coordinator (staffer: {id, name, tenure_months, role, tool_level}). tool_level controls which tiers are available:

• full — T1/T2 local, T3 cloud, cloud rescue on failure
• local — T1/T2/T3 all local (gpt-oss:20b as overseer)
• basic — kimi-k2.5 cloud executor + local reviewer + local T3, no rescue
• minimal — kimi-k2.5 cloud executor, no T3, no rescue. Playbook inheritance is the only signal.

Measured 2026-04-21 on a 36-run demo (4 staffers × 3 contracts × 3 rounds): James Park (mid, local tools) ranked first at 92.9% fill and 36.8 cites/run, Maria Chen (senior, full tools) second at 81.0%. Cloud T3 adds latency without measurable benefit on this workload. Alex Rivera (trainee, minimal) still hit 59.5% fill purely from playbook inheritance — proof the memory carries knowledge when the model is capable.

Contract inference from external signal

The concrete example running on devop.live/lakehouse is Chicago Department of Buildings permit data (public Socrata API). Every permit is a signal that construction — and therefore staffing — is coming.

Flow

Coverage forecast

/intelligence/staffing_forecast aggregates the last 30 days of permits into predicted role-level demand, joins against the IL bench supply, computes coverage %, and classifies each role as critical / tight / watch / ok. The dashboard's top panel renders this — staffers see supply gaps before they query.

What a CRM can't do (and why)

How it gets better over time

Path 1 — Playbook boost with geo + role prefilter (Phase 19 + refinement)

Every sealed fill is seeded to playbook_memory. The boost fires inside /vectors/hybrid when use_playbook_memory: true. Math, tightened 2026-04-21 after a diagnostic pass found globally-ranked playbooks were missing the SQL-filtered candidate pool entirely:

per_worker = cosine(query_emb, playbook_emb) × 0.5 × e^(-age/30) × 0.5^failures / n_workers
boost[(city, state, name)] = min(Σ per_worker, 0.25)

Multi-strategy retrieval (new): before cosine, compute_boost_for_filtered_with_role(target_geo, target_role) prefilters to same-city playbooks, then gives exact (role, city, state) matches similarity=1.0 and fills up to half the top-k. Cosine fills the rest. Mirrors 2026 Mem0/Zep guidance on parallel-strategy rerank.

Measured lift: before geo-filter, Nashville Welder query returned boosts=170 matched=0 (zero intersection with candidate pool). After: boosts=36 matched=11. On the Riverfront Steel scenario, total playbook citations went from 2 → 28 per run — a 14× delta on identical inputs. The diagnostic log playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=? runs on every hybrid call so the class of silent-miss bug stays visible.

Path 2 — Pattern discovery (meta-index)

/vectors/playbook_memory/patterns goes beyond "who was endorsed" to answer "what did past similar fills have in common?" Aggregates recurring certifications, skills, archetype, reliability distribution across the top-K semantically similar playbooks. Surfaces signal the operator didn't explicitly query for.

Path 3 — Autotune agent

The vectord::agent background task runs continuously. Watches the HNSW trial journal, proposes configs, executes trials, promotes Pareto winners — without human intervention. Operator sees "the index got faster overnight" and doesn't know why. The journal knows why.

Path 4 — Knowledge Base + pathway recommender (Phase 22)

Meta-layer over playbook_memory. Files under data/_kb/:

• signatures.jsonl — sig_hash + embedding of every run's event shape
• outcomes.jsonl — per-run summary (models, fill rate, turns, citations, rescue stats, staffer, elapsed)
• pathway_recommendations.jsonl — AI-synthesized advice for the next run of a similar sig
• error_corrections.jsonl — fail→succeed deltas on the same sig
• config_snapshots.jsonl — hash of active models + tool_level at each run

Cycle: scenario ends → kb.indexRun() appends outcome → kb.recommendFor(nextSpec) finds k-NN signatures, feeds outcome history to an overview model, writes structured JSON advice → next scenario reads it via kb.loadRecommendation(spec) and injects pathway_notes into the executor's context alongside prior T3 lessons.

Path 5 — Cloud rescue on failure (Phase 22 item B)

When an event fails (drift abort, JSON parse, pool exhaustion) and cloud T3 is enabled, requestCloudRemediation() feeds the full failure trace — SQL filters attempted, row counts, reviewer drift notes, gap signals, contract terms — to gpt-oss:120b on Ollama Cloud. Cloud returns structured {retry, new_city, new_state, new_role, new_count, rationale}. Event retries once with the pivot. Verified on stress_01: Gary IN (zero workers indexed) misplacement → cloud proposed South Bend IN → retry filled 1/1.

Path 6 — Staffer competence-weighted retrieval (Phase 23)

Answers "who handled this" as a first-class matrix-index dimension. Each scenario carries staffer: {id, name, tenure_months, role, tool_level}. After every run, recomputeStafferStats(staffer_id) aggregates their fill_rate, turn efficiency, citation density, rescue rate into a single competence_score (0.45·fill + 0.20·turn_eff + 0.20·cites + 0.15·rescue).

findNeighbors returns weighted_score = cosine × max_staffer_competence — top-performer playbooks rank above juniors' on similar scenarios. Auto-discovery emerges: running 4 staffers × 3 contracts × 3 rounds surfaced Rachel D. Lewis (Welder Nashville) with 18 endorsements across all 4 staffers, Angela U. Ward (Machine Op Indianapolis) with 19 — reliable-performer labels the system built without human tagging.

Path 7 — Pathway memory (ADR-021 — semantic-correctness matrix layer)

Memory at the system layer, not the worker layer. Every accepted scrum review writes a PathwayTrace with the full backtrack: file fingerprint, model used, signal class, KB chunks consulted, observer events, semantic flags (UnitMismatch, TypeConfusion, OffByOne, StaleReference, DeadCode, BoundaryViolation, …), bug fingerprints. A new query that fingerprints to the same trace can hot-swap to the prior result without re-running the 9-rung escalation. Five-factor hot-swap gate: narrow fingerprint match AND audit consensus pass AND replay_count ≥ 3 (probation) AND success_rate ≥ 0.80 AND NOT retired AND vector cosine ≥ 0.90.

Live state (verified on this load): 88 traces · 11 / 11 successful replays · 100% reuse rate · probation gate crossed. Endpoints: /vectors/pathway/insert · /query · /record_replay · /stats · /bug_fingerprints. Spec: docs/DECISIONS.md ADR-021.

Path 8 — Per-staffer hot-swap index

Memory scoped to whoever's acting. /intelligence/chat accepts staffer_id; on match, defaults state filter to staffer territory, scopes playbook-pattern geo to staffer's primary city/state, and surfaces response.staffer.name so the UI relabels MEMORY → MARIA'S MEMORY. Same query "forklift operators" returns 167 IL workers as Maria, 89 IN as Devon, 16 WI as Aisha. The corpus stays intact; the relevance gradient is per coordinator; each accumulates fills independently.

Roster: /staffers endpoint reads from STAFFERS in mcp-server/index.ts. Three personas today (Maria/Devon/Aisha); architecture generalizes — every new metro adds territories, not code paths.

Path 9 — Construction Activity Signal Engine

Memory at the network layer. Every contractor in the corpus is also a forward indicator on the public equities they touch via three attribution flavors: direct (contractor IS the public issuer — SEC tickers index match), parent (subsidiary of a public parent — curated KNOWN_PARENT_MAP, e.g. Turner → HOC.DE via Hochtief AG), associated (co-permit network — Bob's Electric appears with TARGET CORPORATION 3+ times → inherits TGT). The associated path is the moat: a staffing-permit dataset that maps contractor-to-public-issuer is not commercially available; we synthesize it from the Socrata co-occurrence graph.

BAI (Building Activity Index) = attribution-weighted average day-change across surfaced issuers. Indexed build value = total $ of permits attributable to ANY public issuer in scope. Network depth = issuers / total attribution edges. Cross-metro replication explicit in the architecture — Chicago is Phase 1; NYC DOB / LA County / Houston BCD / Boston ISD / DC DCRA are all Socrata-shaped, ship as config-only adapters.

Path 10 — Observer outcome ingest (Phase 24)

Observer runs as lakehouse-observer.service, now with an HTTP listener on :3800. Scenarios POST per-event outcomes to /event with full provenance (staffer_id, sig_hash, event_kind, role, city, state, rescue flags). Observer's ERROR_ANALYZER and PLAYBOOK_BUILDER loops consume them alongside MCP-wrapped ops. Persistence switched from the old /ingest/file REPLACE path to an append-only data/_observer/ops.jsonl journal so the trace survives across restarts.

Input normalizer + unified memory query

Two surfaces added 2026-04-21 to make the memory stack respond coherently to any input shape:

• normalizeInput(raw) — accepts structured JSON, natural language, or mixed. Three-tier: structured fast-path → regex (handles "need 3 welders in Nashville, TN" in 0ms without an LLM call) → qwen3 LLM fallback for low-signal inputs.
• POST /memory/query — one endpoint, returns every memory surface in a single bundle: playbook_workers (with boost + citations), pathway_recommendation (KB), neighbor_signatures (competence-weighted), prior_lessons (T3 overseer history), top_staffers (leaderboard), discovered_patterns (auto-surfaced reliable workers for this role+city), latency_ms (per-source). Natural-language query end-to-end: 319ms.

Honest gaps — what we can still implement

Three of the five 2026-era memory findings remain unwired. Flagged for near-term implementation, not hidden:

• Zep-style validity windows — playbook entries have timestamps but no valid_until or schema_fingerprint. Load-bearing: when a schema migration changes a column, stale playbooks silently keep boosting. Biggest-value remaining fix.
• Mem0-style UPDATE / DELETE / NOOP ops — /seed only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Playbook file grows faster than necessary.
• Letta working-memory hot cache — every query scans all 1560 playbook entries from disk. Cheap today at 1.5K, not at 100K. Solution: LRU in-process cache for the last N playbooks or the current sig's neighborhood.

Validity windows is next — preserves the trust signal (boost only fires on playbooks that are still true given the current schema) rather than the latency signal (which the current scale doesn't need yet).

Scale story — 20 staffers, 300 contracts, a surge

20 concurrent staffers

Axum is async. The gateway handles concurrent requests on Tokio with work-stealing. No per-request thread. Tested at 10 parallel queries in 82ms total on this hardware.

Per-staffer profile isolation. Each staffer activates their own profile (Phase 17) or workspace (Phase 8.5). Profile scopes their search to bound datasets. Workspace carries their in-progress contracts across sessions.

Per-client blacklists. Auto-applied when the caller passes client: "X" on /search. Staffer A filling for Acme never sees Acme's flagged workers. Staffer B filling for MidState sees them normally.

300 active contracts

SQL on job_orders is cheap. 300 rows is nothing — a scan is microseconds.

Workspace per contract. Each contract gets its own workspace with saved searches, shortlists, activity log. Zero-copy handoff between staffers (pointer swap, not data copy).

Forecast remains coherent. /intelligence/staffing_forecast aggregates 30-day permit data regardless of contract count. The bench supply query (GROUP BY role over workers_500k) is a single sub-second SQL.

Midday surge: +20 contracts, +1M profiles

The delta arrives at 12:30. Here's what happens in the following minutes:

Known pain points at this scale

• Ollama inference is serial. Embedding 1M rows at ~50 chunks/sec through nomic-embed-text = ~6 hours. Acceptable for overnight refresh, not for "immediate." Mitigated by incremental refresh (only deltas).
• RAM ceiling on HNSW. Around 5M vectors × 768d, HNSW stops fitting in 128GB comfortably. Mitigation: per-profile vector_backend: lance flip — disk-resident IVF_PQ scales past the RAM line (ADR-019).
• VRAM ceiling for model variety. A4000 16GB holds 1-2 loaded models. Multi-model recruiter surfaces are a sequential swap, not parallel (Ollama keep_alive=0). Phase 17 profile activation unloads the prior model on swap.
• playbook_memory growth. 1936 entries today. Phase 25 (2026-04-21) added retirement via valid_until + schema_fingerprint fields + POST /vectors/playbook_memory/retire endpoint (manual or schema-drift triggered). Active vs retired split surfaced on GET /vectors/playbook_memory/status. Brute-force cosine still sub-ms at current size; Letta-style working-memory hot cache deferred until entry count crosses ~100K.

Error surfaces & recovery

Per-staffer context

Per-staffer hot-swap index (the recent layer)

Maria runs Chicago. Devon runs Indianapolis. Aisha runs Wisconsin/Michigan. They share one corpus, but search results, recurring-skill patterns, and playbook context all reshape to whoever is acting. /intelligence/chat accepts staffer_id; on match, defaults state filter to the staffer's territory, scopes playbook-pattern geo to their primary city/state, and surfaces response.staffer.name so the UI relabels MEMORY → MARIA'S MEMORY.

Verified end-to-end: same query "forklift operators" returns 167 IL workers as Maria, 89 IN as Devon, 16 WI as Aisha (live numbers; refresh the profiler page to recompute). The corpus stays intact; the relevance gradient is per coordinator. As each accumulates fills, their slice of the playbook compounds independently. Roster: /staffers endpoint, declared in STAFFERS in mcp-server/index.ts. Adding a staffer is one append; the architecture is metro-agnostic by construction.

Active profile (Phase 17)

Scopes every search. A staffing-recruiter profile bound to workers_500k sees only that dataset. A security-analyst profile bound to threat_intel cannot see worker data. GET /vectors/profile/<id>/audit records every tool invocation by model identity.

Workspace (Phase 8.5)

Per-contract state. Each workspace has daily/weekly/monthly tiers, saved searches, shortlists, activity logs. Survives across sessions. Instant zero-copy handoff between staffers — pointer swap, not data copy. Persisted to object storage, rebuilt on startup.

Client blacklist

Per-client worker exclusion. Populated via POST /clients/:client/blacklist. Auto-applied when the caller passes client: "X" on /search. JSON-backed; would move to catalog table under real client load.

Audit trail

Phase 12 tool registry logs every governed-action invocation (who called what, with what args, when, outcome). GET /tools/audit queryable. Phase 13 access control layers on top — role-based field masking, query audit log.

Daily summary per staffer

Workspace activity log + per-staffer filter on the event journal gives "what did Sarah do today" as a direct query. The foundation for shift-handoff reports.

Staffer identity + competence-weighted retrieval (Phase 23)

Each scenario run carries an explicit staffer: {id, name, tenure_months, role, tool_level}. The KB aggregates per-staffer stats (data/_kb/staffers.jsonl) that roll up into a single competence_score:

competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate

When any query runs kb.findNeighbors(spec, k), the ranking isn't just cosine similarity — it's weighted_score = cosine × max_staffer_competence over the best coordinator who ran that signature. Senior staffers' playbooks surface above juniors' on similar scenarios, even when the juniors' scenario was marginally closer in embedding space.

The tool_level knob (full / local / basic / minimal) controls which tiers are available to a given staffer's runs. See Ch 3 for the mapping. Variance is real and measurable: 36-run demo produced a 46pt fill-rate delta between James (local tools, 93%) and Alex (minimal tools, 60%) on identical contracts.

Auto-discovered reliable-performer labels

A second-order output of the competence path: when multiple staffers independently endorse the same worker on similar-signature playbooks, that worker accumulates cross-staffer endorsements. scripts/kb_staffer_report.py surfaces them — after 36 runs, Rachel D. Lewis (Welder Nashville) had 18 endorsements across 4 staffers, Angela U. Ward (Machine Op Indianapolis) 19. These are high-confidence "reliable" labels the system produced without human tagging. The UI could badge these workers on future queries; today they're visible via /memory/query's discovered_patterns bundle.

A day in the life — from morning brief to EOD retrospective

SMS + email drafts in the pipeline

After each sealed fill (via scenario.ts or manual /log flow with downstream hooks), generateArtifacts in the scenario runner produces: (a) one SMS per worker (TO: Name, message under 180 chars), (b) one client confirmation email. Drafts are saved to sms.md and emails.md under the scenario output dir. Ollama drafts them; the staffer reviews and sends. No auto-send; human-in-the-loop.

Known limits & non-goals

Deferred — real architectural work, just not shipped yet

• BAI persistence + backtesting. Building Activity Index is computed live per page load. To validate the thesis (permit activity precedes equity moves) we need the daily series saved over months. Architectural support exists (data/_kb/audit_baselines.jsonl append pattern); just hasn't run long enough.
• NYC DOB adapter. Architecture is metro-agnostic — Chicago is Phase 1. NYC DOB ships next as a config-only Socrata adapter; LA County, Houston BCD, Boston ISD, DC DCRA queue behind it. Each new metro multiplies network edges without multiplying the codebase.
• 12 awaiting public-data sources for contractor profile. DOL Wage & Hour, EPA ECHO, MSHA, BBB, PACER civil suits, UCC liens, D&B credit, State licensure, Surety bonds, DOT/FMCSA, State UI claims, DOL RAPIDS apprenticeships. Listed by name on every contractor profile with a one-line "would show:" sample. Each ships as a Socrata-style adapter; engineering scope is concrete.
• Rate / margin awareness. Worker pay expectations vs contract bill rate not modeled. Requires adding pay_rate to workers, bill_rate to contracts, and a filter + warning path. Partially addressed via ContractTerms.budget_per_hour_max passed to T3/rescue prompts, but the match-time filter isn't wired yet.
• Mem0-style UPDATE / DELETE / NOOP operations on playbooks. Today /seed only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Cheap to add, moderate payoff.
• Letta working-memory hot cache. Every boost query scans all active playbook entries from in-memory state. ~5K today; cheap. Will bite somewhere north of 100K. Deferred until the ceiling approaches.
• Confidence calibration. Top-K is a rank, not a probability. No calibrated "85% likely to accept" score. Requires outcome-labeled training data.
• SEC name-to-ticker fuzzy precision. Current matcher requires ≥2 non-stopword overlap; rare false positives still surface (saw FLG attach to a PNC-adjacent contractor once). Tightenable to require an explicit allow-list for production trading use.
• Tighter integration of pathway memory + scrum loop. ADR-021 substrate is shipped (88 traces, 11/11 replays). The hot-swap gate fires correctly; what's deferred is automatic mode-runner short-circuit when a high-confidence pathway match is available before any cloud call burns.

Non-goals — explicitly out of scope

• Cloud deployment. Local-first by design. Works offline after setup.
• Full ACID transactions. Single-writer model is sufficient; Delta Lake-grade MVCC is deliberately not attempted.
• Real-time streaming / CDC. Batch ingest is the model. Scheduled refresh, not transactional replication.
• Replacing the CRM. This is the analytical + AI layer behind the CRM. Operational CRUD stays with the existing system.
• Custom file formats. Parquet for datasets, sidecar indexes for vectors. No proprietary formats (ADR-008, ADR-018 reaffirm).
• Hard multi-tenant isolation. Profiles and federation provide soft isolation. Adversarial multi-tenant is not a goal — this system assumes a single-trust operator.