Lakehouse — Technical Specification

Repository layout

Data ingest pipeline

Measurement & indexing

What gets measured per dataset

• Row count (from parquet footer, not a SELECT COUNT). O(1).
• Schema fingerprint — SHA-256 over (column_name, type, nullability, sort) tuples. Drives ADR-020 idempotent register.
• Owner / sensitivity / freshness SLA — catalog v2 metadata. PII auto-detected; owner assigned on ingest.
• Lineage — source_system → ingest_job → dataset. Who put this here, when, from what.
• Last embedded at — when the vector index covering this dataset was last refreshed. Drives stale-detection.

How vector indexes are built

Two backends, chosen per profile (ADR-019):

Autotune

The vectord::agent background task runs continuously. Per index, it proposes HNSW configurations (ef_construction × ef_search), executes a trial against a stored eval set, journals the result as JSONL, and — if recall beats the min_recall gate (0.9) and latency wins the Pareto test — promotes the new config atomically via promotion_registry. No downtime. Rollback in milliseconds.

Per-profile / per-staffer indexing

Model profiles (Phase 17) are not routing strings — they are named scopes. Each profile has bound_datasets[], hnsw_config, vector_backend, and bucket. When a staffer activates a profile:

• EmbeddingCache warms for bound indexes only
• HNSW is rebuilt with the profile's config (if different from current)
• Search via POST /vectors/profile/<id>/search rejects out-of-scope queries with 403 + list of allowed bindings
• Ollama swaps to the profile's model via keep_alive=0; only one model in VRAM at a time

Model matrix (Phase 20)

Five tiers declared in config/models.json. Each call site picks the tier appropriate to its purpose — hot-path JSON emitters get fast local, overview/strategic/gatekeeper decisions get thinking models on cloud. Every tier carries context_window, context_budget, and overflow_policy.

Key mechanical finding (2026-04-21): qwen3.5 and qwen3 are thinking models — they burn ~650 tokens of hidden reasoning before emitting the visible response. For hot-path JSON emitters this meant 400-token budgets returned empty strings. Fix: think: false plumbed through sidecar's /generate endpoint; hot path disables thinking (structure matters more than reasoning depth), overseer tiers keep it on. Mistral was dropped entirely after a 0/14 fill rate on complex scenarios (decoder-level malformed-JSON bug, not a prompt issue).

Continuation primitive (Phase 21): generateContinuable() handles output-overflow without max_tokens tourniquets — empty response → geometric backoff retry; truncated-JSON → continue with partial as scratchpad. generateTreeSplit() handles input-overflow via map-reduce with running scratchpad. Both respect assertContextBudget() so silent truncation can't happen.

Per-staffer tool_level (Phase 23)

Scenarios can be scoped to a specific coordinator (staffer: {id, name, tenure_months, role, tool_level}). tool_level controls which tiers are available:

• full — T1/T2 local, T3 cloud, cloud rescue on failure
• local — T1/T2/T3 all local (gpt-oss:20b as overseer)
• basic — kimi-k2.5 cloud executor + local reviewer + local T3, no rescue
• minimal — kimi-k2.5 cloud executor, no T3, no rescue. Playbook inheritance is the only signal.

Measured 2026-04-21 on a 36-run demo (4 staffers × 3 contracts × 3 rounds): James Park (mid, local tools) ranked first at 92.9% fill and 36.8 cites/run, Maria Chen (senior, full tools) second at 81.0%. Cloud T3 adds latency without measurable benefit on this workload. Alex Rivera (trainee, minimal) still hit 59.5% fill purely from playbook inheritance — proof the memory carries knowledge when the model is capable.

Contract inference from external signal

The concrete example running on devop.live/lakehouse is Chicago Department of Buildings permit data (public Socrata API). Every permit is a signal that construction — and therefore staffing — is coming.

Flow

Coverage forecast

/intelligence/staffing_forecast aggregates the last 30 days of permits into predicted role-level demand, joins against the IL bench supply, computes coverage %, and classifies each role as critical / tight / watch / ok. The dashboard's top panel renders this — staffers see supply gaps before they query.

What a CRM can't do (and why)

How it gets better over time

Path 1 — Playbook boost with geo + role prefilter (Phase 19 + refinement)

Every sealed fill is seeded to playbook_memory. The boost fires inside /vectors/hybrid when use_playbook_memory: true. Math, tightened 2026-04-21 after a diagnostic pass found globally-ranked playbooks were missing the SQL-filtered candidate pool entirely:

per_worker = cosine(query_emb, playbook_emb) × 0.5 × e^(-age/30) × 0.5^failures / n_workers
boost[(city, state, name)] = min(Σ per_worker, 0.25)

Multi-strategy retrieval (new): before cosine, compute_boost_for_filtered_with_role(target_geo, target_role) prefilters to same-city playbooks, then gives exact (role, city, state) matches similarity=1.0 and fills up to half the top-k. Cosine fills the rest. Mirrors 2026 Mem0/Zep guidance on parallel-strategy rerank.

Measured lift: before geo-filter, Nashville Welder query returned boosts=170 matched=0 (zero intersection with candidate pool). After: boosts=36 matched=11. On the Riverfront Steel scenario, total playbook citations went from 2 → 28 per run — a 14× delta on identical inputs. The diagnostic log playbook_boost: boosts=N sources=N parsed=N matched=N target_geo=? target_role=? runs on every hybrid call so the class of silent-miss bug stays visible.

Path 2 — Pattern discovery (meta-index)

/vectors/playbook_memory/patterns goes beyond "who was endorsed" to answer "what did past similar fills have in common?" Aggregates recurring certifications, skills, archetype, reliability distribution across the top-K semantically similar playbooks. Surfaces signal the operator didn't explicitly query for.

Path 3 — Autotune agent

The vectord::agent background task runs continuously. Watches the HNSW trial journal, proposes configs, executes trials, promotes Pareto winners — without human intervention. Operator sees "the index got faster overnight" and doesn't know why. The journal knows why.

Path 4 — Knowledge Base + pathway recommender (Phase 22)

Meta-layer over playbook_memory. Files under data/_kb/:

• signatures.jsonl — sig_hash + embedding of every run's event shape
• outcomes.jsonl — per-run summary (models, fill rate, turns, citations, rescue stats, staffer, elapsed)
• pathway_recommendations.jsonl — AI-synthesized advice for the next run of a similar sig
• error_corrections.jsonl — fail→succeed deltas on the same sig
• config_snapshots.jsonl — hash of active models + tool_level at each run

Cycle: scenario ends → kb.indexRun() appends outcome → kb.recommendFor(nextSpec) finds k-NN signatures, feeds outcome history to an overview model, writes structured JSON advice → next scenario reads it via kb.loadRecommendation(spec) and injects pathway_notes into the executor's context alongside prior T3 lessons.

Path 5 — Cloud rescue on failure (Phase 22 item B)

When an event fails (drift abort, JSON parse, pool exhaustion) and cloud T3 is enabled, requestCloudRemediation() feeds the full failure trace — SQL filters attempted, row counts, reviewer drift notes, gap signals, contract terms — to gpt-oss:120b on Ollama Cloud. Cloud returns structured {retry, new_city, new_state, new_role, new_count, rationale}. Event retries once with the pivot. Verified on stress_01: Gary IN (zero workers indexed) misplacement → cloud proposed South Bend IN → retry filled 1/1.

Path 6 — Staffer competence-weighted retrieval (Phase 23)

Answers "who handled this" as a first-class matrix-index dimension. Each scenario carries staffer: {id, name, tenure_months, role, tool_level}. After every run, recomputeStafferStats(staffer_id) aggregates their fill_rate, turn efficiency, citation density, rescue rate into a single competence_score (0.45·fill + 0.20·turn_eff + 0.20·cites + 0.15·rescue).

findNeighbors returns weighted_score = cosine × max_staffer_competence — top-performer playbooks rank above juniors' on similar scenarios. Auto-discovery emerges: running 4 staffers × 3 contracts × 3 rounds surfaced Rachel D. Lewis (Welder Nashville) with 18 endorsements across all 4 staffers, Angela U. Ward (Machine Op Indianapolis) with 19 — reliable-performer labels the system built without human tagging.

Path 7 — Observer outcome ingest (Phase 24)

Observer runs as lakehouse-observer.service, now with an HTTP listener on :3800. Scenarios POST per-event outcomes to /event with full provenance (staffer_id, sig_hash, event_kind, role, city, state, rescue flags). Observer's ERROR_ANALYZER and PLAYBOOK_BUILDER loops consume them alongside MCP-wrapped ops. Persistence switched from the old /ingest/file REPLACE path to an append-only data/_observer/ops.jsonl journal so the trace survives across restarts.

Input normalizer + unified memory query

Two surfaces added 2026-04-21 to make the memory stack respond coherently to any input shape:

• normalizeInput(raw) — accepts structured JSON, natural language, or mixed. Three-tier: structured fast-path → regex (handles "need 3 welders in Nashville, TN" in 0ms without an LLM call) → qwen3 LLM fallback for low-signal inputs.
• POST /memory/query — one endpoint, returns every memory surface in a single bundle: playbook_workers (with boost + citations), pathway_recommendation (KB), neighbor_signatures (competence-weighted), prior_lessons (T3 overseer history), top_staffers (leaderboard), discovered_patterns (auto-surfaced reliable workers for this role+city), latency_ms (per-source). Natural-language query end-to-end: 319ms.

Honest gaps — what we can still implement

Three of the five 2026-era memory findings remain unwired. Flagged for near-term implementation, not hidden:

• Zep-style validity windows — playbook entries have timestamps but no valid_until or schema_fingerprint. Load-bearing: when a schema migration changes a column, stale playbooks silently keep boosting. Biggest-value remaining fix.
• Mem0-style UPDATE / DELETE / NOOP ops — /seed only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Playbook file grows faster than necessary.
• Letta working-memory hot cache — every query scans all 1560 playbook entries from disk. Cheap today at 1.5K, not at 100K. Solution: LRU in-process cache for the last N playbooks or the current sig's neighborhood.

Validity windows is next — preserves the trust signal (boost only fires on playbooks that are still true given the current schema) rather than the latency signal (which the current scale doesn't need yet).

Scale story — 20 staffers, 300 contracts, a surge

20 concurrent staffers

Axum is async. The gateway handles concurrent requests on Tokio with work-stealing. No per-request thread. Tested at 10 parallel queries in 82ms total on this hardware.

Per-staffer profile isolation. Each staffer activates their own profile (Phase 17) or workspace (Phase 8.5). Profile scopes their search to bound datasets. Workspace carries their in-progress contracts across sessions.

Per-client blacklists. Auto-applied when the caller passes client: "X" on /search. Staffer A filling for Acme never sees Acme's flagged workers. Staffer B filling for MidState sees them normally.

300 active contracts

SQL on job_orders is cheap. 300 rows is nothing — a scan is microseconds.

Workspace per contract. Each contract gets its own workspace with saved searches, shortlists, activity log. Zero-copy handoff between staffers (pointer swap, not data copy).

Forecast remains coherent. /intelligence/staffing_forecast aggregates 30-day permit data regardless of contract count. The bench supply query (GROUP BY role over workers_500k) is a single sub-second SQL.

Midday surge: +20 contracts, +1M profiles

The delta arrives at 12:30. Here's what happens in the following minutes:

Known pain points at this scale

• Ollama inference is serial. Embedding 1M rows at ~50 chunks/sec through nomic-embed-text = ~6 hours. Acceptable for overnight refresh, not for "immediate." Mitigated by incremental refresh (only deltas).
• RAM ceiling on HNSW. Around 5M vectors × 768d, HNSW stops fitting in 128GB comfortably. Mitigation: per-profile vector_backend: lance flip — disk-resident IVF_PQ scales past the RAM line (ADR-019).
• VRAM ceiling for model variety. A4000 16GB holds 1-2 loaded models. Multi-model recruiter surfaces are a sequential swap, not parallel (Ollama keep_alive=0). Phase 17 profile activation unloads the prior model on swap.
• playbook_memory growth. 1936 entries today. Phase 25 (2026-04-21) added retirement via valid_until + schema_fingerprint fields + POST /vectors/playbook_memory/retire endpoint (manual or schema-drift triggered). Active vs retired split surfaced on GET /vectors/playbook_memory/status. Brute-force cosine still sub-ms at current size; Letta-style working-memory hot cache deferred until entry count crosses ~100K.

Error surfaces & recovery

Per-staffer context

Active profile (Phase 17)

Scopes every search. A staffing-recruiter profile bound to workers_500k sees only that dataset. A security-analyst profile bound to threat_intel cannot see worker data. GET /vectors/profile/<id>/audit records every tool invocation by model identity.

Workspace (Phase 8.5)

Per-contract state. Each workspace has daily/weekly/monthly tiers, saved searches, shortlists, activity logs. Survives across sessions. Instant zero-copy handoff between staffers — pointer swap, not data copy. Persisted to object storage, rebuilt on startup.

Client blacklist

Per-client worker exclusion. Populated via POST /clients/:client/blacklist. Auto-applied when the caller passes client: "X" on /search. JSON-backed; would move to catalog table under real client load.

Audit trail

Phase 12 tool registry logs every governed-action invocation (who called what, with what args, when, outcome). GET /tools/audit queryable. Phase 13 access control layers on top — role-based field masking, query audit log.

Daily summary per staffer

Workspace activity log + per-staffer filter on the event journal gives "what did Sarah do today" as a direct query. The foundation for shift-handoff reports.

Staffer identity + competence-weighted retrieval (Phase 23)

Each scenario run carries an explicit staffer: {id, name, tenure_months, role, tool_level}. The KB aggregates per-staffer stats (data/_kb/staffers.jsonl) that roll up into a single competence_score:

competence_score = 0.45·fill_rate + 0.20·turn_efficiency + 0.20·citation_density + 0.15·rescue_rate

When any query runs kb.findNeighbors(spec, k), the ranking isn't just cosine similarity — it's weighted_score = cosine × max_staffer_competence over the best coordinator who ran that signature. Senior staffers' playbooks surface above juniors' on similar scenarios, even when the juniors' scenario was marginally closer in embedding space.

The tool_level knob (full / local / basic / minimal) controls which tiers are available to a given staffer's runs. See Ch 3 for the mapping. Variance is real and measurable: 36-run demo produced a 46pt fill-rate delta between James (local tools, 93%) and Alex (minimal tools, 60%) on identical contracts.

Auto-discovered reliable-performer labels

A second-order output of the competence path: when multiple staffers independently endorse the same worker on similar-signature playbooks, that worker accumulates cross-staffer endorsements. scripts/kb_staffer_report.py surfaces them — after 36 runs, Rachel D. Lewis (Welder Nashville) had 18 endorsements across 4 staffers, Angela U. Ward (Machine Op Indianapolis) 19. These are high-confidence "reliable" labels the system produced without human tagging. The UI could badge these workers on future queries; today they're visible via /memory/query's discovered_patterns bundle.

A day in the life — from morning brief to EOD retrospective

SMS + email drafts in the pipeline

After each sealed fill (via scenario.ts or manual /log flow with downstream hooks), generateArtifacts in the scenario runner produces: (a) one SMS per worker (TO: Name, message under 180 chars), (b) one client confirmation email. Drafts are saved to sms.md and emails.md under the scenario output dir. Ollama drafts them; the staffer reviews and sends. No auto-send; human-in-the-loop.

Known limits & non-goals

Deferred — real architectural work, just not shipped yet

• Rate / margin awareness. Worker pay expectations vs contract bill rate not modeled. Requires adding pay_rate to workers, bill_rate to contracts, and a filter + warning path. Partially addressed via ContractTerms.budget_per_hour_max passed to T3/rescue prompts, but the match-time filter isn't wired yet.
• Mem0-style UPDATE / DELETE / NOOP operations on playbooks. Today /seed only ADDs. Same (operation, date) pair appends a duplicate instead of refining an existing entry. Phase 26 item — cheap to add, moderate payoff.
• Letta working-memory hot cache. Every boost query scans all active playbook entries from in-memory state. 1.9K today; cheap. Will bite somewhere north of 100K. LRU for the last-N playbooks or current-sig neighborhood deferred until that ceiling approaches.
• Chunking cache (Phase 21 Rust port). TS primitives generateContinuable + generateTreeSplit are wired, but crates/aibridge/src/{continuation.rs, tree_split.rs} + crates/storaged/src/chunk_cache.rs remain queued. Gateway-side callers currently don't have the same protection against silent truncation that the TS test harness does.
• Confidence calibration. Top-K is a rank, not a probability. No calibrated "85% likely to accept" score. Requires outcome-labeled training data.
• Neural re-ranker. Phase 19 is statistical + semantic (now with geo + role prefilter, Phase 25 retirement). A (query, candidate, outcome)-trained re-ranker is deferred only if the statistical floor plateaus below usable recall — current 14× citation lift on identical inputs suggests it hasn't.
• Observer → autotune feedback wire. Phase 24 streams scenario outcomes into data/_observer/ops.jsonl; autotune agent still runs on its own HNSW-trial schedule and hasn't subscribed to the outcome metric stream yet. Phase 26+ item — connects the last loop.
• call_log cross-reference. Infrastructure present; current synthetic candidates table is too small to cross-ref. Fixes when real ATS lands.

Non-goals — explicitly out of scope

• Cloud deployment. Local-first by design. Works offline after setup.
• Full ACID transactions. Single-writer model is sufficient; Delta Lake-grade MVCC is deliberately not attempted.
• Real-time streaming / CDC. Batch ingest is the model. Scheduled refresh, not transactional replication.
• Replacing the CRM. This is the analytical + AI layer behind the CRM. Operational CRUD stays with the existing system.
• Custom file formats. Parquet for datasets, sidecar indexes for vectors. No proprietary formats (ADR-008, ADR-018 reaffirm).
• Hard multi-tenant isolation. Profiles and federation provide soft isolation. Adversarial multi-tenant is not a goal — this system assumes a single-trust operator.