7 Commits

Author SHA1 Message Date
root
db2e57402e STATE_OF_PLAY: capture multi-coord stress wave (Phase 1-3 verified)
Anchor was last touched at v4 split-threshold; since then the
multi-coord stress harness landed end-to-end across 11 commits.
Future sessions reading this file need to see the verified state,
not derive from git log.

Major additions:
- New "Multi-coordinator stress test (Phase 1 → 3)" section in
  VERIFIED WORKING. 11-row capability table covering per-coord
  playbook isolation, diversity metrics, paraphrase handover,
  ExcludeIDs swap, fresh-resume two-tier, inbox endpoints, LLM
  demand parsing, judge re-rating, Langfuse tracing.
- Substrate-gains list under that section: ExcludeIDs on
  SearchRequest, observer.SourceInbox + /observer/inbox,
  internal/langfuse client, embedd default bumped to v2-moe,
  two-tier fresh_workers index pattern.
- Last-verified bumped to 16:42 CDT on the run #011 anchor.

DO NOT RELITIGATE expanded with five new locks:
1. Boost / inject use SEPARATE thresholds (0.5 / 0.20)
2. Multi-coord product theory is empirically VALIDATED
3. Fresh content uses two-tier indexing (fresh_workers)
4. embedd.default_model = nomic-embed-text-v2-moe (don't downgrade)
5. Inbox flow: parse + search + judge + trace
6. Langfuse Go-side client lives at internal/langfuse/

OPEN list refresh:
- Removed: re-judge metric (shipped as b13b5cd), adjacent-query as
  separate item (folded into a single "judge-approves-before-inject"
  follow-up), liberal-paraphrase (kept).
- Added: real-time 48-hour clock, wider Langfuse instrumentation,
  periodic fresh→main merge job.

RECENT VERIFIED WAVE table extended with 11 commits (b13b5cd..5d49967).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 17:30:04 -05:00
root
87cbd10090 STATE_OF_PLAY: v4 split-threshold result + adjacent-query observation
- Reality test table extends from #001-#003 to #001-#004; v4 row marked
  as "the honest configuration" because OOD cross-pollination is gone.
- Shape B section gains the split-threshold rationale (boost safe at
  loose, inject structurally riskier so tighter).
- Verbatim drop framing rewritten — v3→v4 is configuration evolution,
  not regression.
- OPEN: closed "Shape B cap/decay" + the conditional Q15 boost-math
  item (Shape B + split threshold addressed both). Replaced with two
  finer-grained follow-ups: adjacent-query Q6↔Q7 swap (might be
  correct, verify with v4 re-judge metric) and liberal-paraphrase
  recovery loss (Q9/Q15 missed because qwen2.5 drifted >0.20).
- RECENT VERIFIED WAVE adds 94fc3b6 + 67d1957.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 07:26:23 -05:00
root
94fc3b67ec STATE_OF_PLAY: capture v3 reality test + Shape B + cross-pollination
- Reality test section now spans v1/v2/v3 across one table — the
  product story (boost-only verbatim → paraphrase gap → Shape B
  closes the gap) is legible without reading the reports.
- Verbatim-lift drop v1→v3 (7→2) explicitly framed as
  cross-pollination, NOT regression — and filed as v4 re-judge metric
  in OPEN.
- "DO NOT RELITIGATE" gains: Shape B is the stance now (don't revert
  to boost-only); local_judge stays on qwen2.5 (don't bump to qwen3.5
  for cleanliness — vision-SSM cost geometry).
- OPEN list: removed the now-closed paraphrase v2 row + the boost-math
  Q15 row (Shape B may have addressed it; flagged for verify after v4).
  Added v4 re-judge metric and Shape B injection cap/decay design call.
- RECENT VERIFIED WAVE adds the four new commits past 6c02c90
  (2c71d1c, 9ce067b, e9822f0, 154a72e).
- Matrix indexer §5/5 component description now references
  InjectPlaybookMisses + the run #002→#003 evidence chain.
- [models] tier registry comment locks the local_judge=qwen2.5 choice
  with the rationale inline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 07:09:31 -05:00
root
2c71d1c637 ADR-005: observer fail-safe semantics
Closes the OPEN item from STATE_OF_PLAY. Required because observerd is
now on the prod-realistic data path via the lift harness boot (b2e45f7),
so the next consumer (scrum runner / distillation rebuild / production
workflow) needs the fail-safe rationale locked, not implicit.

The Rust "verdict:accept on crash" anti-pattern doesn't translate
one-to-one to the Go observer (witness, not gate). But four adjacent
fail-safe decisions are real and live:

5.1 Persist failure is logged-not-fatal; ring is in-flight source of
    truth. Persist-required mode deferred to a future opt-in ADR.

5.2 Mode failure → Success=false, no panic-swallow path. The runner
    catches mode errors and surfaces them via node.Error; downstream
    consumers see failures explicitly rather than as fake successes
    (the Rust anti-pattern surface).

5.3 One row per node, recorded post-run. A workflow with N nodes
    produces N audit rows, never a per-workflow catch-all that
    survives partial crashes. Known gap: recording happens after
    runner.Run returns (acceptable for short workflows; streaming
    callback is the right shape when workflows get longer).

5.4 /observer/event accepts on full ring (oldest evicted). Refusing
    to write would translate every burst into client errors — wrong
    direction for an audit witness.

Mostly ratifies existing behavior; cross-checked claims against
actual code (caught one error in Decision 5.3 draft — recording is
post-run-batched, not per-node-as-it-completes — and the ADR now
states reality).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 06:32:12 -05:00
root
b2e45f7f26 playbook_lift: harness expansion + reality test #001 (7/8 lift, 87.5%)
The 5-loop substrate's load-bearing gate is verified — playbook +
matrix indexer give the results we're looking for. Per the report's
rubric, lift ≥ 50% of discoveries means matrix is doing real work;
7/8 = 87.5% blew through that.

Harness was structurally hiding bugs behind a 5-daemon stripped boot.
Expanding to the full 10-daemon prod stack surfaced 7 fixes in cascade:

1. driver→matrixd: {"query": ...} → {"query_text": ...} field name
2. harness temp toml missing [s3] → wrong default bucket → catalogd
   rehydrate 500 on first call
3. harness→queryd SQL probe: {"q": ...} → {"sql": ...} field name
4. expand boot from 5 → 10 daemons in dep-ordered launch
5. add SQL surface probe (3-row CSV ingest → COUNT(*)=3 assertion)
6. candidates corpus was synthetic SWE-tech (Swift/iOS, Scala/Spark) —
   wrong domain for staffing queries; replaced with ethereal_workers
   (10K rows, real staffing schema, "e-" id prefix to avoid collision
   with workers' "w-"). staffing_workers driver gains -index-name +
   -id-prefix flags so the same binary serves both corpora
7. local_judge qwen3.5:latest is a vision-SSM 256K-ctx build running
   ~30s per judge call against the lift loop; reverted to
   qwen2.5:latest (~1s/call, 30× faster, held lift theory)

Each contract drift (1, 3) is now locked into a cmd/<bin>/main_test.go
so future drift fires in `go test`, not in a reality run. R-005 closed:

- cmd/matrixd/main_test.go (new) — playbook record drift detector +
  score bounds + 6 routes mounted
- cmd/queryd/main_test.go — wrong-field-name drift detector
- cmd/pathwayd/main_test.go (new) — 9 routes + add round-trip + retire
- cmd/observerd/main_test.go (new) — 4 routes + invalid-op + unknown-mode

`go test ./cmd/{matrixd,queryd,pathwayd,observerd}` all green.

Reality test results (reports/reality-tests/playbook_lift_001.{json,md}):
  Queries              21 (staffing-domain, 7 categories)
  Discoveries          8 (judge ≠ cosine top-1)
  Lifts                7/8 (87.5%)
  Boosts triggered     9
  Mean Δ distance      -0.053 (warm closer than cold)
  OOD honesty          dental/RN/SWE rated 1, no fake matches
  Cross-corpus boosts  confirmed (e- ↔ w- swaps in lifts)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 06:22:21 -05:00
root
511083ae40 docs: SPEC §3.9 (chatd) + §3.10 (local-review-harness sibling)
- SPEC §1 component table: add chatd row marked DONE; replaces
  Rust gateway's v1::ollama_cloud / openrouter / opencode adapters
  + the aibridge crate.
- SPEC §3.9 — chatd shipped: 5-provider routing (ollama, ollama_cloud,
  openrouter, opencode, kimi) by model-name prefix or :cloud suffix.
  Captures the Anthropic 4.7 temperature-deprecation quirk + the
  local-Ollama think=false default that the playbook_lift judge
  needed. Mentions scrum_review.sh as the reusable cross-lineage
  vehicle eating chatd's own /v1/chat.
- SPEC §3.10 — local-review-harness sibling tool: separate repo at
  git.agentview.dev/profit/local-review-harness, MVP shipped today.
  Documents the cross-pollination plan for when both substrates
  stabilize (chatd as the harness's LLM backend; harness findings
  as Lakehouse pathway-memory drift signal; .memory/known-risks
  as a matrix corpus). Explicit "don't re-port" so future Claudes
  don't try to absorb the harness into Lakehouse.
- STATE_OF_PLAY.md: SIBLING TOOLS section with 1-line summary
  + pointer to SPEC §3.10.

No code changes. just verify still PASS — touched only docs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:01:23 -05:00
root
c5c31b6ca6 docs: STATE_OF_PLAY.md — Go-side truth anchor (mirrors Rust convention)
Adds the "verified working RIGHT NOW / DO NOT RELITIGATE / OPEN"
anchor at the repo root, mirroring /home/profit/lakehouse/STATE_OF_PLAY.md.
Memory files (project_golang_lakehouse.md) supplement; this file is
the verified-truth pointer.

Sections:
- VERIFIED WORKING: 13 cmd binaries + 18 smokes + 5 matrix components
  + Mem0 pathway + observerd + workflow runner + chatd 5-provider
  dispatcher + model tier registry. just verify PASS in 31s.
- DO NOT RELITIGATE: 4 ratified ADRs (DECISIONS.md ADR-001..004) +
  today's scrum dispositions (B-1..B-4 fixed, FP-A1/A2/C1 dismissed)
  + session frame items (Rust legacy is maintenance-only, etc.).
- OPEN: reality test held on J's queries, 3 daemon main_test.go gap,
  Sprint 4 deployment, ADR-005 observer fail-safe, ADR-006 auth posture.
- RECENT WAVE: 6-commit table 05273ac..e4ee002 documenting today's
  4 phases + scrum + tooling.
- RUNTIME CHEATSHEET: just verify, chatd boot, /v1/chat/providers
  probe, scrum_review.sh usage.
- VISION: 5-loop substrate gate from project_small_model_pipeline_vision.md.

The read-mem skill (in /root/.claude/skills/read-mem/) and project
memory file are updated to reference this file as the primary Go anchor.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 00:37:24 -05:00