golangLAKEHOUSE

profit/golangLAKEHOUSE

Fork 0

Commit Graph

Author	SHA1	Message	Date
root	8278eb9a87	scrum2 cleanup: JSON-marshal in stringifyValue, drop dead detectCycle, name SourceWorkflow 5 small fixes from the §3.8 scrum2 review wave: - workflow.stringifyValue now JSON-marshals maps/slices instead of fmt.Sprint %v (Opus+Kimi convergent: LLM modes were getting Go's map[k:v] syntax, which is unparseable as JSON context). - workflow.detectCycle removed — duplicate of topoSort that discarded the useful node ID. Validate() now calls topoSort directly and returns its wrapped ErrCycle. - observer.SourceWorkflow named constant — was an implicit string cast (observer.Source("workflow")) at the cmd/observerd handler. - Unused context imports + dead silencer comments removed across workflow/modes.go and observerd/main.go. - Unused store parameter dropped from registerBuiltinModes (reserved comment removed; can be re-added when a mode actually needs it). just verify still PASS — these are pure cleanup, no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-29 23:16:07 -05:00
root	bc9ab93afe	H: observerd — autonomous-iteration witness loop (SPEC §2 port) Port of the load-bearing pieces of mcp-server/observer.ts (Rust system, 852 lines TS) per SPEC §2's named target. Implements PRD loop 3 ("Observer loop — watches each run, refines configs"). Routes (all under /v1/observer/* via gateway): GET /observer/health — liveness GET /observer/stats — total / successes / failures / by_source / recent_scenario_ops (matches Rust JSON shape exactly) POST /observer/event — record one ObservedOp; auto-defaults timestamp + source, validates required fields (endpoint), persists to JSONL, appends to ring buffer Architecture: - internal/observer/types.go — ObservedOp model + Source taxonomy (mcp / scenario / langfuse / overseer_correction). Mirrors the Rust shape so JSON round-trips during cutover. - internal/observer/store.go — Store + Persistor. Ring buffer cap matches Rust's 2000; recent_scenarios cap matches Rust's 10. Same persist-then-apply order as pathwayd; same corruption- tolerant replay (skip malformed lines + warn). - cmd/observerd — :3219 HTTP service, fronted by gateway as /v1/observer/*. - lakehouse.toml + DefaultConfig — [observerd] block matches the pathwayd pattern (Bind + PersistPath; empty path = ephemeral). Tests + smoke (all PASS): - 7 unit tests in store_test.go: validation, default fields, stats aggregation, recent-scenarios cap + ordering, ring-buffer rollover at cap, JSONL round-trip persistence, corruption- tolerant replay (1 valid + 1 corrupt + 1 valid → 2 applied) - scripts/observer_smoke.sh: 4 assertions through gateway — record 5 events (3 ok / 2 fail across 2 sources), stats aggregates correctly, empty-endpoint→400, kill+restart preserves via JSONL replay (5 ops, 3 ok, 2 err survive) Deferred (named in package + cmd doc, not in this commit): - POST /observer/review (cloud-LLM hand-review fall-back). The heuristic-only path could land cheaply but the productized cloud path (qwen3-coder fall-back) is multi-day port. - Background loops: analyzeErrors, consolidatePlaybooks, tailOverseerCorrections (read overseer_corrections.jsonl into the ring buffer once per cycle). - escalateFailureClusterToLLMTeam (failure clustering trigger that posts to LLM Team's /api/run with code_review mode). /relevance is NOT duplicated — already ported in 9588bd8 to internal/matrix/relevance.go (component 3 of SPEC §3.4). 16-smoke regression all green (D1-D6, G1, G1P, G2, storaged_cap, pathway, matrix, relevance, downgrade, playbook, observer). 13 binaries now: gateway, storaged, catalogd, ingestd, queryd, vectord, embedd, pathwayd, matrixd, observerd, mcpd, fake_ollama (plus catalogd-only test build). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-29 20:18:02 -05:00

Author

SHA1

Message

Date

root

8278eb9a87

scrum2 cleanup: JSON-marshal in stringifyValue, drop dead detectCycle, name SourceWorkflow

5 small fixes from the §3.8 scrum2 review wave:

- workflow.stringifyValue now JSON-marshals maps/slices instead of
  fmt.Sprint %v (Opus+Kimi convergent: LLM modes were getting Go's
  map[k:v] syntax, which is unparseable as JSON context).
- workflow.detectCycle removed — duplicate of topoSort that discarded
  the useful node ID. Validate() now calls topoSort directly and
  returns its wrapped ErrCycle.
- observer.SourceWorkflow named constant — was an implicit string
  cast (observer.Source("workflow")) at the cmd/observerd handler.
- Unused context imports + dead silencer comments removed across
  workflow/modes.go and observerd/main.go.
- Unused store parameter dropped from registerBuiltinModes (reserved
  comment removed; can be re-added when a mode actually needs it).

just verify still PASS — these are pure cleanup, no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-29 23:16:07 -05:00

root

bc9ab93afe

H: observerd — autonomous-iteration witness loop (SPEC §2 port)

Port of the load-bearing pieces of mcp-server/observer.ts (Rust
system, 852 lines TS) per SPEC §2's named target. Implements PRD
loop 3 ("Observer loop — watches each run, refines configs").

Routes (all under /v1/observer/* via gateway):
  GET  /observer/health   — liveness
  GET  /observer/stats    — total / successes / failures /
                             by_source / recent_scenario_ops
                             (matches Rust JSON shape exactly)
  POST /observer/event    — record one ObservedOp; auto-defaults
                             timestamp + source, validates required
                             fields (endpoint), persists to JSONL,
                             appends to ring buffer

Architecture:
  - internal/observer/types.go — ObservedOp model + Source taxonomy
    (mcp / scenario / langfuse / overseer_correction). Mirrors the
    Rust shape so JSON round-trips during cutover.
  - internal/observer/store.go — Store + Persistor. Ring buffer cap
    matches Rust's 2000; recent_scenarios cap matches Rust's 10.
    Same persist-then-apply order as pathwayd; same corruption-
    tolerant replay (skip malformed lines + warn).
  - cmd/observerd — :3219 HTTP service, fronted by gateway as
    /v1/observer/*.
  - lakehouse.toml + DefaultConfig — [observerd] block matches the
    pathwayd pattern (Bind + PersistPath; empty path = ephemeral).

Tests + smoke (all PASS):
  - 7 unit tests in store_test.go: validation, default fields,
    stats aggregation, recent-scenarios cap + ordering, ring-buffer
    rollover at cap, JSONL round-trip persistence, corruption-
    tolerant replay (1 valid + 1 corrupt + 1 valid → 2 applied)
  - scripts/observer_smoke.sh: 4 assertions through gateway —
    record 5 events (3 ok / 2 fail across 2 sources), stats
    aggregates correctly, empty-endpoint→400, kill+restart preserves
    via JSONL replay (5 ops, 3 ok, 2 err survive)

Deferred (named in package + cmd doc, not in this commit):
  - POST /observer/review (cloud-LLM hand-review fall-back). The
    heuristic-only path could land cheaply but the productized
    cloud path (qwen3-coder fall-back) is multi-day port.
  - Background loops: analyzeErrors, consolidatePlaybooks,
    tailOverseerCorrections (read overseer_corrections.jsonl into
    the ring buffer once per cycle).
  - escalateFailureClusterToLLMTeam (failure clustering trigger
    that posts to LLM Team's /api/run with code_review mode).

/relevance is NOT duplicated — already ported in 9588bd8 to
internal/matrix/relevance.go (component 3 of SPEC §3.4).

16-smoke regression all green (D1-D6, G1, G1P, G2, storaged_cap,
pathway, matrix, relevance, downgrade, playbook, observer).
13 binaries now: gateway, storaged, catalogd, ingestd, queryd,
vectord, embedd, pathwayd, matrixd, observerd, mcpd, fake_ollama
(plus catalogd-only test build).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-29 20:18:02 -05:00

2 Commits