profit/local-review-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
local-review-harness/tests/fixtures/insecure-repo/reports/latest/scrum-test.md
Claude (review-harness setup) e346b54e0f Phase C — local-Ollama LLM review wired end-to-end 
Implements PROMPT.md / docs/REVIEW_PIPELINE.md Phase 2:
- internal/llm/ollama.go — real Ollama provider:
  - HealthCheck probes /api/tags + a 1-token completion + a JSON-mode
    probe ({"ok": true} round-trip), populating the model-doctor.json
    schema documented in docs/LOCAL_MODEL_SETUP.md
  - Complete + CompleteJSON via /api/chat with stream=false
  - think=false set for ALL completions (qwen3.5:latest is reasoning-
    capable but the inner-loop hot path wants direct answers, not
    reasoning traces consuming the token budget — same finding as
    the Lakehouse-Go chatd 2026-04-30 wave)
- internal/llm/review.go — Reviewer wrapper:
  - 2-attempt flow: prompt → parse → repair-prompt → parse
  - Strict JSON shape enforced; markdown fences stripped before parse
  - Severity normalized to enum; out-of-range confidence clamped
  - Per-file chunking (file-level for v0; function-level Phase D+)
  - Bounded by review-profile max_file_bytes + max_llm_chunk_chars
- pipeline.go — Phase 2 wired between static scan + report gen:
  - --enable-llm flag opts in (off by default — static-only is
    cheaper and faster)
  - Raw output ALWAYS saved to llm-findings.raw.json (forensics)
  - Normalized findings → llm-findings.normalized.json
  - LLM findings merged into the report findings list (sourced
    "llm" so consumers can filter)
  - Receipts honestly mark phase status: "ok" | "degraded" | "skipped"
- cli model doctor — real probes replace the Phase A stub.

Verified:
- model doctor: status="ok" with qwen3.5:latest + qwen3:latest both
  loaded, basic_prompt_ok=true, json_mode_ok=true
- insecure-repo with --enable-llm: 9 LLM findings; qwen3.5 correctly
  flagged SQLi, RCE, hardcoded credentials as critical with verbatim
  evidence; 27s wall for 3 chunks
- clean-repo with --enable-llm: 0 LLM findings, 4 parsed chunks, 2.8s
- self-review with --enable-llm: 77 LLM findings + 83 static; 3 of
  ~30 chunks needed retry (PROMPT.md, REPORT_SCHEMA.md,
  SCRUM_TEST_TEMPLATE.md — all eventually parsed); 5min wall

go vet + go test -short clean. Fixture stray.go now `package fixture`
so go-tooling doesn't choke on the orphan.

Phase D (validator cross-check) + Phase E (memory + diff/rules
subcommands) remain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:13:39 -05:00

5.0 KiB
Raw Blame History

Scrum Test — insecure-repo

Generated: 2026-04-30T06:07:13.931830669Z Branch: main · Commit: 70d68757f7

Verdict

blocked — critical-severity finding present. See Confirmed Risks; rotate any leaked credentials, then re-run.

Evidence

  • repo path: /home/profit/share/local-review-harness-full-md/tests/fixtures/insecure-repo
  • file count: 4
  • languages: Go (2), JavaScript (1)
  • dependency manifests: 0 ()
  • test files/dirs: 0
  • LLM review: skipped (Phase C not implemented OR provider unavailable; see model-doctor.json)

Confirmed Risks

Severity File:Line Title Evidence
high .env Environment file in source tree filename=.env
medium . No tests found No test files or test directories detected (looked for *_test.go, *.test.{js,ts}, test_*.py, tests/, spec/)

Suspected Risks

Each entry is a static-scan regex hit awaiting validation (Phase D / LLM cross-check).

Severity File:Line Title Evidence
critical src/handler.go:23 Possible secret committed to source const API_KEY = "sk-1234567890abcdefABCDEFGHIJKLMNOPQRSTUV"
critical src/handler.go:23 Possible secret committed to source const API_KEY = "sk-1234567890abcdefABCDEFGHIJKLMNOPQRSTUV"
critical src/server.js:5 Possible secret committed to source const AWS_KEY = "AKIAIOSFODNN7EXAMPLE";
high src/handler.go:14 Raw SQL interpolation q := fmt.Sprintf("SELECT * FROM users WHERE name = '%s'", name)
high src/handler.go:19 Shell command execution exec.Command("bash", "-c", cmd).Run()
high src/server.js:2 Wildcard CORS res.setHeader("Access-Control-Allow-Origin", "*");
medium src/handler.go:10 Hardcoded absolute path const HARDCODED_PATH = "/home/profit/secrets/key.pem"
medium src/handler.go:11 Hardcoded private-network IP const SERVER_IP = "192.168.1.176"
medium src/huge.go:1-901 Large file 901 lines (limit: 800)
medium src/server.js:7 Mutation route in file with no visible auth app.post("/api/users", function(req, res) { /* no auth */ });
medium src/server.js:8 Mutation route in file with no visible auth app.delete("/api/admin", function(req, res) { /* no auth */ });
low src/handler.go:22 TODO/FIXME comment // FIXME: hardcoded creds
low src/handler.go:9 TODO/FIXME comment // TODO: rotate this and move to env
low src/server.js:1 TODO/FIXME comment // HACK: open CORS for now

Blocked Checks

  • LLM review (Phase 2 in REVIEW_PIPELINE.md). Reason: provider unavailable or stub. Next command: review-harness model doctor

Sprint Backlog

Sprint 0 — Reproducibility Gate

  • Wire just verify (or equivalent) to run the static checks before every commit/PR.
  • Add a CI step that fails on critical findings.
  • Triage the 16 findings emitted by this run; mark each as accepted / blocking / dismiss-with-reason.

Sprint 1 — Trust Boundary Gate

  • Resolve every critical and high finding before non-loopback deploy.
  • Confirm auth posture for any mutation endpoint flagged as exposed.
  • Replace raw SQL interpolation with parameterized queries.

Sprint 2 — Memory Correctness Gate

  • (Phase E) Wire append-only .memory/ writes for known-risks + fixed-patterns.
  • Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.

Sprint 3 — Agent Loop Reality Gate

  • (Phase C) Wire local-Ollama LLM review.
  • (Phase D) Validator pass cross-checks every LLM finding against repo evidence.

Sprint 4 — Deployment Gate

  • Ship the harness as a single static binary (go build -o review-harness).
  • Document operator runbook (model setup, profile editing, output retention).

Acceptance Gates

Each gate must be testable. Format: command + verifiable post-condition.

  1. Reproducibility: review-harness repo . exits 0; reports/latest/repo-intake.json exists with non-zero file_count.
  2. No false positives on a clean fixture: review-harness repo tests/fixtures/clean-repo produces zero confirmed findings.
  3. Every documented static check fires on the insecure fixture: jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json ≥ 8.
  4. Receipts are honest about degraded phases: jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json lists every skipped/stubbed phase.
  5. Critical findings block production deploy: at least one critical finding is currently present; resolve before deploy.

Next Commands

  1. Open the risk register: cat reports/latest/risk-register.md
  2. Triage every critical finding; rotate any leaked credentials immediately.
  • Probe the model provider: review-harness model doctor
  • Re-run after fixes: review-harness repo /home/profit/share/local-review-harness-full-md/tests/fixtures/insecure-repo
  • Generate the full Scrum bundle: review-harness scrum /home/profit/share/local-review-harness-full-md/tests/fixtures/insecure-repo