profit/local-review-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
local-review-harness/docs/SELF_REVIEW_2026-04-30.md
Claude (review-harness setup) 70d68757f7 docs: SELF_REVIEW snapshot from f3ee472 (Phase A+B MVP) 
Per PROMPT.md "Final Deliverable" — capture the harness's review of
its own repo as a tracked artifact. reports/latest/ stays gitignored
for routine runs; this snapshot is the deliverable proof.

Captured: 28 findings, 3 critical (all from tests/fixtures/insecure-repo
planted secrets, intentionally), 16 high (mostly shell-exec patterns
mentioned by name in PROMPT.md and chatd-style scripts), 6 medium,
3 low. Verdict "blocked" because of fixture criticals.

Banner header notes that fixture findings are dismissable; real
codebase secrets land in identical shape — operators dismiss the
fixture path explicitly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 01:02:19 -05:00

116 lines
7.2 KiB
Markdown
Raw Permalink Blame History

# Scrum Test — local-review-harness-full-md
> **Self-review snapshot** — captured 2026-04-30 against commit `f3ee472` (Phase A + B MVP).
> Re-run with `./review-harness scrum .` for live numbers.
>
> The 3 critical findings below are from `tests/fixtures/insecure-repo/` — planted on
> purpose to prove the secret-pattern + env-file analyzers fire. Real codebase secrets
> would land in the same shape; operators dismiss fixture content as known-intentional.
**Generated:** 2026-04-30T05:55:10.657498039Z
**Branch:** (no git) · **Commit:** —
## Verdict
**blocked** — critical-severity finding present. See Confirmed Risks; rotate any leaked credentials, then re-run.
## Evidence
- repo path: `/home/profit/share/local-review-harness-full-md`
- file count: 36
- languages: Go (18), Markdown (7), TypeScript (2), YAML (2), JSON (1)
- dependency manifests: 3 (go.mod, go.sum, tests/fixtures/clean-repo/package.json)
- test files/dirs: 9
- LLM review: **skipped** (Phase C not implemented OR provider unavailable; see model-doctor.json)
## Confirmed Risks
| Severity | File:Line | Title | Evidence |
|---|---|---|---|
| high | `tests/fixtures/insecure-repo/.env` | Environment file in source tree | `filename=.env` |
## Suspected Risks
Each entry is a static-scan regex hit awaiting validation (Phase D / LLM cross-check).
| Severity | File:Line | Title | Evidence |
|---|---|---|---|
| critical | `tests/fixtures/insecure-repo/src/handler.go:23` | Possible secret committed to source | `const API_KEY = "sk-1234567890abcdefABCDEFGHIJKLMNOPQRSTUV"` |
| critical | `tests/fixtures/insecure-repo/src/handler.go:23` | Possible secret committed to source | `const API_KEY = "sk-1234567890abcdefABCDEFGHIJKLMNOPQRSTUV"` |
| critical | `tests/fixtures/insecure-repo/src/server.js:5` | Possible secret committed to source | `const AWS_KEY = "AKIAIOSFODNN7EXAMPLE";` |
| high | `PROMPT.md:419` | Shell command execution | `- `exec(`` |
| high | `PROMPT.md:420` | Shell command execution | `- `spawn(`` |
| high | `PROMPT.md:421` | Shell command execution | `- `Command::new`` |
| high | `PROMPT.md:423` | Raw SQL interpolation | `- `format!("SELECT`` |
| high | `PROMPT.md:429` | Wildcard CORS | `- `Access-Control-Allow-Origin: *`` |
| high | `internal/analyzers/checks.go:101` | Shell command execution | ``Command::new\|` + // Rust` |
| high | `internal/analyzers/checks.go:140` | Raw SQL interpolation | `rawSQLFmtRe = regexp.MustCompile(`(?i)(?:format!\|fmt\.Sprintf\|String::from\|f"\|f')[^\n]{0,80}?(?:SELECT\|INSERT\|UPDA…` |
| high | `internal/analyzers/checks.go:143` | Raw SQL interpolation | `rawSQLConcatRe = regexp.MustCompile(`(?i)(?:SELECT\|INSERT\|UPDATE\|DELETE)\b[^\n]{0,40}(?:\+\s*\w\|%s\|%v\|\$\{\|` + "`…` |
| high | `internal/analyzers/checks.go:177` | Wildcard CORS | `// styles: Express's res.setHeader("Access-Control-Allow-Origin", "*"),` |
| high | `internal/analyzers/checks.go:180` | Wildcard CORS | `var corsAnyRe = regexp.MustCompile(`Access-Control-Allow-Origin[^\n]{0,40}\*`)` |
| high | `internal/analyzers/checks.go:194` | Wildcard CORS | `Reason: "Access-Control-Allow-Origin: * permits cross-origin reads from any domain. Narrow to an explicit allowlist un…` |
| high | `internal/analyzers/checks.go:89` | Shell command execution | `// === 2. shell execution (exec, spawn, Command::new, subprocess) ===` |
| high | `tests/fixtures/insecure-repo/src/handler.go:14` | Raw SQL interpolation | `q := fmt.Sprintf("SELECT * FROM users WHERE name = '%s'", name)` |
| high | `tests/fixtures/insecure-repo/src/handler.go:19` | Shell command execution | `exec.Command("bash", "-c", cmd).Run()` |
| high | `tests/fixtures/insecure-repo/src/server.js:2` | Wildcard CORS | `res.setHeader("Access-Control-Allow-Origin", "*");` |
| medium | `review-harness:1-18860` | Large file | `18860 lines (limit: 800)` |
| medium | `tests/fixtures/insecure-repo/src/handler.go:10` | Hardcoded absolute path | `const HARDCODED_PATH = "/home/profit/secrets/key.pem"` |
| medium | `tests/fixtures/insecure-repo/src/handler.go:11` | Hardcoded private-network IP | `const SERVER_IP = "192.168.1.176"` |
| medium | `tests/fixtures/insecure-repo/src/huge.go:1-901` | Large file | `901 lines (limit: 800)` |
| medium | `tests/fixtures/insecure-repo/src/server.js:7` | Mutation route in file with no visible auth | `app.post("/api/users", function(req, res) { /* no auth */ });` |
| medium | `tests/fixtures/insecure-repo/src/server.js:8` | Mutation route in file with no visible auth | `app.delete("/api/admin", function(req, res) { /* no auth */ });` |
| low | `tests/fixtures/insecure-repo/src/handler.go:22` | TODO/FIXME comment | `// FIXME: hardcoded creds` |
| low | `tests/fixtures/insecure-repo/src/handler.go:9` | TODO/FIXME comment | `// TODO: rotate this and move to env` |
| low | `tests/fixtures/insecure-repo/src/server.js:1` | TODO/FIXME comment | `// HACK: open CORS for now` |
## Blocked Checks
- LLM review (Phase 2 in REVIEW_PIPELINE.md). Reason: provider unavailable or stub. Next command: `review-harness model doctor`
## Sprint Backlog
**Sprint 0 — Reproducibility Gate**
- Wire `just verify` (or equivalent) to run the static checks before every commit/PR.
- Add a CI step that fails on `critical` findings.
- Triage the 28 findings emitted by this run; mark each as accepted / blocking / dismiss-with-reason.
**Sprint 1 — Trust Boundary Gate**
- Resolve every `critical` and `high` finding before non-loopback deploy.
- Confirm auth posture for any mutation endpoint flagged as exposed.
- Replace raw SQL interpolation with parameterized queries.
**Sprint 2 — Memory Correctness Gate**
- (Phase E) Wire append-only `.memory/` writes for known-risks + fixed-patterns.
- Add a regression test that re-runs the harness and asserts no regression in confirmed-finding count.
**Sprint 3 — Agent Loop Reality Gate**
- (Phase C) Wire local-Ollama LLM review.
- (Phase D) Validator pass cross-checks every LLM finding against repo evidence.
**Sprint 4 — Deployment Gate**
- Ship the harness as a single static binary (`go build -o review-harness`).
- Document operator runbook (model setup, profile editing, output retention).
## Acceptance Gates
Each gate must be testable. Format: command + verifiable post-condition.
1. **Reproducibility:** `review-harness repo .` exits 0; `reports/latest/repo-intake.json` exists with non-zero `file_count`.
2. **No false positives on a clean fixture:** `review-harness repo tests/fixtures/clean-repo` produces zero `confirmed` findings.
3. **Every documented static check fires on the insecure fixture:** `jq '[.findings[] | .check_id] | unique | length' reports/latest/static-findings.json` ≥ 8.
4. **Receipts are honest about degraded phases:** `jq '[.phases[] | select(.status == "degraded")]' reports/latest/receipts.json` lists every skipped/stubbed phase.
5. **Critical findings block production deploy:** at least one critical finding is currently present; resolve before deploy.
## Next Commands
1. Open the risk register: `cat reports/latest/risk-register.md`
2. Triage every `critical` finding; rotate any leaked credentials immediately.
- Probe the model provider: `review-harness model doctor`
- Re-run after fixes: `review-harness repo /home/profit/share/local-review-harness-full-md`
- Generate the full Scrum bundle: `review-harness scrum /home/profit/share/local-review-harness-full-md`
Reference in New Issue View Git Blame Copy Permalink