llm-team-ui/docs/PRD.md

LLM Team UI — Product Requirements (for scrum-master audits)

This doc is the anchor the lakehouse scrum-master uses when auditing this codebase. It is the "ground truth" — every scrum finding is evaluated against what this doc says the system should be. Written 2026-04-24 to give the scrum pipeline a target surface; keep it tight and aligned with what the code actually tries to do. Last reconciled 2026-06-20 against commit ea0ba33 — the mode registry and unknown-mode handling described below were verified against the live source on that date.

What LLM Team UI is

Flask-based web UI that lets a user configure and run multi-model "teams" — pipelines of LLMs (OpenRouter + Ollama Cloud + local Ollama) that collaborate on a task with roles like executor, reviewer, critic, sentinel. Lives at devop.live:5000 behind nginx. Main file llm_team_ui.py handles HTTP routes, session auth, model orchestration, SSE streaming. Backed by PostgreSQL for users + runs + audit.

Core invariants (scrum should flag if violated)

1. Every mode accepted by /api/run must be in the _VALID_MODES registry and have a matching run_<mode> handler. As of 2026-06-20 there are 24 registered modes (brainstorm, pipeline, debate, validator, roundrobin, redteam, consensus, codereview, ladder, tournament, evolution, blindassembly, staircase, drift, mesh, hallucination, timeloop, research, eval, extract, refine, adaptive, deep_analysis, distill), each with a run_<mode> function. run_team() validates mode against _VALID_MODES before opening the SSE stream and rejects anything else with HTTP 400 (see invariant below). The registry is kept in sync with the handlers manually — adding a run_<mode> without listing it in _VALID_MODES (or vice-versa) is the hard fail to flag.
2. Unknown modes must fail fast with HTTP 400 + the valid-mode list. run_team() returns {"error": "Unknown mode: X", "valid_modes": [...]} with status 400 before streaming. Returning 200-with-error-body (the old behavior) is a hard fail — it forces callers to parse SSE to learn the request was bad.
3. All authenticated routes must call @login_required decorator. The SESSION_COOKIE_HTTPONLY + SAMESITE=Lax + SECURE defaults must hold (SECURE added 2026-06-20; the app sits behind nginx TLS). Any route that mutates state without auth is a hard fail.
4. Security logging goes to /var/log/llm-team-security.log via the sec_log logger. Fail2ban watches this file. Any new auth path must emit there on failure.
5. PostgreSQL connections come from DB_URL env. Direct literal connection strings are a hard fail.
6. SMTP alerts via send_security_alert(subject, body) — never log secrets in the body.
7. SSE streams must flush incrementally — if an endpoint returns all-at-once, it should not claim SSE in the Content-Type.
8. Password storage: bcrypt only. Any plain-text / SHA256 password path is a hard fail.
9. Rate limiting enforced on /login, /register, and model-invocation endpoints. No unbounded loops that invoke paid APIs.

Known gaps (scrum can work on these)

• Only extract mode is registered. RESOLVED (2026-06-20). The mode set grew from one handler to 24 (run_brainstorm … run_distill), all listed in _VALID_MODES and dispatched by run_team(). Note the mode named in code is codereview (not code_review); callers using the underscored name still get a 400. Historical context: observer.ts in lakehouse was escalating to /api/run?mode=code_review and silently hit "Unknown mode" for weeks (see lakehouse memory reference_llm_team_modes.md) — the underscore-vs-no-underscore mismatch is the remaining footgun, not the missing handler.
• File is 13K lines. Cohesive modules (auth, teams, runs, alerts, SSE, templates) should be separated — but that's a refactor, out of scope for individual scrum-master reviews.
• Audit log schema is thin. The PostgreSQL runs table stores the fact of a run but not fine-grained per-step outcomes. Downstream correlation is manual.
• Templates inline in Python as render_template_string(...). Makes HTML review impossible via the scrum pipeline — HTML is hidden inside Python string literals.

Change proposal axis (what the scrum should optimize for)

1. Every Unknown-mode endpoint should either be registered or return 404/not-a-mode. DONE (2026-06-20). run_team() now returns HTTP 400 (not 404 — chosen so it reads as "bad request / wrong mode" rather than "route missing") with a valid_modes list, before streaming. Callers can now distinguish a wrong call from a missing mode. Remaining: see the codereview vs code_review naming footgun in Known gaps.
2. Auth decorators should be typed — @login_required is a runtime check; adding an Annotated[User, Depends(auth)]-style type hint (possible via Flask's g object) would make review mechanical.
3. Security logging should be structured — current sec_log.warning( f"...") loses fields. Move to logger.warning(msg, extra={...}) so fail2ban and external tools can parse.
4. Secret handling — FLASK_SECRET in env is correct, but the fallback to ~/.llm-team-secret means dev mode silently generates a persistent secret. Not bad, but should log once.

How scrum-master audits this repo

The lakehouse scrum pipeline is repo-agnostic via env vars. See /root/llm-team-ui/CLAUDE.md for the exact runbook.