- Add iteration tracking and stuck detection to orchestrator
- Add triggerAutoRecovery function for automatic pipeline respawn
- Store structured failure context (proposals, conflicts, reason)
- Force GAMMA agent on recovery attempts for conflict resolution
- Limit auto-recovery to 3 attempts to prevent infinite loops
- Add UI status badges for rebooting/aborted states
- Add failure-context API endpoint for orchestrator handoff
- Add test_auto_recovery.py with 6 passing tests
Exit codes: 0=success, 1=error, 2=consensus failure, 3=aborted
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major additions:
- marketplace/: Agent template registry with FTS5 search, ratings, versioning
- observability/: Prometheus metrics, distributed tracing, structured logging
- ledger/migrations/: Database migration scripts for multi-tenant support
- tests/governance/: 15 new test files for phases 6-12 (295 total tests)
- bin/validate-phases: Full 12-phase validation script
New features:
- Multi-tenant support with tenant isolation and quota enforcement
- Agent marketplace with semantic versioning and search
- Observability with metrics, tracing, and log correlation
- Tier-1 agent bootstrap scripts
Updated components:
- ledger/api.py: Extended API for tenants, marketplace, observability
- ledger/schema.sql: Added tenant, project, marketplace tables
- testing/framework.ts: Enhanced test framework
- checkpoint/checkpoint.py: Improved checkpoint management
Archived:
- External integrations (Slack/GitHub/PagerDuty) moved to .archive/
- Old checkpoint files cleaned up
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements detection and recovery for when agents fail to reach consensus:
- Orchestrator exits with code 2 on consensus failure (distinct from error=1)
- Records failed run context (proposals, agent states, conflicts) to Dragonfly
- Provides fallback options: rerun same, rerun with GAMMA, escalate tier, accept partial
- Adds UI alert with action buttons for user-driven recovery
- Adds failure details modal and downloadable failure report
- Only marks pipeline complete when consensus achieved or user accepts fallback
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Vault token issuance per pipeline with 2-hour TTL
- Automatic token renewal loop every 30 minutes
- Error budget tracking with threshold-based revocation
- Observability-driven token revocation for policy violations
- Diagnostic pipeline spawning on error threshold breach
- Structured handoff reports for error recovery
- Agent lifecycle status API
- New API endpoints: /api/pipeline/token, /api/pipeline/errors,
/api/observability/handoff, /api/observability/diagnostic
Orchestrator now reports errors to parent pipeline's observability
system via PIPELINE_ID environment variable.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The orchestrator process was hanging after completing its work because:
1. Fire-and-forget Redis operations in MessageBus.handleMessage() left
unhandled promises that kept the event loop alive
2. No explicit process.exit() call after cleanup
Changes:
- coordination.ts: Add .catch(() => {}) to fire-and-forget Redis ops
- orchestrator.ts: Add explicit process.exit(exitCode) after cleanup
- orchestrator.ts: Improve error handling in main() with proper exit codes
Tested: Pipeline mksup1wq completed full flow and exited cleanly.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
