agent-governance/pipeline/README.md

# Agent Governance Pipeline System

This directory contains the **authoritative pipeline implementation** for the AI Agent Governance System.

## Architecture Reference

See `/opt/agent-governance/docs/ARCHITECTURE.md` for the full system design.

## Directory Structure

```
pipeline/
├── core.py                    # AUTHORITATIVE: Core definitions, enums, constants
├── pipeline.py                # Pipeline DSL parser and executor
├── README.md                  # This file
├── schemas/
│   └── pipeline.schema.json   # JSON Schema for pipeline validation
├── templates/
│   ├── default.yaml           # Generic observer-tier agent
│   ├── terraform.yaml         # Infrastructure specialist (T1)
│   ├── ansible.yaml           # Configuration management (T1)
│   └── code-review.yaml       # Code review specialist (T0)
└── examples/
    ├── infrastructure-deploy.yaml
    └── multi-agent-analysis.yaml
```

## Core Module (`core.py`)

**All code should import pipeline definitions from `pipeline.core`** to ensure consistency.

### Agent Lifecycle Phases

The official agent lifecycle follows these phases in order:

```
BOOTSTRAP → PREFLIGHT → PLAN → EXECUTE → VERIFY → PACKAGE → REPORT → EXIT
```

| Phase | Description | Output Type |
|-------|-------------|-------------|
| BOOTSTRAP | Agent initialization and authentication | Alpha |
| PREFLIGHT | Pre-execution validation (sandbox, inventory, deps) | Alpha |
| PLAN | Generate and validate execution plan | Beta |
| EXECUTE | Perform the planned actions | Beta |
| VERIFY | Validate execution results | Gamma |
| PACKAGE | Bundle artifacts and evidence | Gamma |
| REPORT | Generate completion report | Gamma |
| EXIT | Clean shutdown and resource release | Gamma |
| REVOKED | Agent was revoked (terminal state) | - |

### Importing Core Definitions

```python
from pipeline.core import (
    # Enums
    AgentPhase,
    AgentStatus,
    OutputType,
    ChaosCondition,
    StageType,
    StageStatus,

    # Data classes
    AgentOutput,
    ClarifiedPlan,
    ErrorBudget,
    StageResult,
    PipelineContext,

    # Constants
    AGENT_PHASE_NAMES,
    AGENT_PHASES_ORDERED,
    PHASE_OUTPUT_TYPES,
    DEFAULT_REDIS_HOST,
    DEFAULT_REDIS_PORT,
    DEFAULT_REDIS_PASSWORD,
    DEFAULT_LEDGER_PATH,

    # Key patterns
    RedisKeys,

    # Utilities
    get_output_type_for_phase,
    is_terminal_phase,
    next_phase,
)
```

### Output Types (Alpha/Beta/Gamma)

Agents produce outputs at checkpoints, classified as:

- **Alpha**: Initial/draft outputs (plans, analysis)
- **Beta**: Refined outputs (validated plans, partial results)
- **Gamma**: Final outputs (completed work, verified results)

### DragonflyDB Key Patterns

Use `RedisKeys` class for consistent key naming:

```python
from pipeline.core import RedisKeys

# Agent keys
agent_state = RedisKeys.agent_state("agent-001")  # "agent:agent-001:state"
agent_lock = RedisKeys.agent_lock("agent-001")    # "agent:agent-001:lock"

# Project keys
project_agents = RedisKeys.project_agents("proj-001")  # "project:proj-001:agents"
```

## Pipeline DSL (`pipeline.py`)

The pipeline DSL supports four stage types:

### Stage Types

1. **agent**: Execute an agent task
2. **gate**: Approval/consensus checkpoint
3. **parallel**: Concurrent execution of branches
4. **condition**: Conditional branching (if/then/else)

### Example Pipeline

```yaml
name: example-pipeline
version: "1.0"
timeout: 30m

stages:
  - name: plan
    type: agent
    template: default
    config:
      tier: 1
      timeout: 10m

  - name: review
    type: gate
    requires: [plan]
    config:
      gate_type: approval
      approvers: ["team-lead"]
      timeout: 30m

  - name: execute
    type: agent
    requires: [review]
    template: terraform
    config:
      tier: 2
```

### Running a Pipeline

```bash
# Validate a pipeline
python pipeline/pipeline.py validate pipeline/examples/infrastructure-deploy.yaml

# Run a pipeline
python pipeline/pipeline.py run pipeline/examples/infrastructure-deploy.yaml \
    --input environment=staging

# List available templates
python pipeline/pipeline.py list
```

## Chaos Testing Integration

The chaos test framework (`tests/multi-agent-chaos/`) imports from `pipeline.core` to ensure consistency:

```python
# In tests/multi-agent-chaos/orchestrator.py
from pipeline.core import (
    AgentPhase,
    OutputType,
    ChaosCondition,
    AGENT_PHASE_NAMES,
    RedisKeys,
)
```

### Running Chaos Tests

```bash
python tests/multi-agent-chaos/orchestrator.py
```

The chaos test:
1. Spawns multiple real agents (Python, Bun, Diagnostic)
2. Injects chaos conditions (lock loss, error spikes, etc.)
3. Tracks Alpha/Beta/Gamma outputs
4. Triggers plan clarification when error threshold crossed
5. Verifies unified objective reached via DragonflyDB readiness checks

## Files Changed for Unification

The following files were aligned with the architecture spec:

| File | Change |
|------|--------|
| `pipeline/core.py` | NEW: Authoritative definitions |
| `tests/multi-agent-chaos/orchestrator.py` | Updated to import from `pipeline.core` |
| `pipeline/README.md` | NEW: This documentation |

## Consistency Checklist

When adding new pipeline-related code:

1. **Import from `pipeline.core`** - never define enums/constants locally
2. **Use official phase names** - `AGENT_PHASE_NAMES` or `AGENT_PHASES_ORDERED`
3. **Use RedisKeys class** - for consistent DragonflyDB key naming
4. **Follow output types** - `PHASE_OUTPUT_TYPES` maps phases to Alpha/Beta/Gamma
5. **Include PACKAGE phase** - often forgotten, but required for artifact bundling

## Version History

- **v1.0** (2026-01-23): Initial unified pipeline consolidation