agent-governance/pipeline

Agent Governance Pipeline System

This directory contains the authoritative pipeline implementation for the AI Agent Governance System.

Architecture Reference

See /opt/agent-governance/docs/ARCHITECTURE.md for the full system design.

Directory Structure

pipeline/
├── core.py                    # AUTHORITATIVE: Core definitions, enums, constants
├── pipeline.py                # Pipeline DSL parser and executor
├── README.md                  # This file
├── schemas/
│   └── pipeline.schema.json   # JSON Schema for pipeline validation
├── templates/
│   ├── default.yaml           # Generic observer-tier agent
│   ├── terraform.yaml         # Infrastructure specialist (T1)
│   ├── ansible.yaml           # Configuration management (T1)
│   └── code-review.yaml       # Code review specialist (T0)
└── examples/
    ├── infrastructure-deploy.yaml
    └── multi-agent-analysis.yaml

Core Module (core.py)

All code should import pipeline definitions from pipeline.core to ensure consistency.

Agent Lifecycle Phases

The official agent lifecycle follows these phases in order:

BOOTSTRAP → PREFLIGHT → PLAN → EXECUTE → VERIFY → PACKAGE → REPORT → EXIT

Phase	Description	Output Type
BOOTSTRAP	Agent initialization and authentication	Alpha
PREFLIGHT	Pre-execution validation (sandbox, inventory, deps)	Alpha
PLAN	Generate and validate execution plan	Beta
EXECUTE	Perform the planned actions	Beta
VERIFY	Validate execution results	Gamma
PACKAGE	Bundle artifacts and evidence	Gamma
REPORT	Generate completion report	Gamma
EXIT	Clean shutdown and resource release	Gamma
REVOKED	Agent was revoked (terminal state)	-

Importing Core Definitions

from pipeline.core import (
    # Enums
    AgentPhase,
    AgentStatus,
    OutputType,
    ChaosCondition,
    StageType,
    StageStatus,

    # Data classes
    AgentOutput,
    ClarifiedPlan,
    ErrorBudget,
    StageResult,
    PipelineContext,

    # Constants
    AGENT_PHASE_NAMES,
    AGENT_PHASES_ORDERED,
    PHASE_OUTPUT_TYPES,
    DEFAULT_REDIS_HOST,
    DEFAULT_REDIS_PORT,
    DEFAULT_REDIS_PASSWORD,
    DEFAULT_LEDGER_PATH,

    # Key patterns
    RedisKeys,

    # Utilities
    get_output_type_for_phase,
    is_terminal_phase,
    next_phase,
)

Output Types (Alpha/Beta/Gamma)

Agents produce outputs at checkpoints, classified as:

• Alpha: Initial/draft outputs (plans, analysis)
• Beta: Refined outputs (validated plans, partial results)
• Gamma: Final outputs (completed work, verified results)

DragonflyDB Key Patterns

Use RedisKeys class for consistent key naming:

from pipeline.core import RedisKeys

# Agent keys
agent_state = RedisKeys.agent_state("agent-001")  # "agent:agent-001:state"
agent_lock = RedisKeys.agent_lock("agent-001")    # "agent:agent-001:lock"

# Project keys
project_agents = RedisKeys.project_agents("proj-001")  # "project:proj-001:agents"

Pipeline DSL (pipeline.py)

The pipeline DSL supports four stage types:

Stage Types

1. agent: Execute an agent task
2. gate: Approval/consensus checkpoint
3. parallel: Concurrent execution of branches
4. condition: Conditional branching (if/then/else)

Example Pipeline

name: example-pipeline
version: "1.0"
timeout: 30m

stages:
  - name: plan
    type: agent
    template: default
    config:
      tier: 1
      timeout: 10m

  - name: review
    type: gate
    requires: [plan]
    config:
      gate_type: approval
      approvers: ["team-lead"]
      timeout: 30m

  - name: execute
    type: agent
    requires: [review]
    template: terraform
    config:
      tier: 2

Running a Pipeline

# Validate a pipeline
python pipeline/pipeline.py validate pipeline/examples/infrastructure-deploy.yaml

# Run a pipeline
python pipeline/pipeline.py run pipeline/examples/infrastructure-deploy.yaml \
    --input environment=staging

# List available templates
python pipeline/pipeline.py list

Chaos Testing Integration

The chaos test framework (tests/multi-agent-chaos/) imports from pipeline.core to ensure consistency:

# In tests/multi-agent-chaos/orchestrator.py
from pipeline.core import (
    AgentPhase,
    OutputType,
    ChaosCondition,
    AGENT_PHASE_NAMES,
    RedisKeys,
)

Running Chaos Tests

python tests/multi-agent-chaos/orchestrator.py

The chaos test:

1. Spawns multiple real agents (Python, Bun, Diagnostic)
2. Injects chaos conditions (lock loss, error spikes, etc.)
3. Tracks Alpha/Beta/Gamma outputs
4. Triggers plan clarification when error threshold crossed
5. Verifies unified objective reached via DragonflyDB readiness checks

Files Changed for Unification

The following files were aligned with the architecture spec:

File	Change
pipeline/core.py	NEW: Authoritative definitions
tests/multi-agent-chaos/orchestrator.py	Updated to import from pipeline.core
pipeline/README.md	NEW: This documentation

Consistency Checklist

When adding new pipeline-related code:

1. Import from pipeline.core - never define enums/constants locally
2. Use official phase names - AGENT_PHASE_NAMES or AGENT_PHASES_ORDERED
3. Use RedisKeys class - for consistent DragonflyDB key naming
4. Follow output types - PHASE_OUTPUT_TYPES maps phases to Alpha/Beta/Gamma
5. Include PACKAGE phase - often forgotten, but required for artifact bundling

Version History

• v1.0 (2026-01-23): Initial unified pipeline consolidation