agent-governance/docs/CONTEXT_MANAGEMENT.md

# Context Management System

> Unified guide to checkpoints, STATUS files, and the external memory layer.

## Overview

The Agent Governance System uses three integrated components to maintain context across long-running sessions while staying within token limits:

| Component | Purpose | Token Impact |
|-----------|---------|--------------|
| **Checkpoints** | Capture full session state at points in time | ~3000 tokens |
| **STATUS Files** | Track per-directory progress and tasks | ~50 tokens each |
| **Memory Layer** | Store large outputs externally with summaries | Minimal (refs only) |

```
┌─────────────────────────────────────────────────────────────────────┐
│                    CONTEXT MANAGEMENT SYSTEM                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐  │
│   │  Checkpoint  │◄───│    STATUS    │◄───│   Memory Layer       │  │
│   │              │    │    Files     │    │                      │  │
│   │ • Phase      │    │              │    │ • Large outputs      │  │
│   │ • Tasks      │    │ • Per-dir    │    │ • Auto-chunked       │  │
│   │ • Deps       │    │ • Progress   │    │ • Summarized         │  │
│   │ • Memory refs│    │ • Tasks      │    │ • Searchable         │  │
│   │ • Status sum │    │ • Memory ptr │    │                      │  │
│   └──────────────┘    └──────────────┘    └──────────────────────┘  │
│          │                   │                      │               │
│          └───────────────────┴──────────────────────┘               │
│                              │                                       │
│                    ┌─────────▼─────────┐                            │
│                    │   Token Window    │                            │
│                    │   (Summaries +    │                            │
│                    │    References)    │                            │
│                    └───────────────────┘                            │
└─────────────────────────────────────────────────────────────────────┘
```

## Quick Reference

### CLI Commands

```bash
# Checkpoints
checkpoint now --notes "Description"     # Create checkpoint
checkpoint load                          # Load latest
checkpoint report                        # Combined status view
checkpoint timeline                      # History view

# Status
status sweep                             # Check all directories
status sweep --fix                       # Create missing files
status update <dir> --phase <phase>      # Update directory
status dashboard                         # Overview

# Memory
memory log "content"                     # Store content
memory log --stdin                       # Store from pipe
memory fetch <id>                        # Retrieve full
memory fetch <id> -s                     # Summary only
memory list                              # List entries
memory search "query"                    # Search
memory stats                             # Statistics
```

## Component Details

### 1. Checkpoints

Checkpoints capture complete session state at a point in time.

**What's Captured:**
- Project phase and status
- Completed phases list
- Active tasks
- Dependency states (Vault, DragonflyDB, Ledger)
- Directory status summary (aggregated from STATUS files)
- Memory references (links to stored content)
- Variables and recent outputs

**When to Create:**
- After completing significant work
- Before context-heavy operations
- At natural stopping points
- Automatically when updating STATUS (creates delta)

**Commands:**
```bash
# Create with notes
checkpoint now --notes "Completed pipeline consolidation"

# Load latest
checkpoint load

# Load specific
checkpoint load ckpt-20260123-123456-abcd

# View combined report
checkpoint report

# View history
checkpoint timeline --limit 10

# Compare checkpoints
checkpoint diff --from ckpt-aaa --to ckpt-bbb
```

**Data Location:** `/opt/agent-governance/checkpoint/storage/`

---

### 2. STATUS Files

Every directory has `README.md` and `STATUS.md` for local context.

**README.md Contains:**
- Directory purpose
- Key files and their roles
- Interfaces/APIs
- Current status badge
- Link to parent architecture

**STATUS.md Contains:**
- Current phase (complete/in_progress/blocked/needs_review)
- Task checklist
- Dependencies
- Issues/blockers
- Activity log with timestamps

**Phase Values:**
| Phase | Icon | Meaning |
|-------|------|---------|
| `complete` | ✅ | Work finished |
| `in_progress` | 🚧 | Active development |
| `blocked` | ❗ | Waiting on dependencies |
| `needs_review` | ⚠️ | Requires attention |
| `not_started` | ⬜ | No work begun |

**Commands:**
```bash
# Check all directories
status sweep

# Auto-create missing files
status sweep --fix

# Update a directory
status update ./pipeline --phase complete --task "Unified with core.py"

# View dashboard
status dashboard

# Initialize new directory
status init ./new-module
```

**Integration with Checkpoints:**
- `status update` automatically creates a checkpoint delta
- Checkpoints aggregate status from all directories
- Use `--no-checkpoint` to skip delta creation

---

### 3. Memory Layer

External storage for large content that would overwhelm the token window.

**Token Thresholds:**
| Size | Strategy |
|------|----------|
| < 500 tokens | Inline in database |
| 500-4000 tokens | File + summary |
| > 4000 tokens | Auto-chunked + summary |

**Entry Types:**
| Type | Purpose |
|------|---------|
| `output` | Command outputs, logs |
| `transcript` | Conversation logs |
| `context` | Saved state |
| `chunk` | Part of larger entry |

**Commands:**
```bash
# Store content
memory log "Test output here"
memory log --file results.txt --tag "tests"
pytest 2>&1 | memory log --stdin --tag "pytest"

# Retrieve
memory fetch mem-xxx                    # Full content
memory fetch mem-xxx -s                 # Summary only
memory fetch mem-xxx --chunk 2          # Specific chunk

# Browse
memory list --type output --limit 10
memory search "error"
memory refs --checkpoint ckpt-xxx

# Maintain
memory stats
memory prune --keep-days 7
```

**Data Location:**
- Metadata: `/opt/agent-governance/memory/memory.db`
- Chunks: `/opt/agent-governance/memory/chunks/`

---

## Integration Patterns

### Pattern 1: Store Large Output

When a command produces large output:

```bash
# Instead of printing 10000 lines to chat
pytest tests/ 2>&1 | memory log --stdin --tag "pytest" --directory ./tests

# Report summary in chat
echo "Tests complete. Results stored in mem-xxx (8500 tokens)."
echo "Summary: 156 passed, 2 failed (test_auth, test_db)"
```

### Pattern 2: Link Memory to Status

In STATUS.md, reference stored content:

```markdown
## Context References

- Full test output: `mem-20260123-100000-abcd` (8500 tokens)
- Build log: `mem-20260123-100001-efgh` (3200 tokens)

Retrieve with: `memory fetch <id> --summary-only`
```

### Pattern 3: Checkpoint with Memory

Checkpoints automatically include memory references:

```bash
# Create checkpoint after storing output
memory log --file build.log --tag "build"
checkpoint now --notes "Build complete"

# Later, checkpoint report shows memory refs
checkpoint report
# [MEMORY]
#   3 entries, 12000 total tokens
```

### Pattern 4: Recovery Workflow

After context reset:

```bash
# 1. Load checkpoint
checkpoint load
# Shows: phase, deps, status summary, memory refs

# 2. Check directory status
checkpoint report
# Shows: active directories, completion %

# 3. Get memory summaries
memory list --limit 5
memory fetch mem-xxx -s  # Summary only

# 4. Fetch specific chunk if needed
memory fetch mem-xxx --chunk 2

# 5. Resume work
status update ./current-dir --task "Resuming: fix auth bug"
```

---

## Token Efficiency

### Before Memory Layer
```
Token Window:
├── System prompt: 2000 tokens
├── Conversation: 5000 tokens
├── Large output: 15000 tokens  ← Problem!
└── Total: 22000 tokens (overflow risk)
```

### With Memory Layer
```
Token Window:
├── System prompt: 2000 tokens
├── Conversation: 5000 tokens
├── Memory reference: 50 tokens  ← "mem-xxx: test results (15000 tokens)"
├── Summary: 200 tokens          ← "156 passed, 2 failed..."
└── Total: 7250 tokens (safe)

External Memory:
└── Full content: 15000 tokens (retrievable on demand)
```

### Token Budget Guidelines

| Content | Recommended Action |
|---------|-------------------|
| < 500 tokens | Include inline |
| 500-2000 tokens | Consider storing, include summary |
| > 2000 tokens | Always store, use summary + refs |
| > 10000 tokens | Store chunked, fetch chunks as needed |

---

## File Locations

```
/opt/agent-governance/
├── checkpoint/
│   ├── checkpoint.py          # Checkpoint manager
│   ├── storage/               # Checkpoint JSON files
│   └── templates/             # Summary templates
├── memory/
│   ├── memory.py              # Memory manager
│   ├── memory.db              # SQLite metadata
│   ├── chunks/                # Compressed content files
│   └── summaries/             # Generated summaries
├── bin/
│   ├── checkpoint             # CLI wrapper
│   ├── status                 # CLI wrapper
│   └── memory                 # CLI wrapper
├── docs/
│   ├── CONTEXT_MANAGEMENT.md  # This file
│   ├── MEMORY_LAYER.md        # Memory details
│   └── STATUS_PROTOCOL.md     # Status details
└── */
    ├── README.md              # Directory overview
    └── STATUS.md              # Directory status
```

---

## Best Practices

### For Checkpoints
1. Create checkpoints at natural stopping points
2. Include descriptive notes
3. Use `checkpoint report` to verify state
4. Review timeline before resuming work

### For STATUS Files
1. Update status when entering/leaving directories
2. Keep task lists current
3. Document blockers immediately
4. Link to relevant memory entries

### For Memory
1. Store proactively (before context fills)
2. Use meaningful tags for searchability
3. Fetch summaries first, full content only when needed
4. Prune regularly to manage storage

### For Recovery
1. Always start with `checkpoint load`
2. Use `checkpoint report` for overview
3. Check `status dashboard` for active work
4. Fetch memory summaries before full content

---

## Troubleshooting

### "Checkpoint not found"
```bash
checkpoint list  # See available checkpoints
```

### "Memory entry not found"
```bash
memory list --limit 50  # Browse recent entries
memory search "keyword"  # Search by content
```

### "STATUS.md missing"
```bash
status sweep --fix  # Auto-create missing files
```

### "Token window overflow"
```bash
# Store large content externally
memory log --stdin < large-output.txt
# Use summary instead of full content
memory fetch mem-xxx -s
```

---

*Part of the [Agent Governance System](/opt/agent-governance/docs/ARCHITECTURE.md)*