cdc24d8bd0
Some checks failed
lakehouse/auditor 1 blocking issue: todo!() macro call in tests/real-world/scrum_master_pipeline.ts
The `aibridge::context::estimate_tokens` deprecation has been pointing
at `shared::model_matrix::ModelMatrix::estimate_tokens` for a while,
but that module didn't exist — so the deprecation was aspirational
noise, not actionable guidance.
Built the minimal target: `shared::model_matrix::ModelMatrix` with
an associated `estimate_tokens(text: &str) -> usize` method. Same
chars/4 ceiling heuristic as the deprecated helper. 6 tests cover
empty/3/4/5-char cases, multi-byte UTF-8 (emoji count as 1 char each),
and linear scaling to 400-char inputs.
Migrated 5 call sites:
- aibridge/context.rs:88 — opts.system token count
- aibridge/context.rs:89 — prompt token count
- aibridge/tree_split.rs:22 — import (now uses ModelMatrix)
- aibridge/tree_split.rs:84, 89 — truncate_scratchpad budget loop
- aibridge/tree_split.rs:282 — scratchpad post-truncation assertion
- aibridge/context.rs:183 — system-prompt budget test
Also cleaned up two parallel test warnings:
- aibridge/context.rs legacy estimate_tokens_ceiling_divides_by_four
test deleted (ModelMatrix's tests cover the same behavior now).
- vectord/playbook_memory.rs:1650 unused_mut on e_alive.
Net workspace warning count: 11 → 0 (including --tests build).
The deprecated `estimate_tokens` wrapper stays in aibridge/context.rs
for external callers. Future commits can remove it entirely once no
public API surface still references it.
The applier's warning-count gate now has a floor of 0 — any future
patch that introduces a single warning trips the gate automatically.
Previously a floor of 11 tolerated noise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
70 lines
2.5 KiB
Rust
70 lines
2.5 KiB
Rust
//! Per-model token accounting. Entry point for the ModelMatrix work
|
|
//! the aibridge `context::estimate_tokens` deprecation has been pointing
|
|
//! at. Starts minimal — just `estimate_tokens` — so call sites can
|
|
//! migrate off the deprecated helper. Extend with per-model context
|
|
//! windows, max_tokens defaults, provider hints, etc. as we move the
|
|
//! rest of `aibridge::context::known_windows` over.
|
|
|
|
/// Namespace for per-model token + context accounting. Methods are
|
|
/// associated functions — no instance required — because the underlying
|
|
/// estimates are deterministic and stateless.
|
|
pub struct ModelMatrix;
|
|
|
|
impl ModelMatrix {
|
|
/// Rough token count — char count divided by 4, rounded up. This
|
|
/// is the same heuristic OpenAI's cookbook uses for English text;
|
|
/// it's within ±15% of BPE tokenizers for code + prose and doesn't
|
|
/// require a tokenizer lookup. Good enough for budget math where
|
|
/// the goal is "don't blow the context window" rather than exact
|
|
/// billing.
|
|
///
|
|
/// Moved from `aibridge::context::estimate_tokens` (still there with
|
|
/// a `#[deprecated]` pointer — callers should migrate here). Empty
|
|
/// string → 0; one char → 1 (ceiling of 1/4 = 1).
|
|
pub fn estimate_tokens(text: &str) -> usize {
|
|
(text.chars().count() + 3) / 4
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::ModelMatrix;
|
|
|
|
#[test]
|
|
fn empty_string_is_zero_tokens() {
|
|
assert_eq!(ModelMatrix::estimate_tokens(""), 0);
|
|
}
|
|
|
|
#[test]
|
|
fn three_chars_is_one_token() {
|
|
// 3 → ceil(3/4) = 1. Matches the deprecated helper's behavior
|
|
// so the migration is a drop-in replacement.
|
|
assert_eq!(ModelMatrix::estimate_tokens("abc"), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn four_chars_is_one_token() {
|
|
assert_eq!(ModelMatrix::estimate_tokens("abcd"), 1);
|
|
}
|
|
|
|
#[test]
|
|
fn five_chars_is_two_tokens() {
|
|
assert_eq!(ModelMatrix::estimate_tokens("abcde"), 2);
|
|
}
|
|
|
|
#[test]
|
|
fn counts_chars_not_bytes() {
|
|
// Multi-byte UTF-8 chars count as 1 char each — important for
|
|
// prompts with emoji or non-ASCII text. "héllo" is 5 chars
|
|
// (5 unicode scalars) → ceil(5/4) = 2 tokens, same as "hello".
|
|
assert_eq!(ModelMatrix::estimate_tokens("héllo"), 2);
|
|
assert_eq!(ModelMatrix::estimate_tokens("📚📚📚📚"), 1); // 4 chars
|
|
}
|
|
|
|
#[test]
|
|
fn large_text_scales_linearly() {
|
|
assert_eq!(ModelMatrix::estimate_tokens(&"x".repeat(400)), 100);
|
|
assert_eq!(ModelMatrix::estimate_tokens(&"x".repeat(401)), 101);
|
|
}
|
|
}
|