dbe00d018f
Four shipped features and a PRD realignment, all measured end-to-end:
HNSW trial system (Phase 15 horizon item → complete)
- vectord: EmbeddingCache, harness (eval sets + brute-force ground truth),
TrialJournal, parameterized HnswConfig on build_index_with_config
- /vectors/hnsw/trial, /hnsw/trials/{idx}, /hnsw/trials/{idx}/best,
/hnsw/evals/{name}/autogen, /hnsw/cache/stats
- Measured on resumes_100k_v2 (100K × 768d): brute-force 44ms -> HNSW 873us
at 100% recall@10. ec=80 es=30 locked as HnswConfig::default()
- Lower ec values trade recall for build time: 20/30 = 0.96 recall in 8s,
80/30 = 1.00 recall in 230s
Catalog manifest repair
- catalogd: resync_from_parquet reads parquet footers to restore row_count
and columns on drifted manifests
- POST /catalog/datasets/{name}/resync + POST /catalog/resync-missing
- All 7 staffing tables recovered to PRD-matching 2,469,278 rows
Federation foundation (ADR-017)
- shared::secrets: SecretsProvider trait + FileSecretsProvider (reads
/etc/lakehouse/secrets.toml, enforces 0600 perms)
- storaged::registry::BucketRegistry — multi-bucket resolution with
rescue_bucket read fallback and reachability probing
- storaged::error_journal — bucket op failures visible in one HTTP call
- storaged::append_log — write-once batched append pattern (fixes the RMW
anti-pattern llms3.com calls out; errors and trial journals both use it)
- /storage/buckets, /storage/errors, /storage/bucket-health,
/storage/errors/{flush,compact}
- Bucket-aware I/O at /storage/buckets/{bucket}/objects/{*key} with
X-Lakehouse-Rescue-Used observability headers on fallback
Postgres streaming ingest
- ingestd::pg_stream: DSN parser, batched ORDER BY + LIMIT/OFFSET pagination
into ArrowWriter, lineage redacts password
- POST /ingest/db — verified against live knowledge_base.team_runs
(586 rows × 13 cols, 6 batches, 196ms end-to-end)
PRD realignment (2026-04-16)
- Dual use case: staffing analytics + local LLM knowledge substrate
- Removed "multi-tenancy (single-owner system)" from non-goals
- Added invariants 8-11: indexes hot-swappable, per-reader profiles,
trials-as-data, operational failures findable in one HTTP call
- New phases 16 (hot-swap generations), 17 (model profiles + dataset
bindings), 18 (Lance vs Parquet+sidecar evaluation)
- Known ceilings table documents the 5M vector wall and escape hatches
- ADR-017 (federation), ADR-018 (append-log pattern) added
- EXECUTION_PLAN.md sequences phases B-E with success gates and
decision rules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
49 lines
1.3 KiB
TOML
49 lines
1.3 KiB
TOML
[workspace]
|
|
resolver = "2"
|
|
members = [
|
|
"crates/shared",
|
|
"crates/proto",
|
|
"crates/storaged",
|
|
"crates/catalogd",
|
|
"crates/queryd",
|
|
"crates/aibridge",
|
|
"crates/ingestd",
|
|
"crates/vectord",
|
|
"crates/journald",
|
|
"crates/gateway",
|
|
"crates/ui",
|
|
]
|
|
|
|
[workspace.dependencies]
|
|
tokio = { version = "1", features = ["full"] }
|
|
axum = "0.8"
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
tracing = "0.1"
|
|
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
|
|
thiserror = "2"
|
|
uuid = { version = "1", features = ["v4", "serde"] }
|
|
chrono = { version = "0.4", features = ["serde"] }
|
|
tower-http = { version = "0.6", features = ["cors", "trace"] }
|
|
object_store = { version = "0.12", features = ["aws"] }
|
|
arrow = "55"
|
|
parquet = { version = "55", features = ["arrow", "async"] }
|
|
datafusion = "47"
|
|
bytes = "1"
|
|
futures = "0.3"
|
|
sha2 = "0.10"
|
|
url = "2"
|
|
tonic = "0.13"
|
|
prost = "0.13"
|
|
tonic-build = "0.13"
|
|
opentelemetry = "0.28"
|
|
opentelemetry_sdk = { version = "0.28", features = ["rt-tokio"] }
|
|
opentelemetry-stdout = { version = "0.28", features = ["trace"] }
|
|
tracing-opentelemetry = "0.29"
|
|
toml = "0.8"
|
|
csv = "1"
|
|
lopdf = "0.35"
|
|
encoding_rs = "0.8"
|
|
instant-distance = "0.6"
|
|
tokio-postgres = { version = "0.7", features = ["with-serde_json-1", "with-chrono-0_4", "with-uuid-1"] }
|