ac01fffd9a
Architectural snapshot of the lakehouse codebase at the point where the
full matrix-driven agent loop with Mem0 versioning + deletion was
validated end-to-end.
WHAT THIS REPO IS
A clean single-commit snapshot of the lakehouse code. Heavy test data
(.parquet datasets, vector indexes) excluded — see REPLICATION.md for
regen path. Full lakehouse history at git.agentview.dev/profit/lakehouse.
WHAT WAS PROVEN
- Vector retrieval across multi-corpora matrix (chicago_permits + entity
briefs + sec_tickers + distilled procedural + llm_team runs)
- Observer hand-review (cloud + heuristic fallback) gating each candidate
- Local-model agent loop (qwen3.5:latest) with tool use + scratchpad
- Playbook seal on success → next-iter retrieval surfaces it as preamble
- Mem0 versioning + deletion in pathway_memory:
* UPSERT: ADD on new workflow, UPDATE bumps replay_count on identical
* REVISE: chains versions, parent.superseded_at + superseded_by stamped
* RETIRE: marks specific trace retired with reason, excluded from retrieval
* HISTORY: walks chain root→tip, cycle-safe
KEY DIRECTORIES
- crates/vectord/src/pathway_memory.rs — Mem0 ops live here
- crates/vectord/src/playbook_memory.rs — original Mem0 reference
- tests/agent_test/ — local-model agent harness + PRD + session archives
- scripts/dump_raw_corpus.sh — MinIO bucket dump (raw test corpus)
- scripts/vectorize_raw_corpus.ts — corpus → vector indexes
- scripts/analyze_chicago_contracts.ts — real inference pipeline
- scripts/seal_agent_playbook.ts — Mem0 upsert from agent traces
Replication: see REPLICATION.md for Debian 13 clean install + cloud-only
adaptation (no local Ollama).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
38 lines
1.5 KiB
TOML
38 lines
1.5 KiB
TOML
[package]
|
|
name = "vectord-lance"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
# Firewall crate: the Lance stack (Arrow 57, DataFusion 52) is isolated
|
|
# from the rest of vectord (Arrow 55, DataFusion 47). Public API uses
|
|
# only std types — no Arrow types cross the crate boundary — so no
|
|
# version conflict propagates outward.
|
|
#
|
|
# See docs/ADR-019-vector-storage.md for the rationale: "productionizing
|
|
# will need either workspace-wide upgrade or a firewall via a dedicated
|
|
# vectord-lance crate." This is that firewall.
|
|
|
|
[dependencies]
|
|
# S3 support: Lance delegates to its internal object_store crate when
|
|
# given s3:// URIs. The "dynamodb" feature enables DynamoDB-based
|
|
# commit locking for multi-writer S3; we don't need that (single-writer)
|
|
# so just the base AWS/S3 feature is enough.
|
|
# Lance 4.0 feature "aws" enables S3-compatible storage via its internal
|
|
# object_store + opendal crates. Reads AWS_* env vars for credentials.
|
|
lance = { version = "4.0", default-features = false, features = ["aws"] }
|
|
lance-index = { version = "4.0", default-features = false }
|
|
lance-linalg = { version = "4.0", default-features = false }
|
|
|
|
# These Arrow/Parquet versions MUST match Lance's expectations — Lance
|
|
# re-exports their types through its API so any mismatch is a hard
|
|
# compile error. Keep in sync with lance-bench.
|
|
arrow = "57"
|
|
arrow-array = "57"
|
|
arrow-schema = "57"
|
|
parquet = "57"
|
|
|
|
tokio = { version = "1", features = ["full"] }
|
|
futures = "0.3"
|
|
serde = { version = "1", features = ["derive"] }
|
|
bytes = "1"
|