root 5d30b3da89 lance: auto-build doc_id btree in migrate handler (root-cause for 10M doc-fetch slowness) ...

scale_test_10m doc-fetch p50 was ~100ms — full table scan over 35GB. Root
cause: the auto-build at service.rs:1492-1503 only fires for IndexMeta-
registered indexes during set_active_profile warming. lance-bench writes
datasets through /vectors/lance/migrate/* directly, bypassing IndexMeta,
so its datasets never get the doc_id btree that ADR-019 depends on.

Fix: build the btree inline at the end of lance_migrate. Costs ~1.2s on
10M rows (+269MB on disk), drops doc-fetch from ~100ms to ~5ms (20x).
Failure is non-fatal — logs a warning and the dataset stays queryable.

Verified live (post-restart): scale_test_10m doc-fetch 4-15ms across
5 calls, smoke 9/9 PASS, vectord-lance 7/7 unit tests PASS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-02 21:38:00 -05:00

.archon/workflows

.archon: add lakehouse-architect-review workflow

2026-04-26 18:05:43 -05:00

auditor

auditor: layer-2 path-traversal guard — symlink resolution before read

2026-04-27 08:32:33 -05:00

bot

infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths

2026-04-28 06:13:30 -05:00

config

aibridge: drop Python sidecar from hot path; AiClient → direct Ollama

2026-05-02 04:59:47 -05:00

crates

lance: auto-build doc_id btree in migrate handler (root-cause for 10M doc-fetch slowness)

2026-05-02 21:38:00 -05:00

data

staffing: face pool fetch preserves prior tags + --shrink gate + atomic manifest write

2026-04-28 06:01:04 -05:00

docs

docs: pointer to ARCHITECTURE_COMPARISON.md source in golangLAKEHOUSE

2026-05-01 04:57:09 -05:00

inbox/processed

Scheduled ingest: file watcher auto-ingests from ./inbox

2026-03-27 20:04:40 -05:00

logs

Public URL: devop.live/lakehouse/proof — SSL, no IP needed

2026-04-17 12:41:53 -05:00

mcp-server

mcp-server: /_go/* pass-through for G5 cutover slice

2026-05-01 03:44:10 -05:00

ops/systemd

Scrum-driven fixes: P5-001 auth wired, P42-001 truth evaluator, P9-001 journal on ingest

2026-04-24 02:25:43 -05:00

proto

Phase 5: hardening — gRPC, observability, auth, config

2026-03-27 06:37:07 -05:00

reports

lance: auto-build doc_id btree in migrate handler (root-cause for 10M doc-fetch slowness)

2026-05-02 21:38:00 -05:00

scripts

lance backend: 4-pack — bug fix + smoke + tests + 10M re-bench

2026-05-02 20:06:56 -05:00

sidecar

qwen3.5 executor + continuation primitive + think:false

2026-04-20 20:19:02 -05:00

tests

infra: replace gpt-oss with Ollama Pro + OpenCode Zen across hot paths

2026-04-28 06:13:30 -05:00

truth

phase-42: truth/ repo-root dir + TOML rule loader

2026-04-24 13:44:23 -05:00

ui

pathway_memory: consensus-designed sidecar + hot-swap learning loop

2026-04-24 05:15:32 -05:00

.gitignore

Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs

2026-04-28 06:01:04 -05:00

.mcp.json

mcp: register gitea-mcp server — closes Phase 40 repo-ops gap

2026-04-24 13:19:46 -05:00

bun.lock

Phase 28-36 body of work

2026-04-22 02:41:15 -05:00

Cargo.lock

aibridge: LRU embed cache - 236x RPS gain on warm workloads. Per architecture_comparison.md universal-win for Rust side. Cache key (model,text), default 4096 entries, in-process inside gateway. Load test: 128 RPS -> 30k+ RPS, p50 78ms -> 129us.

2026-05-01 04:45:20 -05:00

Cargo.toml

phase-43: new crates/validator — trait, staffing impls, devops scaffold

2026-04-24 13:35:22 -05:00

justfile

Phase 4: Dioxus frontend with dataset browser and SQL query editor

2026-03-27 06:24:15 -05:00

lakehouse.toml

gateway: IterateResponse echoes trace_id + enable session_log_path

2026-05-02 06:24:41 -05:00

STATE_OF_PLAY.md

Synthetic face pool — 1000 StyleGAN headshots, ComfyUI hot-swap, 60x smaller thumbs

2026-04-28 06:01:04 -05:00

Description

Rust-first object storage system

6.3 GiB

Languages

TypeScript 38.4%

Rust 35.8%

HTML 13.9%

Python 7.8%

Shell 2.1%

Other 2%