7-step autonomous test via cron (every 2 minutes):
1. Register 10M × 768d Parquet (28.8 GB, already generated)
2. Migrate Parquet → Lance (proves Lance handles what HNSW can't)
3. Build IVF_PQ (3162 partitions for √10M, 192 sub_vectors)
4. Search benchmark (10 searches, measure p50/p95)
5. Hot-swap profile test (create scale-10m profile, activate)
6. Agent test (5 contract matches on 500K via gateway, autonomous)
7. Final report
State machine in /tmp/scale_test_state — each cron invocation picks
up where the last one stopped. Lock file prevents concurrent runs.
All output to /home/profit/lakehouse/logs/scale_test.log.
Monitor: tail -f /home/profit/lakehouse/logs/scale_test.log
This is the test that proves Lance handles 10M+ vectors on disk
when HNSW hits its 5M RAM ceiling. No human intervention needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
