ab1f12296d
Replaces the single-source quote feed with a 3-source consensus system,
the differentiator for the index: pull each ticker from independent
feeds, take the median, and score confidence by how many corroborate it.
Disagreement becomes a signal instead of a silently-wrong number.
Context: Stooq retired its free q/l/ CSV quote API (404s server-side;
q/d/l behind a JS challenge), which broke the profiler basket. The
interim Yahoo-only fix worked but had a hidden wrong-baseline bug —
Yahoo's chartPreviousClose over a 5d window references the pre-window
close, so day-change read +11.65% for CAT when the real figure is
+1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class
of quiet error, which a single source can never self-report.
- quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA;
Yahoo prev-close now derived from the daily candle, not the buggy
meta field), median consensus, confidence = corroborating sources / 3,
outlier flagging, and a bun:sqlite hourly time-series cache.
- quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} —
hourly snapshot into the cache (real-time not required per product
call). Runs as root to match the mcp-server that co-owns the db;
busy_timeout serializes the two writers.
- /intelligence/ticker_quotes — serves cache-first (fast), aggregates
cold tickers on-demand once and caches them. New /quote_ingest admin
trigger.
- profiler.html — per-card confidence badge (green/amber/red) + a
per-source breakdown tooltip; "3-source consensus" labeling.
- entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief
(kept fetchStooqQuote dead-but-reversible).
Verified live on devop.live/lakehouse/profiler: liquid names read
100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only)
honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly
timer armed. Cache db (data/_quote_cache/) is gitignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
32 lines
1.2 KiB
TypeScript
32 lines
1.2 KiB
TypeScript
// Hourly quote ingest — CLI entry for the systemd timer (and manual runs).
|
|
//
|
|
// bun run quote_ingest.ts AAPL MSFT CAT # ingest a specific list
|
|
// bun run quote_ingest.ts # refresh every tracked ticker
|
|
//
|
|
// Writes multi-source consensus snapshots into the local SQLite cache
|
|
// (data/_quote_cache/quotes.db). See quotes.ts for the aggregation logic.
|
|
import { ingestQuotes, trackedTickers } from "./quotes.ts";
|
|
|
|
const args = process.argv.slice(2).map((s) => s.toUpperCase());
|
|
const t0 = Date.now();
|
|
const tracked = trackedTickers();
|
|
const target = args.length ? args : tracked;
|
|
|
|
if (!target.length) {
|
|
console.log(
|
|
"[quote_ingest] nothing to ingest — cache is empty and no tickers passed.\n" +
|
|
" Seed it by passing symbols (e.g. `bun run quote_ingest.ts AAPL CAT HD`),\n" +
|
|
" or just hit the profiler once so the serving path warms the cache.",
|
|
);
|
|
process.exit(0);
|
|
}
|
|
|
|
console.log(
|
|
`[quote_ingest] ${args.length ? "explicit list" : "refreshing tracked universe"}: ${target.length} tickers`,
|
|
);
|
|
const r = await ingestQuotes(target);
|
|
console.log(
|
|
`[quote_ingest] done in ${Date.now() - t0}ms — ${r.written}/${r.requested} written` +
|
|
(r.failed.length ? ` · no source answered for: ${r.failed.join(", ")}` : ""),
|
|
);
|