ab1f12296d
Replaces the single-source quote feed with a 3-source consensus system,
the differentiator for the index: pull each ticker from independent
feeds, take the median, and score confidence by how many corroborate it.
Disagreement becomes a signal instead of a silently-wrong number.
Context: Stooq retired its free q/l/ CSV quote API (404s server-side;
q/d/l behind a JS challenge), which broke the profiler basket. The
interim Yahoo-only fix worked but had a hidden wrong-baseline bug —
Yahoo's chartPreviousClose over a 5d window references the pre-window
close, so day-change read +11.65% for CAT when the real figure is
+1.11%. Cross-checking against Nasdaq + CNBC exposes exactly that class
of quiet error, which a single source can never self-report.
- quotes.ts — Yahoo + Nasdaq + CNBC fetchers (all keyless, browser UA;
Yahoo prev-close now derived from the daily candle, not the buggy
meta field), median consensus, confidence = corroborating sources / 3,
outlier flagging, and a bun:sqlite hourly time-series cache.
- quote_ingest.ts + ops/systemd/lakehouse-quote-ingest.{service,timer} —
hourly snapshot into the cache (real-time not required per product
call). Runs as root to match the mcp-server that co-owns the db;
busy_timeout serializes the two writers.
- /intelligence/ticker_quotes — serves cache-first (fast), aggregates
cold tickers on-demand once and caches them. New /quote_ingest admin
trigger.
- profiler.html — per-card confidence badge (green/amber/red) + a
per-source breakdown tooltip; "3-source consensus" labeling.
- entity.ts — the earlier Stooq->Yahoo swap for the single-ticker brief
(kept fetchStooqQuote dead-but-reversible).
Verified live on devop.live/lakehouse/profiler: liquid names read
100% / 3-source consensus; thin microcaps (BLPG, ESHSF — Yahoo-only)
honestly read 33% / 1-source. Full basket (17 tickers) seeded; hourly
timer armed. Cache db (data/_quote_cache/) is gitignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
92 lines
2.8 KiB
Bash
Executable File
92 lines
2.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Install the lakehouse-auditor + lakehouse-context7-bridge systemd units.
|
|
# Idempotent: re-running just reloads + restarts.
|
|
#
|
|
# Usage (as root):
|
|
# bash ops/systemd/install.sh
|
|
#
|
|
# What it does:
|
|
# 1. Copies *.service to /etc/systemd/system/
|
|
# 2. systemctl daemon-reload
|
|
# 3. systemctl enable --now both services
|
|
# 4. Prints post-install status
|
|
|
|
set -euo pipefail
|
|
|
|
UNIT_DIR="$(dirname "$(readlink -f "$0")")"
|
|
TARGET_DIR=/etc/systemd/system
|
|
|
|
UNITS=(
|
|
lakehouse-auditor.service
|
|
lakehouse-context7-bridge.service
|
|
lakehouse-retention-sweep.service
|
|
lakehouse-retention-sweep.timer
|
|
lakehouse-quote-ingest.service
|
|
lakehouse-quote-ingest.timer
|
|
)
|
|
|
|
if [[ $EUID -ne 0 ]]; then
|
|
echo "install.sh: must run as root (writes to $TARGET_DIR)" >&2
|
|
exit 1
|
|
fi
|
|
|
|
for unit in "${UNITS[@]}"; do
|
|
src="$UNIT_DIR/$unit"
|
|
dst="$TARGET_DIR/$unit"
|
|
if [[ ! -f "$src" ]]; then
|
|
echo "install.sh: missing source $src" >&2
|
|
exit 1
|
|
fi
|
|
echo "→ copy $unit"
|
|
install -m 0644 "$src" "$dst"
|
|
done
|
|
|
|
echo "→ systemctl daemon-reload"
|
|
systemctl daemon-reload
|
|
|
|
for unit in "${UNITS[@]}"; do
|
|
# For .timer units: enable + start the timer (which fires its
|
|
# paired oneshot service on schedule). For long-running .service
|
|
# units that DON'T have a timer: enable + restart so changes
|
|
# land. For oneshot .service units that ARE driven by a timer,
|
|
# do NOT enable/start them directly — the timer pulls them in.
|
|
base="${unit%.*}"
|
|
case "$unit" in
|
|
*.timer)
|
|
echo "→ enable + (re)start $unit"
|
|
systemctl enable "$unit" >/dev/null
|
|
systemctl restart "$unit"
|
|
;;
|
|
*.service)
|
|
# Skip if a paired .timer exists in this install set.
|
|
paired_timer="${base}.timer"
|
|
paired_in_set=0
|
|
for u2 in "${UNITS[@]}"; do
|
|
[[ "$u2" == "$paired_timer" ]] && paired_in_set=1 && break
|
|
done
|
|
if [[ $paired_in_set -eq 1 ]]; then
|
|
echo "→ skip direct start of $unit (driven by $paired_timer)"
|
|
else
|
|
echo "→ enable + (re)start $unit"
|
|
systemctl enable "$unit" >/dev/null
|
|
systemctl restart "$unit"
|
|
fi
|
|
;;
|
|
esac
|
|
done
|
|
|
|
echo ""
|
|
echo "─── post-install status ───"
|
|
for unit in "${UNITS[@]}"; do
|
|
active=$(systemctl is-active "$unit" 2>/dev/null || true)
|
|
enabled=$(systemctl is-enabled "$unit" 2>/dev/null || true)
|
|
printf " %-44s active=%s enabled=%s\n" "$unit" "$active" "$enabled"
|
|
done
|
|
echo ""
|
|
echo "Live logs: journalctl -u lakehouse-auditor.service -f"
|
|
echo " journalctl -u lakehouse-retention-sweep.service -f"
|
|
echo "Pause: touch /home/profit/lakehouse/auditor.paused"
|
|
echo "Resume: rm /home/profit/lakehouse/auditor.paused"
|
|
echo "Sweep test: systemctl start lakehouse-retention-sweep.service # one-shot, completes immediately"
|
|
echo "Next sweep: systemctl list-timers lakehouse-retention-sweep.timer"
|