Refresh queue

Loading…

Each sweep (DO alarm every N ms) lists traders whose last refresh start is older than TTL and kicks off refreshes. Fan-out is how each trader refresh is started: SELF binding stays in the sweeper colo; Public HTTP goes through the front door like dashboard “Run scrape”; Auto tries both, then keeps the healthier path (success rate, then speed) and re-checks. Max in-flight is the concurrent scrape cap. Starts/sweep limits how many new refreshes one sweep may start. Pick is which stale traders those starts go to: Oldest first (the original queue) starts the most overdue; Newest first starts the least overdue among those already stale.

• 0 — fill available slots up to max in-flight (start all stale traders that fit; fastest catch-up)
• 1 — one new refresh per sweep (gentle on scrape worker; stale queue builds up)
• 2, 3, 5… — that many new refreshes per sweep at most (e.g. 3 → up to 3 new starts each sweep)

Sweep interval is the gap after each sweep finishes (not while it runs). With starts/sweep 0, a sweep should finish in ms and the DO alarm fires about every interval.

TTL: target gap between scrape starts (Δ prev). Scrapes often take ~0.6–1.0s; cache age stays a bit under TTL. 1s is usually too tight (scrape can exceed TTL). 2s is a good balance; 3s is stable. See notes/ttl-cadence.md.

Email when fleet scrape success over the last N minutes is below the threshold. Checked every cron minute. Sends once when it drops, then at most every 15 minutes until it recovers.

Loading…

Loading…