Polymarket Bot - Operations Guide

Version: 3.7 | Last Updated: May 31, 2026 | Status: ✅ PRODUCTION

Table of Contents

1. Quick Start
2. Infrastructure
3. Running Services
4. Data Collection
5. Backfill Status
6. Database
7. Dashboard
8. Monitoring
9. Troubleshooting

1. Quick Start

Server Access

# SSH to server
ssh root@[hidden]

# Or use DigitalOcean Console
# Droplets → polymarket-collector → Access → Launch Console

Check Status

# All running processes
ps aux | grep -E 'gamma_collector|backfill|uvicorn' | grep -v grep

# Check backfill progress
cat [hidden]/trades_backfill_state.json | python3 -m json.tool
cat [hidden]/price_backfill_state.json | python3 -m json.tool

# View logs
tail -f /var/log/polymarket/gamma_collector.log
tail -f /var/log/polymarket/price_backfill.log

Dashboard URL

http://[hidden]:8080

2. Infrastructure

DigitalOcean Resources

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    DIGITALOCEAN INFRASTRUCTURE                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  MANAGED POSTGRESQL + TIMESCALEDB                        │  │
│   │  Host: db-postgresql-nyc2-polymarket-do-user-*.m.db...   │  │
│   │  Port: 25060 | Database: defaultdb | SSL: Required       │  │
│   │  Upgraded: 2 vCPU / 4GB RAM (Jan 2026)                   │  │
│   └──────────────────────────────────────────────────────────┘  │
│                              │                                   │
│                              │ Private Network                   │
│                              │                                   │
│   ┌──────────────────────────────────────────────────────────┐  │
│   │  DROPLET: polymarket-collector ([hidden])         │  │
│   │                                                          │  │
│   │  Services (post-Dome-EOL May 10, 2026 — Findings #287-#289): │
│   │  • continuous_collector.py    (live; gamma + clob + data-api)│
│   │  • live_trades_collector.py   (live; data-api /trades poller)│
│   │  • websocket_collector.py     (live; WSS market channel)     │
│   │  • backfill_trades.py         (weekly; data-api /trades T1Δ) │
│   │  • polymarket-resolver.timer  (daily 06:00 ET; outcome res.) │
│   │  • polymarket-resolver-intraday.timer (30-min light pass,    │
│   │    06-10 + 16-18 ET — same-day actual_beat, #343)            │
│   │  • polymarket-data-backfill.timer (Sun 04:00 ET; 6-pass heal)│
│   │  • polymarket-equity-bars.timer (daily; Polygon equity bars) │
│   │  • polymarket-strategy-analysis.timer         (Sun 13:00 UTC)│
│   │  • polymarket-api-health.timer     (daily API health; #289.a)│
│   │  • polymarket-integrity-monitor (5-min cycle; freshness etc.) │
│   │  • uvicorn                    (dashboard on port 8080)       │
│   │                                                              │
│   │  Decommissioned services (Dome → Polymarket migration):       │
│   │  • polymarket-orderbook-backfill (Dome /orderbooks; #287)    │
│   │  • polymarket-prices-backfill    (Gamma /markets/{id}/prices,│
│   │                                   not in new docs; #287)     │
│   │  • polymarket-dome-prices-backfill (Dome /market-price; #287)│
│   └──────────────────────────────────────────────────────────┘  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Database Connection

# Connection string format (in .env)
DATABASE_URL=postgresql+pg8000://doadmin:[REDACTED]@HOST:25060/defaultdb

# Note: SSL is required but uses self-signed cert
# Code uses ssl.create_default_context() with verify_mode=CERT_NONE

Access and credential rules (#457): - Trusted sources: the cluster accepts customer connections only from the droplet: rule ip_addr:[hidden], DONE Sep 26, 2026 16:06 UTC (#457). DigitalOcean-internal sessions (pghoard, management-agent, monitoring) have no client address and are not affected. List: doctl databases firewalls list b28d3f40-480e-4931-a38d-68c7ef428546 (on the droplet). Rollback: doctl databases firewalls remove <cluster-id> --uuid <rule-uuid>. - Workstations and agents never connect directly. SSH to the droplet and run psql there. Prod DATABASE_URL uses the SQLAlchemy scheme postgresql+pg8000://, so strip +pg8000 for psql: U=$(grep '^DATABASE_URL=' .env | cut -d= -f2- | sed 's#+pg8000##'); PGSSLMODE=require psql "$U". Allowing a workstation IP is an operator decision. - Private hostname: if DATABASE_URL ever moves to the private-… host, add a VPC rule first (droplet:<id>, or the VPC address). The public-IP rule does not cover private-network traffic. - Where the password lives: prod .env only (0600, root:root). It never goes into tracked files, docs or getenv(..., default=), because the dashboard serves docs/* (#457/#467). tests/unit/test_finding_457_no_live_db_url.py enforces this by shape and with a sha256-prefix denylist of every value known to have leaked. - Manual script runs: export the whole .env, not single names, so every key a script reads is present: cd [hidden] && set -a && . ./.env && set +a && venv/bin/python3 scripts/<script>.py. Per-command recipes elsewhere in this file that export named variables stay valid for those commands. - Changing the password: every consumer reads .env at process start and the pools cache the URL, so restart every long-running DB client: earnings-trader, polymarket-dashboard, polymarket-collector, polymarket-live-trades, polymarket-websocket, polymarket-integrity-monitor, earnings-conviction-tracker. Timer and cron jobs read .env on each run. Never restart ibgateway-ibc for a DB change.

Python environment (Finding #473)

• Prod venv: [hidden]/venv, Python 3.12.3. requirements.txt pins every direct dependency to the version installed there, and constraints.txt is the venv's full pip freeze (captured read-only Sep 26, 2026).
• Rebuild or a fresh dev/test venv: pip install -r requirements.txt -c constraints.txt.
• The deploy (deploy/deploy_from_github.sh) never runs pip, so a deploy cannot change prod packages. A package change is its own operator-approved action on prod; the same commit refreshes the pin and constraints.txt from a new read-only venv/bin/pip freeze --all.

3. Running Services

Current Services (May 2026)

IBKR Gateway Services

CRITICAL (Feb 3, 2026): Only ONE IBC service must be active. Running multiple IBC services causes repeated 2FA requests every 5 minutes due to session conflicts.

IBC Configuration (Updated Feb 3, 2026): - Config file: /opt/ibc/config.ini - Java path: /opt/i4j_jres/Oda-jK0QgTEmVssfllLP/17.0.10.0.101-zulu/bin (bundled, includes JavaFX) - Logs: /opt/ibc/logs/ibc-*.txt - Auto restart: 06:00 AM UTC daily

2FA Schedule (Updated Jan 26, 2026): - Sunday 06:00 UTC (10:00 AM Abu Dhabi): 2FA required - approve on IBKR app - Mon-Sat 06:00 UTC (10:00 AM Abu Dhabi): IBC's in-process AutoRestart keeps the session - NO 2FA (#437). gateway_restart_manager.sh is retired (#476) - Overnight shutdown recovery: Smart wrapper delays restart to 09:00 AM Abu Dhabi

Systemd Services (Recommended)

# Earnings Trading Service
sudo systemctl start earnings-trader    # Start trading
sudo systemctl stop earnings-trader     # Stop trading (graceful)
sudo systemctl status earnings-trader   # Check status
sudo journalctl -u earnings-trader -f   # Live logs

# Dashboard
sudo systemctl start polymarket-dashboard
sudo systemctl status polymarket-dashboard

# IBKR Gateway (IBC-Managed)
sudo systemctl start ibgateway-ibc      # Start IBC (manages Gateway)
sudo systemctl status ibgateway-ibc     # Check status
sudo journalctl -u ibgateway-ibc -f     # Live logs

# IBKR Gateway Monitor
sudo systemctl start ibkr-monitor
sudo systemctl status ibkr-monitor

# Conviction Tracker (Hourly snapshots for ALL earnings)
sudo systemctl start earnings-conviction-tracker
sudo systemctl status earnings-conviction-tracker
tail -f /var/log/polymarket/conviction_tracker.log

Conviction Tracker Service (Added Jan 27, 2026)

Tracks Polymarket conviction for ALL earnings markets (not just traded ones).

Purpose: - Capture conviction at entry times (3:58 PM ET for BMO, 10:34 AM ET for AMC) - Record hourly snapshots for strategy refinement - Track 24h, 12h, 6h, 1h, 30min before earnings - Analyze Polymarket reliability at different time horizons

Data stored in: - earnings_markets.prob_24h_before, prob_12h_before, etc. - conviction_history table (hourly granularity with BMO/AMC labels)

Commands:

# Backfill historical data from price_snapshots
python3 src/data_tracker/collectors/conviction_tracker.py --backfill

# Single collection cycle
python3 src/data_tracker/collectors/conviction_tracker.py --collect

# Continuous tracking (default, run as service)
python3 src/data_tracker/collectors/conviction_tracker.py --run --interval 60

Trading Service Details

The earnings-trader service runs the fully automated trading bot:

Mode persistence hardening (Feb 2026): - Shared helper module: src/earnings/trading_mode_persistence.py - Used by trader service, dashboard API, and reconciliation code paths - Single default across components: testing_prod - Self-healing behavior: if [hidden]/trading_mode.txt is missing/invalid, helper writes testing_prod back to disk automatically - This prevents dashboard/trader mode drift after restarts

Trading Modes:

⚠️ Whole Shares Only (Finding #51, Jan 20, 2026): System uses whole shares only. IBKR's cashQty fractional feature is unreliable with extended hours trading. Position size flex (+57%) enables trading expensive stocks. See STRATEGY.md Section 4.1.2.

Change Trading Mode:

# Dashboard is DB/file read-only (Finding #467.A removed the unauthenticated
# POST /api/set-trading-mode — it was already inert, since get_orchestrator()
# always returns None). Set the mode directly:
echo "testing_prod" > [hidden]/trading_mode.txt
sudo systemctl restart earnings-trader

Verify mode consistency (trader + dashboard):

# 1) persisted file
cat [hidden]/trading_mode.txt

# 2) dashboard view
curl -s http://127.0.0.1:8080/api/state | python3 -m json.tool | rg '"mode"'

# 3) trader startup log must show same mode
tail -100 /var/log/polymarket/earnings_trader.log | rg "Mode:"

Manual Service Start (Legacy)

ssh root@[hidden]
cd [hidden] && source venv/bin/activate

# Start Gamma Collector (continuous)
nohup python3 -m src.data_tracker.collectors.gamma_collector --interval 60 \
  > /var/log/polymarket/gamma_collector.log 2>&1 &

# Start Trades Backfill
nohup python -u src/data_tracker/collectors/backfill_trades.py \
  > /var/log/polymarket/trades_backfill.log 2>&1 &

# Start Price Backfill
nohup python -u src/data_tracker/collectors/backfill_prices.py \
  > /var/log/polymarket/price_backfill.log 2>&1 &

# Start Dashboard (prefer systemd instead)
nohup uvicorn earnings_dashboard.app:app --host 0.0.0.0 --port 8080 \
  > /var/log/polymarket/dashboard.log 2>&1 &

Stop Services

# Systemd services
sudo systemctl stop earnings-trader
sudo systemctl stop polymarket-dashboard

# Legacy processes
ps aux | grep gamma_collector | grep -v grep
kill <PID>

# Or kill all Python processes (use carefully)
pkill -f "python.*gamma_collector"
pkill -f "python.*backfill"
pkill uvicorn

Cron Jobs

# View current crontab
crontab -l
# Compare with the managed file (Finding #476) — read-only
cd [hidden] && venv/bin/python3 deploy/install_crontab.py --check

# Managed root crontab = deploy/cron/root.crontab, installed by the deploy:
#   50 5 * * 0  /bin/bash scripts/sunday_2fa_reminder.sh   (the one Sunday reminder)
#   0 4 * * *   /bin/bash scripts/ibkr_schedule_audit.sh   (drift audit, log only)
# Edit the repo file and deploy; a hand-added host line makes the installer
# refuse (crontab unchanged) and cron:schedule_audit shows the drift.
# Current scheduled jobs:
# Daily API health check now runs via polymarket-api-health.timer
# (systemd timer, NOT cron — Finding #289.a)

Earnings Market Sync (Added Feb 13, 2026)

Service: polymarket-earnings-sync.timer (hourly at :05)

Syncs Polymarket earnings markets to earnings_markets DB table via Gamma API. Previously had no automated schedule -- markets went 16 days without sync (Jan 29 - Feb 13).

# Check timer status
systemctl status polymarket-earnings-sync.timer

# View sync logs
tail -50 /var/log/polymarket/earnings_sync.log

# Run manual sync (Gamma → earnings_markets; includes MCAP backfill on fetcher path)
cd [hidden] && source .env && export DATABASE_URL
python3 -m src.data_tracker.collectors.earnings_market_fetcher

# Finding #480: do NOT run scripts/sync_earnings_markets.py or
# python -m src.db.earnings_sync. Both are retired guards (exit 2) because the
# legacy writer could change earnings_date without earnings_date_audit.

# One-off: set BMO/AMC for a ticker when DB has NULL/TBD/empty only
venv/bin/python3 scripts/backfill_ticker_earnings_timing.py --ticker SYMBOL --timing BMO

# Check what markets are in DB for next week
psql $DATABASE_URL -c "SELECT ticker, earnings_date, earnings_time FROM earnings_markets WHERE earnings_date >= CURRENT_DATE ORDER BY earnings_date LIMIT 20"

# Market cap on upcoming rows (Finding #219): filled after each live Gamma sync
# by EarningsMarketFetcher -> EarningsMarketSync.backfill_upcoming_null_mcaps.
# Fallback chain in src/utils/caching.py:get_market_cap (Finding #249, Apr 26, 2026):
#   yfinance (rotated session, primary) → Polygon /v3/reference/tickers/{X} (2nd tier, ~98% hit rate, ~150ms/call) → Finnhub /stock/profile2 (last-resort tail).
# Requires POLYGON_KEY and FINNHUB_API_KEY in .env. Polygon ahead of Finnhub because (a) more reliable on yfinance-rate-limited days, (b) returns mcap + name + exchange + shares in one response.

# BMO/AMC on upcoming rows (Finding #220): the live fetcher resolves timing in
# Python; the one-ticker NULL/TBD backfill above is the manual path. Web/yfinance
# timing is **not** written to DB — only calendar + DEFAULT_TIMINGS/overrides
# (amended by #349: a ≥2-vote calendar consensus may also correct timing).
# Dashboard **✓ TRADE** requires known BMO/AMC; otherwise **NEEDS TIMING**.

# Earnings-date confidence (Finding #349, Jul 29 2026): the multi-source verifier
# (runs before each trader signal cycle) stamps earnings_markets.date_confidence:
#   confirmed — ≥2-vote consensus (DB/NASDAQ/Finnhub/yfinance); date+timing LOCKED
#               against all three Polymarket-derived sync writers.
#   disputed  — sources disagree, no consensus; row NOT tradeable + dashboard ⚠ DATE?
#               badge + integrity WARN. Resolve via official IR, then either wait for
#               the verifier to re-vote or fix manually:
#   psql $DATABASE_URL -c "UPDATE earnings_markets SET earnings_date='YYYY-MM-DD', earnings_time='AMC', date_confidence='confirmed', date_verified_at=NOW() WHERE ticker='SYM' AND earnings_date >= CURRENT_DATE"
#   (also INSERT an earnings_date_audit row with source='operator' for the churn detector)
# Change history: SELECT * FROM earnings_date_audit ORDER BY changed_at DESC LIMIT 20;
# Integrity: consistency:earnings_dates_disputed (WARN), consistency:earnings_date_churn
# (HIGH — ≥3 date changes in 7d = a writer is fighting the verifier; find it before it trades).
# Migration: src/data_tracker/migrations/029_earnings_date_confidence.sql — applied once, recorded
# in the schema_migrations ledger; deploys no longer re-run it (Finding #444).

Outcome Resolver — Learning Loop (Added Apr 21, 2026 — Finding #227)

Services: - polymarket-resolver.timer — daily full pass at 06:00 ET (200-row limits, 90-day lookback; completeness backstop). - polymarket-resolver-intraday.timer — NEW (Finding #343, Jul 15 2026): light pass every 30 min during release windows (06:15–10:45 + 16:15–18:45 ET at :15/:45; --earnings-limit 50 --skipped-limit 0 --opportunity-limit 0 --lookback-days 3 --include-today). Rationale: #342 made the resolver the SOLE actual_beat writer, and the daily 06:00 run fires BEFORE same-day BMO releases — outcomes would otherwise settle ~24h late (JNJ/CTAS Jul-15 incident: crowd pinned 0.9995 all morning, dashboard stayed blank). Same-day rows resolve via tier 2 (basis-matched EPS) or tier 3 (sustained post-release crowd, release-anchored window per #357/#372) — both inherently post-release; tier 1 lands whenever Polymarket resolves (p50 ≈ 2.9 h after close).

Single writer for the four post-resolution data feeds: 1. earnings_markets.actual_beat / eps_actual / revenue_actual / stock_price_* / stock_return_* - actual_beat 3-tier system (Finding #372, Aug 21 2026 — A-1; replaces the old EPS-surprise → PM-snapshot-0.95 → pre-event-cascade order): tier 1 = the market's OWN resolution (Gamma umaResolutionStatus='resolved' + degenerate outcomePrices, fetched closed=true) — terminal, write-once-LOCKED (actual_beat_source='pm_resolution_v1'; every other writer is a CASE-guard no-op on those rows; non-degenerate resolutions are never auto-written — integrity-flagged); tier 2 = the market's stated benchmark (benchmark_basis/benchmark_comparator parsed from the slug at discovery): basis-matched actuals ONLY (non-GAAP → Yahoo/Finnhub street; GAAP → Polygon SEC financials; never cross-basis), rounded to the cent, strict > (tie → NO), abstain on unparseable (benchmark_eps_v1); tier 3 = sustained post-release crowd verdict — the w357 method (release-anchored window, ≥3 prints over ≥30 min one-sided beyond 0.90/0.10) (crowd_verdict_v1, provisional, upgradeable by tiers 1–2). RETIRED as label sources: the pre-event conviction cascade (prob_30min…prob_24h) and the single-snapshot ≥0.95/≤0.05 read. - Current coverage: ~95-99.8% of past rows (Finding #321); tier-1 coverage 99.8% of resolved-market history (#372 census). - Retry guard (Finding #371, Aug 21 2026 — A-2): Pass 1 retries each unresolved row at most once per 6 hours, keyed on the resolver-OWNED earnings_markets.resolver_attempted_at (migration 033). Through Aug 21 it keyed on the SHARED updated_at, which the hourly polymarket-earnings-sync.timer touches — unresolved rows were re-locked every hour and coverage silently decayed to 76.0% (measured Aug 16, populations.md §1.2). --force still bypasses the guard for one-shot backfills, but is no longer needed for routine healing: NULL markers are immediately eligible, so backlog self-heals on normal timer fires. No other writer may touch resolver_attempted_at. 2. skipped_signals.actual_outcome / would_have_won / hypothetical_pnl[_pct] 3. polymarket_opportunity_log.hypothetical_* 4. trade_audit_log.prediction_correct (joined from earnings_markets.actual_beat; corrective since #372 — recomputed whenever the label changes, not fill-only)

# Status / logs
systemctl status polymarket-resolver.timer
journalctl -u polymarket-resolver.service -n 200 --no-pager
tail -100 /var/log/polymarket/outcome_resolver.log

# Manual incremental run (matches the daily timer)
cd [hidden] && source .env && export DATABASE_URL POLYGON_KEY FINNHUB_API_KEY
venv/bin/python3 -m src.data_tracker.collectors.outcome_resolver \
    --earnings-limit 200 --skipped-limit 200 --opportunity-limit 200 \
    --polygon-sleep 0.6 --lookback-days 90

# One-shot HISTORICAL backfill (no date floor; loops until exhausted)
venv/bin/python3 scripts/backfill_learning_loop.py --polygon-sleep 0.6

# Backfill regime / VIX on existing trade_audit_log rows (one time)
venv/bin/python3 scripts/backfill_regime_for_audit_log.py

# Acceptance queries (Finding #227)
# Targets after the daily resolver settles:
#   1. opp_log unresolved          → 0
#   2. skipped past-date unresolved → 0
#   3. actual_beat coverage         → ~95-99.8% (Finding #321)
#   4. new trades missing vix       → 0
psql $DATABASE_URL -c "SELECT count(*) FROM polymarket_opportunity_log WHERE counterfactual_computed_at IS NULL"
psql $DATABASE_URL -c "SELECT count(*) FROM skipped_signals WHERE earnings_date < CURRENT_DATE AND counterfactual_computed_at IS NULL"
psql $DATABASE_URL -c "SELECT count(*) past, count(*) FILTER (WHERE actual_beat IS NOT NULL) resolved FROM earnings_markets WHERE earnings_date < CURRENT_DATE"
psql $DATABASE_URL -c "SELECT count(*) FROM trade_audit_log WHERE entry_time > '2026-04-22' AND vix_at_entry IS NULL"

Polygon plan guidance (verified Apr 22, 2026 against polygon.io account): - Stocks Basic ($0/mo, 5 req/min, EOD only) → --polygon-sleep 12.5 (~14h for full historical pass) - Stocks Starter ($29/mo, UNLIMITED API calls, 15-min-delayed) → --polygon-sleep 0.05 for one-shot manual backfills is fine, but scheduled units use 0.25 since Finding #348 (Jul 29, 2026): "unlimited" is soft-throttled — two daily jobs colliding at 20 req/s each produced a 429 storm. The shared fetch helper retries 429/5xx with 2s/8s/30s backoff. - Stocks Advanced ($199/mo) → same speed as Starter + real-time data + Trades/Quotes. Only needed if you want sub-15-min-delayed live pricing (the resolver doesn't need this; it works on historical aggs).

Set POLYGON_KEY in [hidden]/.env (template line in .env.example). Without it, the resolver still runs but marks counterfactual rows no_polygon_v1 instead of computing P&L.

Guard overrides (Finding #356, Aug 9, 2026): to temporarily disable or weaken a live defense (session stop breaker, thresholds, watchdog), the .env line MUST carry a dated expiry comment — same line or the line above:

# GUARD-OVERRIDE-UNTIL: 2026-08-05
SESSION_STOP_BREAKER_ENABLED=false

Integrity rule consistency:guard_overrides (HIGH) stays green while the date is in the future (the active override is listed in the rule detail) and FAILS the moment it passes — or immediately for an undated override. Expiry is inclusive through end-of-day UTC. Registered guards live in GUARD_REGISTRY (src/data_tracker/integrity_monitor.py); add new guard knobs there when introducing them. Born from the #355 incident: a same-night restore plan died with an interrupted agent session and the breaker silently stayed off for two sessions.

Cross-process rate limiter + plan-downgrade alarm (Finding #350, Jul 29, 2026): The subscription silently dropped to the free tier (5 req/min) around Jul-28 — five services degraded over two days and #348's collision diagnosis turned out to be the wrong trigger. Defenses now in place: - All scheduled/live Polygon consumers share one token bucket via src/utils/polygon_rate_limiter.py (flock'd state file). Env knobs: POLYGON_MAX_RPS (default 10 combined), POLYGON_BURST (default 10), POLYGON_BUCKET_PATH (default /var/tmp/polymarket_polygon_bucket.json). Fail-open by design — a limiter problem can never fail a fetch. - Plan-limit 429 bodies ("upgrade your subscription") write logs/polygon_quota_downgrade.json; integrity rule consistency:polygon_plan_downgrade (HIGH) alarms on it by name. The flag self-clears on the next successful Polygon request. If it fires: check the Polygon/Massive billing account — code cannot fix a lapsed plan. - etfstack-signal retries now span (15/30/65s) so they escape an exhausted per-minute window, and its live run may fall back to yfinance for ETF dailies (WARNING-logged, whole-series, no splicing).

Cross-process Finnhub limiter (Finding #407, Aug 22, 2026): Free tier is 60 req/min. Live callers (caching.py profile/quote, earnings calendar verifier, outcome_resolver /stock/earnings) share one token bucket via src/utils/finnhub_rate_limiter.py. Env knobs: FINNHUB_MAX_RPM (default 40 combined — headroom under 60), FINNHUB_BURST (default 10), FINNHUB_BUCKET_PATH (default /var/tmp/polymarket_finnhub_bucket.json). Fail-open — a limiter problem can never fail a fetch. Live yfinance goes through yfinance_limiter; PRNewswire financial RSS is fetched only by _get_prnewswire_rss (30s cache, #395).

Polymarket data backfill timer (Added Apr 25, 2026 — Finding #231)

Service: polymarket-data-backfill.timer (weekly Sun 04:00 ET; 6 ExecStart passes — see below).

Self-heals two data-pipeline gaps documented in Findings #229 and #230:

1. markets.volume_total for newly-resolved earnings markets that the ?active=true Gamma sync excluded.
2. earnings_markets.prob_*_before for newly-resolved markets whose orderbook samples didn't cover the 24h/12h/6h-before windows.

Six sequential ExecStart passes per run (all idempotent):

# Status / next fire
systemctl status polymarket-data-backfill.timer
systemctl list-timers polymarket-data-backfill.timer

# Logs
tail -200 /var/log/polymarket/data_backfill.log
journalctl -u polymarket-data-backfill.service -n 200 --no-pager

# Manual one-shot (e.g. after fixing a data issue)
systemctl start polymarket-data-backfill.service

# Or run individual passes manually
cd [hidden] && source .env && export DATABASE_URL
venv/bin/python3 scripts/backfill_markets_volume.py --apply --age-days 60
venv/bin/python3 src/data_tracker/collectors/conviction_tracker.py --backfill
venv/bin/python3 scripts/backfill_conviction_from_trades.py --apply --age-days 60

The 60-day age window matches the cadence of new earnings resolutions; the historical full-window backfill is a one-shot operator action (drop the --age-days arg or extend it; see the per-finding docs for the full historical pass procedure).

Additional passes were added in Phase 1 of Strategy Analysis Revamp (Finding #238) and later phases, bringing the unit to 6 ExecStart passes total per run. One of them keeps polymarket_opportunity_log.hypothetical_pnl populated weekly:

venv/bin/python3 scripts/compute_counterfactual_pnl.py --limit 500 --sleep-seconds 0.05

Polymarket trades-backfill (Added May 9, 2026 — Finding #269 smart Option A; migrated to Polymarket data-api May 10, 2026 — Finding #288)

Service: polymarket-trades-backfill.service (Type=oneshot, ~330-line Python script). Timer: polymarket-trades-backfill.timer (weekly Sun 06:00 ET, Persistent=true, RandomizedDelaySec=600).

Populates the trades TimescaleDB hypertable from the Polymarket data-api /trades?market=<cid> endpoint for each market in our DB. (Originally hit Dome /polymarket/orders until Apr 28, 2026 EOL; migrated to data-api in Finding #288 with no logic change beyond endpoint + field-shape update.) Unlike the live polymarket-collector.service (which streams real-time trades), this batch job fills in historical pre-collector trade history + catches anything live collector missed during bursts. The binding constraint is the Polymarket data-api free-tier rate limit (200 req/10s = 20 q/s sustained for /trades).

Smart Option A targeting: - T1 — Delta backfill via client-side cutoff. Per-market MAX(time) lookup (with 1-hour overlap); paginate data-api /trades newest-first via offset; stop when crossing the cutoff. data-api has no server-side start_time param so the filter is client-side; trade_dedupe handles overlap-window duplicates idempotently. - T2 — Smart re-fetch policy. Active markets and recently-resolved (earnings_date within 7 days) markets get re-fetched even if previously "completed". Old closed markets are stable-skipped. - T3 — Earnings-only scope by default. INNER JOIN earnings_markets reduces market count from ~5,735 → ~929 (6× reduction).

CLI flags:

# default — earnings-only scope, delta when state exists, full when cold
venv/bin/python3 src/data_tracker/collectors/backfill_trades.py

# scope override — sweep the entire markets table (~5,735 markets)
venv/bin/python3 src/data_tracker/collectors/backfill_trades.py --all-markets

# ignore completed_markets state — re-fetch everything from scratch
venv/bin/python3 src/data_tracker/collectors/backfill_trades.py --full-resync

# count markets + estimate API budget WITHOUT making calls or writing
venv/bin/python3 src/data_tracker/collectors/backfill_trades.py --dry-run

Free-tier Polymarket data-api budget (200 req/10s for /trades = 20 q/s sustained, 0.05s gap; 20× faster than Dome was): - Cold bootstrap (--full-resync, 929 earnings markets): ~9k API calls / ~8 minutes wall time (was ~3 hours on Dome) - Warm weekly delta (default): ~900 API calls / ~1-2 minutes wall time (was ~18 minutes on Dome)

Cold-bootstrap → enable-timer policy (MUST follow this order):

deploy/deploy_from_github.sh INSTALLS the timer file but does NOT enable it. The first run should be a manual cold bootstrap so the weekly timer fires warm thereafter (~1-2 min) instead of cold (~8 min). On the post-#288 free-tier rate budget the cold bootstrap is much cheaper than it was on Dome — but the principle of "verify before automating" still applies.

# 1. After deploy completes, verify the timer is installed but NOT enabled.
systemctl list-timers --all polymarket-trades-backfill.timer
# Expected: LOAD=loaded, ACTIVE=inactive, NEXT=- (timer not yet enabled)

# 2. Sanity-check the budget estimate matches expectations.
cd [hidden] && source .env && export DATABASE_URL
venv/bin/python3 src/data_tracker/collectors/backfill_trades.py --dry-run

# 3. Trigger the cold bootstrap. Type=oneshot service runs in the background
#    and writes to /var/log/polymarket/trades_backfill.log + journal.
systemctl start polymarket-trades-backfill.service

# 4. Monitor. Expect ~8 minutes total wall time on Polymarket free tier
#    (was ~3 hours on Dome free tier — Finding #288 migration is 20× faster).
tail -f /var/log/polymarket/trades_backfill.log
journalctl -u polymarket-trades-backfill.service -f --no-pager
# State file populates incrementally — survives interruption + resumes.
cat [hidden]/trades_backfill_state.json | jq '.completed_markets | length'

# 5. After cold bootstrap COMPLETES (script logs "BACKFILL COMPLETE!"),
#    enable the weekly timer. The next Sunday 06:00 ET fire will be warm.
systemctl enable --now polymarket-trades-backfill.timer

# 6. Verify the timer is now armed.
systemctl list-timers polymarket-trades-backfill.timer
# Expected: LOAD=loaded, ACTIVE=active (waiting), NEXT=Sun YYYY-MM-DD 06:00 ET

Operational checks:

# Timer status / next fire
systemctl status polymarket-trades-backfill.timer
systemctl list-timers polymarket-trades-backfill.timer

# Service status (most recent run)
systemctl status polymarket-trades-backfill.service

# Logs
tail -200 /var/log/polymarket/trades_backfill.log
journalctl -u polymarket-trades-backfill.service -n 200 --no-pager

# State file — see what's been completed
jq '. | {completed: (.completed_markets | length), trades_saved, current_market}' \
  [hidden]/trades_backfill_state.json

Recovery procedures: - Run interrupted mid-market: state file's current_offset resumes from the last saved position. Just re-run systemctl start polymarket-trades-backfill.service. - Polymarket rate-limit storm (sustained 429s): the script's _fetch_trades_page already does exponential backoff with 30s cap + 4 retries. If problems persist, check docs.polymarket.com/api-reference/rate-limits.md or run --dry-run to verify connectivity. The 5-consecutive-failure circuit breaker (Finding #269.v3) trips and exits non-zero, surfacing as STATUS_FAIL on integrity-monitor's systemd_last_run:polymarket-trades-backfill.service rule (Telegram alert). - Want to force a full re-fetch: systemctl stop polymarket-trades-backfill.service (if running), then venv/bin/python3 src/data_tracker/collectors/backfill_trades.py --full-resync. Wipes completed_markets from state. - Newly-discovered data gap on specific markets: delete those markets' condition_id entries from state['completed_markets'] (jq + atomic write), then re-run normally — the smart re-fetch policy will pull deltas.

Integrity-monitor coverage: systemd_timer:polymarket-trades-backfill.timer rule with 8-day max-age (matches polymarket-data-backfill.timer). Returns STATUS_UNKNOWN until enabled (no false alarms during install-but-not-enable window).

Re-evaluation trigger: Phase 3 unified simulator PR 1 shipped May 28, 2026 (Findings #306.a + #306.b + #306.c + #307 + #308). With PR 1 complete, the trades-backfill cadence + scope can be revisited in PR 2 design review (volume-gate retune may need higher-density trade backfill). See Finding #269 in docs/FINDINGS.md for canonical narrative.

Polymarket equity 1m bar collector (Added Apr 26, 2026 — Finding #240)

Service: polymarket-equity-bars.timer (daily at 11:35 UTC ≈ 06:35-07:35 ET, before market open — moved from 10:00 UTC by Finding #348: the old slot collided with the daily resolver at 06:00 ET = 10:00 UTC EDT, both hammering the shared Polygon key → Jul-29 429 storm, 30/35 jobs failed. Pacing also 0.05 → 0.25s and the shared fetch helper now retries 429/5xx with backoff).

Populates equity_bars_1m TimescaleDB hypertable with Polygon 1-minute aggregates for every symbol with a Polymarket earnings market in the trade-eligibility band. Phase 2 of Strategy Analysis Revamp; load-bearing infrastructure for Phase 3 unified simulator (per docs/STRATEGY_ANALYSIS_ARCHITECTURE.md §0.6 + §0.10).

# Status / next fire
systemctl status polymarket-equity-bars.timer
systemctl list-timers polymarket-equity-bars.timer

# Logs
tail -200 /var/log/polymarket/equity_bars_collector.log
journalctl -u polymarket-equity-bars.service -n 200 --no-pager

# Manual one-shot (e.g. after a hypertable repair or to backfill)
systemctl start polymarket-equity-bars.service

# Or run individual modes manually:
cd [hidden] && source .env && export DATABASE_URL POLYGON_KEY PYTHONPATH=[hidden]

# Default: 7-day upcoming window
venv/bin/python3 scripts/equity_bars_collector.py

# Backfill past in-band markets (one-shot operator action)
venv/bin/python3 scripts/equity_bars_collector.py --backfill-past 365 --limit 500

# One-off symbol fetch
venv/bin/python3 scripts/equity_bars_collector.py --symbol AAPL --lookback-days 5

# Manual retention sweep (substitute for Community-edition add_retention_policy)
venv/bin/python3 scripts/equity_bars_collector.py --prune-older-than 730

Cache hit threshold: 80% of expected bars per day (~1140 bars/day × window days). Full Polygon refetch on cache miss; idempotent UPSERT via PRIMARY KEY (symbol, ts).

src/earnings/equity_bar_cache.py:fetch_minute_bars_cached() is the single shared helper per memory §M — analyze_finding_a_full.py uses it when DATABASE_URL is set. Other simulators can opt in via the same function.

Apache TimescaleDB note: prod runs Apache edition (not Community), so add_retention_policy and add_compression_policy are not available. Migration 025 wraps both in DO blocks that fail gracefully; manual retention is enforced via the daily collector's --prune-older-than 730 arg. (Same constraint applies to price_snapshots — its retention/compression in migration 001 also silently failed for years; no impact observed.)

Phase 3 Unified Simulator (Added May 28, 2026 — Findings #306.a/b/c + #307 + #308; PR 1 COMPLETE)

Companion to the Apr-25 Finding-A Simulator below. The Phase 3 simulator generalizes the replay engine to support arbitrary variant rule sweeps (volume-gate / sizing / regime / depth) with the 5-check bias gauntlet integrated. Stop-loss math stays anchored in scripts/analyze_finding_a_full.py via Q1 HYBRID delegation (scripts/phase3_simulator/reference_adapter.py). Design doc: docs/PHASE_3_SIMULATOR_DESIGN.md (867 lines, 16 sections).

Package layout (scripts/phase3_simulator/):

Common operator commands:

cd [hidden] && source .env && export DATABASE_URL POLYGON_KEY PYTHONPATH=[hidden]

# CLI smoke (no DB / no Polygon)
venv/bin/python3 scripts/phase3_simulator/runner.py --skip-db

# Load + print trade summary (no replay; cheap sanity check)
venv/bin/python3 scripts/phase3_simulator/runner.py --population all-closed --limit 200

# Full pipeline: baseline replay + 5-check gauntlet + emit JSON
# Wall time ~5-8 min depending on equity_bars_1m cache hit rate (Finding #240).
venv/bin/python3 scripts/phase3_simulator/runner.py --run-gauntlet --limit 200 \
    --output /tmp/p3sim_baseline.json --log-level INFO

# §10.1 cross-check acceptance gate vs analyze_finding_a_full.py
# Expects verdict PASS with match_pct >= 95% within $1 on every rule
venv/bin/python3 scripts/phase3_simulator/cross_check_baseline.py --limit 200 --log-level INFO

# §10.2 reproducibility check on real prod data (operator-runnable)
venv/bin/python3 scripts/phase3_simulator/runner.py --run-gauntlet --limit 200 --output /tmp/run1.json
venv/bin/python3 scripts/phase3_simulator/runner.py --run-gauntlet --limit 200 --output /tmp/run2.json
venv/bin/python3 -c "import json, sys; sys.path.insert(0, '[hidden]'); \
    from scripts.phase3_simulator.output import mask_volatile_fields; \
    a=mask_volatile_fields(json.load(open('/tmp/run1.json'))); \
    b=mask_volatile_fields(json.load(open('/tmp/run2.json'))); \
    print('IDENTICAL' if a==b else 'DIFFER')"

Acceptance gates (verified May 28, 2026 LATE evening on prod):

Status (PR 2-5 per design doc §12) — ✅ ALL COMPLETE; Phase 4 COMPLETE (Findings #306-#322, incl. #320):

• PR 2 — Volume-gate retune sweep ✅ COMPLETE. gate_variants.py $2k → $1k → $500 → $250 → $100 → $0 sweep + replay.py counterfactual-admission extension for skip_reason='low_volume' opportunity_log rows (activates the latent opp_log_pm_fields_by_audit_id plumbing for pm_v2_orderbook cost-model on newly-admitted markets) + first phase3_sim_gate_retune dashboard section + operator AskQuestion ship decisions per variant. (Shipped the $1,000 volume gate — Finding #311.)
• PR 3 — Sizing + priority + exposure-cap variants ✅ COMPLETE. sizing_policies.py mcap-tier-pct sweep + tier-collapse + per-direction SKIP rule revalidation + position×regime interaction + exposure_cap_variants.py rare/normal/busy redefinition + capital_simulator.py for realistic overlapping-position capital reservation.
• PR 4 — Regime classifier v2 candidates ✅ COMPLETE. regime_classifiers.py baseline compound_v1 + EMA20 + MA50 + drawdown OR clause. VIX-term-structure / breadth / credit / FCI deferred to Phase 4 nice-to-have unless operator explicitly requests.
• PR 5 — Depth-gate v2 decomposition ✅ COMPLETE. depth_gate_variants.py decomposed spread_gate + depth_gate variants + threshold sweep.

Unit tests: 112 tests in tests/unit/phase3_simulator/ (13 cost_model + 17 gauntlet + 16 output + 20 population + 13 reference_adapter + 9 replay + 24 rules). Plus 42 prior signal-test files = 154/154 PASS on prod Python 3.12.

Finding-A Simulator + Strategy Report Refresh (Added Apr 25, 2026 — Finding #228)

The dashboard 📈 Strategy Analysis tab is fed by scripts/strategy_analysis_report.json, which embeds the 🎯 Action Queue (combined ranked list with status / source / live evidence per row, mirroring docs/STRATEGY_PLAYBOOK.md §2). Two scripts produce that JSON.

cd [hidden] && source .env && export DATABASE_URL POLYGON_KEY PYTHONPATH=[hidden]

# 1. Re-run the comprehensive Finding-A counterfactual simulator (read-only).
#    ~8 minutes for full closed-trade population. Outputs scripts/finding_a_full.csv.
#    Re-run BEFORE proposing any new stop-loss rule (memory.md §K).
venv/bin/python3 scripts/analyze_finding_a_full.py --polygon-sleep 0.05
# Optional: focus on the stopped-trade subset (faster):
# venv/bin/python3 scripts/analyze_finding_a_full.py --population stopped-only

# 2. Regenerate the master strategy report (loads the simulator CSV + queries DB).
#    ~30 seconds. Updates the dashboard's 🎯 Action Queue + 🛡️ Bias Gauntlet +
#    🔬 Finding-A Simulator + 🔁 Learning Loop sections.
venv/bin/python3 scripts/strategy_performance_analysis.py

The dashboard reads the JSON on tab open; no service restart needed. Sanity check after re-run:

venv/bin/python3 -c "
import json
d = json.load(open('scripts/strategy_analysis_report.json'))
print('action_queue summary:', d['action_queue']['summary'])
print('finding_a_simulator available:', d['finding_a_simulator']['available'])
"
# Expected (Apr 25 baseline): {'total': 22, 'in_queue': 8, 'implemented': 7, 'retracted_or_failed': 6, 'info': 1}; available True.

Canonical narratives: - docs/FINDINGS.md Finding #228 — full retraction + simulator design. - docs/STRATEGY_AUDIT_v3.md §10 — audit-trail entry with the per-rule comparison table. - docs/STRATEGY_PLAYBOOK.md — forward-looking source of truth (mirror of dashboard Action Queue).

4. Data Collection

Data Sources

Live Trades Note: Authenticated CLOB trades return only our trades. Live trades are collected via the Polymarket data-api /trades?asset_id= endpoint by live_trades_collector.py (Finding #289 cosmetic rename — same service, same data, just no longer misleadingly named "Dome"); CLOB trades are disabled by default in continuous_collector.py.

⚠️ CRITICAL: Data Source Selection

FOR LIVE PRICES (Conviction/Probability):
  ✅ USE: Gamma API outcomePrices field
  ❌ DO NOT USE: CLOB API /book endpoint

FOR ORDERBOOK DEPTH (Liquidity Analysis):
  ✅ USE: CLOB API /book endpoint

The CLOB orderbook bids/asks represent ORDER LEVELS, not the market price!
This was a critical bug fixed in January 2026.

Collection Schedule

┌─────────────────────────────────────────────────────────────────┐
│                    COLLECTION SCHEDULE                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Every 60 seconds (gamma_collector):                            │
│   ├─ Fetch all earnings markets from Gamma API                  │
│   ├─ Parse outcomePrices (JSON string → float)                  │
│   ├─ Save to price_snapshots table                              │
│   └─ Sync market metadata                                        │
│                                                                  │
│   Weekly Sun 06:00 ET (polymarket-trades-backfill.timer):        │
│   ├─ Fetch historical trades from Polymarket data-api /trades   │
│   ├─ Batch write every 500 trades                               │
│   ├─ Smart Option A: T1 delta + T2 smart re-fetch + T3          │
│   │  earnings-only scope (Findings #269 + #288)                 │
│   └─ Track progress in trades_backfill_state.json               │
│                                                                  │
│   Daily via polymarket-api-health.timer (#289.a):               │
│   └─ API health check                                            │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

5. Backfill Status

Current Gap Status (January 5, 2026)

Price Backfill Details — DECOMMISSIONED Finding #289 (May 10, 2026)

The "Dec 18 - Jan 4 gap" backfill was completed in early 2026 using the old Dome /market-price endpoint. With Dome EOL'd Apr 28, 2026, the historical-prices backfill scripts (prices_backfill_service, dome_prices_backfill, backfill_prices) were stopped and deleted. Live capture via polymarket-collector.service writes price_snapshots continuously and is monitored by the freshness:price_snapshots integrity rule (10-min threshold). If a future need arises, the Polymarket replacement is clob.polymarket.com/prices-history?market=<token>&startTs=...&endTs=... (rate limit 1,000 req/10s = 100 q/s — even faster than the migrated /trades endpoint).

Monitor Backfill Progress

# Price backfill status
cat [hidden]/price_backfill_state.json | python3 -c "
import sys,json
d = json.load(sys.stdin)
total = 422
done = len(d.get('completed_markets', []))
nodata = len(d.get('markets_with_no_data', []))
print(f'Markets: {done}/{total} ({done*100/total:.1f}%)')
print(f'No Data: {nodata}')
print(f'Prices: {d.get(\"prices_saved\", 0):,}')
"

# Trades backfill status
cat [hidden]/trades_backfill_state.json | python3 -c "
import sys,json
d = json.load(sys.stdin)
total = 4709
done = len(d.get('completed_markets', []))
print(f'Markets: {done}/{total} ({done*100/total:.1f}%)')
print(f'Trades: {d.get(\"trades_saved\", 0):,}')
"

Backfill Priority Order

Note (May 10, 2026): the original P0+/P0 priority list referenced Dec 18 - Jan 4 gap and Dome backfill — both completed in early 2026. Current priorities post Dome→Polymarket migration (Findings #287/#288/#289):

1. P1: Continuous Collection — keep polymarket-collector.service + polymarket-live-trades.service + polymarket-websocket.service running 24/7. Monitored by integrity-monitor freshness rules.
2. P2: Weekly trades-backfill (Polymarket data-api /trades) — polymarket-trades-backfill.timer Sun 06:00 ET (Findings #269 + #288).
3. P3: Gap Monitoring — integrity-monitor surfaces freshness + service-exit issues to Telegram (Finding #258 + #268).

6. Database

Polymarket opportunity log (Migration 021, Finding #221)

Append-only table for research and counterfactual analysis. Apply on managed Postgres when deploying the feature:

cd [hidden] && source venv/bin/activate
# From project root; use your real connection string
psql "$DATABASE_URL" -f src/data_tracker/migrations/021_polymarket_opportunity_log.sql

Offline counterfactual pricing (all skip_stage values; rate-limit friendly):

export PYTHONPATH=[hidden]
export DATABASE_URL=...
export POLYGON_KEY=...   # strongly recommended: 1m bars + intraday stop model
# optional: export COUNTERFACTUAL_NOTIONAL_USD=10000
python3 scripts/compute_counterfactual_pnl.py --limit 100 --sleep-seconds 12

• With POLYGON_KEY: Polygon 1-minute aggs; counterfactual_version=polygon_minute_v1; hypothetical_prices_source=polygon.io_1m_close.
• Without Polygon: yfinance daily closes only; yfinance_daily_approx_v1 — no intraday stop (documented in src/earnings/counterfactual_pricing.py).
• Non-simulatable rows (no symbol, no schedule, or non-directional) get no_price_simulation_v1.
• If DATABASE_URL uses postgresql+pg8000://…, psql may reject it; apply SQL migrations via SQLAlchemy or a libpq URL (see prior deploy notes).

Core logic: src/earnings/counterfactual_pricing.py. Tests: tests/unit/test_counterfactual_pricing.py.

Schema Overview

Hypertable retention & archive (OPS-1 — DONE Aug 21 2026 (#383); THE runbook — other docs cite here)

Policy (operator-approved, Aug 2026): keep ~90 days of orderbook_snapshots live in the hypertable; everything older is losslessly archived to DigitalOcean Spaces and dropped from the live table, behind a mandatory per-chunk proof gate. History: native TimescaleDB compression is impossible on this cluster (DO ships the Apache-licensed edition, no columnstore — Finding #376 has the full blocker evidence and the option analysis); the operator then approved option C (archive + drop) with the binding condition that the archived data stays fully detailed and this runbook exists.

What is archived: whole daily chunks of orderbook_snapshots — every row, every column (the 24-column list pinned in scripts/ops1_compression_fidelity_proof.py::OB_COLUMNS), exact types, no downsampling/aggregation. Proven byte-identical through archive→restore→replay before any drop (acceptance run: TTD/RKLB/SG 2026-05-07 — raw-dump sha256, conviction, point-in-time book, and gap-aware counterfactual outputs all byte-matched the pre-archive baseline; Finding #383).

Where it lives: private Spaces bucket polymarket-orderbook-archive (region nyc3, endpoint https://nyc3.digitaloceanspaces.com). Layout per archived chunk range:

orderbook_snapshots/<YYYY-MM-DD>_<HHMMSS of range_start>/
  data.csv.zst    # zstd-9 canonical CSV: header = OB_COLUMNS, ORDER BY condition_id,time,
                  # NULLs as empty, jsonb as compact JSON (json.dumps) so it re-inserts cleanly
  manifest.json   # row_count, csv_sha256, zst_sha256, byte sizes, column names+types,
                  # source chunk name, range bounds, dumped_at  (format: orderbook_archive.v1)

The ledger (start here): DB table orderbook_archive_log (migration 035) — one row per archived range with hashes, object keys, proof_status, dropped flag, restore/spot-verify timestamps. It is the SSOT for what's archived and what's currently dropped; the restore tool and the ConvictionCalculator archived-range guard both read it.

Credentials: ORDERBOOK_ARCHIVE_SPACES_KEY/SECRET/BUCKET/REGION in the server .env (root-only, 0600). The key (polymarket-orderbook-archive, created via doctl spaces keys create) is DELIBERATELY not stored under the generic SPACES_* names — those would silently activate the DB-backup Spaces upload the operator chose to keep local-only (§ exit-code table). If the key is ever lost, mint a new one: doctl spaces keys create <name> --grants 'bucket=;permission=fullaccess' and update .env.

How to LIST what's archived:

cd [hidden] && set -a && source .env && set +a
venv/bin/python3 scripts/orderbook_archive_restore.py --list                # everything
venv/bin/python3 scripts/orderbook_archive_restore.py --list --from 2026-04-25 --to 2026-05-13

How to RESTORE (worked example — this exact sequence was the acceptance test):

# Symptom you'll see first: ConvictionCalculator raises ArchivedRangeError
# ("range [...] is ARCHIVED ... run scripts/orderbook_archive_restore.py ...")
# instead of silently returning nothing.
cd [hidden] && set -a && source .env && set +a
venv/bin/python3 scripts/orderbook_archive_restore.py --restore --from 2026-04-23 --to 2026-05-08
# → downloads each range, verifies object + CSV sha256 against the ledger,
#   inserts the rows BACK INTO the live hypertable. After this, studies need to
#   do NOTHING differently — the range is simply live again.
# One-off inspection without touching the hypertable: add --scratch-table my_table
# When the study is done, re-drop (re-dumps live data, hash-checks vs manifest, drops):
venv/bin/python3 scripts/orderbook_archive.py --archive --from 2026-04-23 --to 2026-05-08

The proof gate (why a drop is safe): scripts/orderbook_archive.py will only drop a chunk after: canonical dump + sha256 → zstd-9 → upload → download-back hash check → scratch-restore of the downloaded copy + canonical re-dump must byte-match the original hash → live range re-dump must still match (nothing changed mid-flight). Byte-identity of the canonical dump implies every SQL query on restored data returns exactly what the live chunk returned — verified end-to-end once by replaying the full counterfactual harness (scripts/ops1_compression_fidelity_proof.py) through an archive→drop→restore cycle (Finding #383; artifacts at /root/ops1_proof/{pre,pre2,post} on the droplet). Any gate failure keeps the chunk live, writes proof_status='fail' (integrity rule archive:orderbook_proof_gate → FAIL), and stops the pipeline.

Automation: polymarket-orderbook-archive.timer — 00:35 UTC (post-US-close extra drain, #411) and 06:35 UTC (overnight quiet window), 18 chunks/run (ARCHIVE_MAX_CHUNKS_PER_RUN), 5 s sleep, TimeoutStartSec=21600 (history: 06:35-only × 40 chunks / 3h kill-clock — SIGTERM of a healthy run Aug 24, Finding #411). Log: /var/log/polymarket/polymarket-orderbook-archive.log. polymarket-orderbook-archive-spotverify.timer — monthly (1st, 06:15 UTC) re-downloads the least-recently-verified archived range and re-hashes it. Integrity rules: archive:orderbook_proof_gate (HIGH), archive:orderbook_backlog (timer progress), archive:orderbook_spaces_reachability (HIGH — the bucket holds the ONLY copy of dropped ranges), archive:orderbook_spot_verify. Storage threshold re-armed at 12 GB for the ~90-day steady state (history: 20 GB until Aug 21 2026). storage:orderbook_snapshots is drain-aware (Finding #390): while the backlog drains, a size FAIL downgrades to WARN only if archive:orderbook_backlog is OK AND today ≤ ORDERBOOK_DRAIN_GRACE_UNTIL (dated constant in integrity_monitor.py, #356-style expiry); it reverts to FAIL on its own if the drain stalls. After ORDERBOOK_DRAIN_GRACE_UNTIL, a size above the line stays FAIL (#487): backlog OK with no live chunk older than 90 days means the drain finished and the 90-day working set is over the line, and the message says that instead of saying the drain failed; a backlog that is not OK still says the drain stalled. The 12 GB threshold itself is never weakened.

Fail-loud guard for studies: ConvictionCalculator raises ArchivedRangeError (with the exact restore command) when a historical conviction window overlaps a dropped range — it never silently falls back or returns None for archived data. Analysis harnesses that go through ConvictionCalculator (the replay/counterfactual paths do) inherit this automatically.

Key Tables

-- Price snapshots (primary price data)
price_snapshots:
  - time: TIMESTAMP WITH TIME ZONE
  - condition_id: VARCHAR(66)
  - yes_price: DECIMAL(10,4)  -- 0.0 to 1.0
  - no_price: DECIMAL(10,4)   -- 1 - yes_price
  - volume_24h: DECIMAL(20,2)
  - liquidity: DECIMAL(20,2)

-- Markets (metadata)
markets:
  - condition_id: VARCHAR(66) PRIMARY KEY
  - token_id_yes: VARCHAR(80)
  - token_id_no: VARCHAR(80)
  - question: TEXT
  - status: VARCHAR(20)  -- 'open' or 'closed'
  - tags: TEXT[]

-- Data gaps (quality tracking)
data_gaps:
  - id: SERIAL PRIMARY KEY
  - gap_type: VARCHAR(50)
  - start_time: TIMESTAMP
  - end_time: TIMESTAMP
  - records_affected: INTEGER
  - status: VARCHAR(20)

-- Trade Audit Log (CRITICAL - all trade history)
trade_audit_log:
  - id: SERIAL PRIMARY KEY
  - trade_id: VARCHAR(50) UNIQUE
  - symbol: VARCHAR(10)
  - direction: VARCHAR(10)  -- LONG/SHORT
  - timing: VARCHAR(10)  -- BMO/AMC
  -- Entry data
  - entry_time, entry_price, entry_order_type
  - entry_slippage, entry_slippage_pct, entry_commission
  - entry_order_id  -- IBKR order ID
  -- Exit data
  - exit_time, exit_price, exit_order_type
  - exit_slippage, exit_slippage_pct, exit_commission
  - exit_reason  -- scheduled/stop_loss/instant_exit/manual
  -- Position sizing
  - shares, position_value, position_size_pct, capital_at_entry
  -- Stop loss
  - stop_price, stop_distance_pct, stop_triggered
  -- P&L
  - realized_pnl, realized_pnl_pct, gross_pnl
  - total_commission, total_slippage
  -- Conviction at entry
  - conviction_at_entry, yes_price_at_entry, spread_at_entry
  - volume_24h_at_entry, conviction_source
  -- Conviction at exit
  - conviction_at_exit, prediction_correct, actual_earnings_result
  -- Market conditions
  - mcap, mcap_tier, vix_at_entry, sp500_change_at_entry
  -- Polymarket info
  - condition_id, polymarket_question, polymarket_volume_total
  -- Strategy factors
  - entry_reason, strategy_version, risk_level_at_entry
  -- Analysis
  - notes, auto_analysis, lessons_learned, trade_grade
  - created_at, updated_at

Trade Data Retention Policy

All trade data is permanently stored. This includes: - Complete IBKR execution details (prices, fees, slippage, order IDs) - Polymarket conviction data at entry AND exit - All decision factors (VIX, market conditions, risk level) - Post-trade analysis and notes

This enables: - Historical performance analysis - Strategy optimization - Regulatory compliance - Tax reporting

Useful Queries

-- Check table row counts
SELECT 
  'markets' as table_name, COUNT(*) FROM markets
UNION ALL SELECT 
  'price_snapshots', COUNT(*) FROM price_snapshots
UNION ALL SELECT 
  'trades', COUNT(*) FROM trades;

-- Check recent price data
SELECT condition_id, time, yes_price 
FROM price_snapshots 
ORDER BY time DESC LIMIT 10;

-- Check data gaps
SELECT * FROM data_gaps ORDER BY start_time DESC;

-- Find earnings markets
SELECT condition_id, question, status 
FROM markets 
WHERE question ILIKE '%beat%earnings%' 
  AND status = 'open';

Database Maintenance

# Run ANALYZE to update statistics (for accurate counts)
ssh root@[hidden]
cd [hidden] && source venv/bin/activate
python3 << 'EOF'
from sqlalchemy import create_engine, text
import os, ssl
with open('.env') as f:
    for line in f:
        if line.startswith('DATABASE_URL='):
            db_url = line.split('=', 1)[1].strip().replace('?ssl=true', '')
            break
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
engine = create_engine(db_url, connect_args={'ssl_context': ssl_context})
with engine.connect() as conn:
    for table in ['markets', 'price_snapshots', 'orderbook_snapshots', 'trades']:
        conn.execute(text(f'ANALYZE {table}'))
        print(f'Analyzed {table}')
    conn.commit()
EOF

Deploy-time migrations — ledger (Finding #444, Sep 26, 2026)

deploy/deploy_from_github.sh runs deploy/migrate.py after git pull and BEFORE any service restart. The runner keeps schema_migrations (migration 038) and applies only the MANIFEST IDs missing from it. A steady-state deploy issues zero DDL. The runner never re-runs 029: its ADD COLUMN IF NOT EXISTS still takes ACCESS EXCLUSIVE on earnings_markets.

• Output: printed in the deploy output and appended to /var/log/polymarket/deploy-migrations.log (one === <UTC> deploy <sha> === header per run). Last line: migrate: done: current=N bootstrap=N applied=N.
• Bootstrap: if a MANIFEST ID is missing from the ledger, every object it creates exists (catalog probe) AND the reviewed baseline in deploy/migration_baseline.json matches, the runner records it as bootstrap and does NOT execute the file. Since Finding #474 the baseline pins every column's type with length/precision (format_type), nullability and default; every PK/UNIQUE/FK/CHECK constraint (pg_get_constraintdef) and index definition (pg_get_indexdef) of the tables the file creates; and every seed row by key with the values the file writes, including later UPDATEs. Catalog text is read with search_path=pg_catalog, so names are schema-qualified. Extra live objects from later migrations are allowed. Objects present but baseline missing or different (for example a partially applied file): the deploy stops (fail closed) — investigate by hand. Deliberately not required: the two btc5m.live_samples rows 031 inserts (the Sep 25, 2026 incident cleanup archived and truncated that table) and the mutable columns of the btc5m singletons.
• Baseline tools (read-only, forced default_transaction_read_only=on): --check-baseline runs the probe + full baseline for every MANIFEST entry whether or not it is in the ledger (prod Sep 26, 2026: 5/5 entries match, PostgreSQL 16.15). --capture-baseline prints the committed file's structure filled with the connected database's facts (JSON on stdout); diff it against the file and review before committing a refresh.
• New DDL: first a preflight lists transactions older than 5 s that hold locks on the tables the DDL needs, plus the pg_blocking_pids queue. Then one transaction applies the file and its ledger row, with lock_timeout=5s and statement_timeout=120s, up to 3 attempts. Every failure exits the deploy before restarts, etfstack/btc5m included (review of Batch A, M1).
• Checksum: a migration whose file sha256 differs from its ledger row stops the deploy (numbered migrations are immutable). Restore the applied file and ship the change as a new numbered migration.
• Adding a migration: add the numbered file, then add a Migration(...) entry to MANIFEST in deploy/migrate.py declaring every table, index, column, schema and hypertable it creates. On prod it is applied, not bootstrapped. The unit suite still needs a baseline entry for every MANIFEST ID (tests/unit/test_finding_444_migration_ledger.py), and tests/unit/test_finding_474_bootstrap_baseline.py checks that the entry encodes what the file declares; after the deploy, confirm it with --check-baseline and refresh it from --capture-baseline. tests/unit/test_finding_444_migration_ledger.py fails if a src/data_tracker/migrations/0NN ≥ 038 is not registered, or if an object is undeclared.
• Pre-flight / what would run (read-only):

ssh root@[hidden] 'cd [hidden] && venv/bin/python3 deploy/migrate.py --dry-run'
# DRY-RUN forces default_transaction_read_only=on; prints would APPLY / would record as bootstrap
ssh root@[hidden] 'cd [hidden] && venv/bin/python3 deploy/migrate.py --check-baseline'
# Finding #474: full baseline for every entry; last line "check-baseline: N/N entries match"

• Ledger: SELECT migration_id, method, applied_at, git_sha FROM schema_migrations ORDER BY applied_at;
• Guarded data corrections (Findings #477/#480): a row-only fix is a numbered file registered in numeric order in DATA_MANIFEST, never schema MANIFEST (there are no objects to probe or bootstrap). The runner applies each once after schema migrations, in the same psql -1 transaction as its ledger row. The file sets local lock/statement timeouts and client_min_messages=notice; every UPDATE uses exact reviewed before-values, then GET DIAGNOSTICS ROW_COUNT and raises unless exactly one row changed. One RAISE NOTICE per row logs before → after. Any mismatch rolls back the entire file and ledger row and stops before restarts; the ledger makes later deploys no-ops. Templates: 040_pnl_fields_bbw_snow.sql and 041_channel_leak_false_rows.sql; tripwires: tests/unit/test_finding_477_migration_040.py and tests/unit/test_finding_480_migration_041.py.
• If a deploy stops on a migration: read the log. The runner has already rolled back. Find the lock holder it printed and let it finish, or pg_cancel_backend(<pid>) if it is a runaway dashboard/analysis read. Never cancel a trader persist. Then re-run the deploy. Services were not restarted, so the old code is still running, but the pulled code is on disk: do not restart the trader by hand until the migration is applied.

Post-deploy: release Sep-26 live-trade poison after migration 042 (Finding #481)

Done for the Sep-26 incident: DONE Sep 27, 2026 02:47 UTC (#482) in deploy #15 860af84 — 3 payload files (2 distinct trades) released, quarantine/ empty, artifact:db_write_spool ok. The steps below stay as the procedure for any future poison release.

Deploy #14 (28b572b) left two quarantine payload files for one market-data trade (same spool key; long outcome overflowed trades.token_label varchar(20)). A second trade's payload appeared at 02:42 UTC before deploy #15. Migration 042 widens token_label to varchar(64) (same as outcome). The collector seeds _seen_trade_ids and poison payload hashes from quarantine files on startup, so widening alone does not insert the rows — every kept quarantine payload must be released before the next collector start.

Order (operator, on prod):

1. Deploy this migration (normal deploy_from_github.sh run applies 042 before service restarts). Confirm schema_migrations shows 042 with method=applied and format_type for public.trades.token_label is character varying(64).
2. systemctl stop polymarket-live-trades (if the deploy already restarted it, stop again before release).
3. List every kept quarantine payload (not .error.json sidecars). db_write_spool_admin.py list only prints counts — use ls for paths: bash ls /var/lib/polymarket/db-write-spool/quarantine/live_trade-*.json # omit any path ending in .error.json; release each remaining payload cd [hidden] venv/bin/python3 scripts/db_write_spool_admin.py show --record /var/lib/polymarket/db-write-spool/quarantine/live_trade-<stamp>-<hash>.json
4. Forensic release each payload (moves payload + .error.json under released/; not a delete). Repeat for every file from step 3: bash venv/bin/python3 scripts/db_write_spool_admin.py release \ --record /var/lib/polymarket/db-write-spool/quarantine/live_trade-<stamp>-<hash>.json \ --operator "<name>" \ --reason "042 token_label widen — re-admit incident trade for API re-fetch" \ --confirm RELEASE-AFTER-VERIFICATION
5. systemctl start polymarket-live-trades.
6. Verify (read-only): quarantine/ has no live_trade-*.json payloads; each released lp_<tx>_<ts> appears in trade_dedupe / trades if the Polymarket API still returns it; artifact:db_write_spool leaves WARN (one CRITICAL RESOLVED page is expected).

Do not release while token_label is still varchar(20) — re-admit would 22001 and quarantine again. Do not skip release on the deploy that ships 042: a restart with any quarantine payload still present marks those trades seen and skips insert even after the widen.

Polymarket Trades Data Backup & Retention (Added Jan 19, 2026)

The trades table stores raw Polymarket trade data and grows continuously. Unlike trading data, this raw market data is subject to retention limits.

Why Backup is Needed: - trades table reached 14GB (~7M rows) causing database memory pressure - DigitalOcean managed PostgreSQL has limited shared memory for locks (mitigations: docs/POSTGRES_OPERATIONS.md, Finding #216) - TimescaleDB compression unavailable in Community Edition

Retention Policy: | Data | Retention | Location | |------|-----------|----------| | Live trades | 30 days | Database | | Archived trades | Indefinite | [hidden]/backups/ |

Backup Files (January 2026):

[hidden]/backups/
├── trades_archive_20260118_165503.csv.gz (800 MB, 5.28M rows)
└── trades_archive_20260118_165503_metadata.txt

Manual Backup & Cleanup:

ssh root@[hidden]
cd [hidden] && source venv/bin/activate

# Step 1: Backup old data
python3 scripts/backup_old_trades.py

# Step 2: Delete old data (runs day-by-day for efficiency)
python3 scripts/delete_old_trades.py

# Verify
python3 -c "
from sqlalchemy import create_engine, text
import os
db_url = open('.env').read().split('DATABASE_URL=')[1].split()[0]
engine = create_engine(db_url)
with engine.connect() as conn:
    r = conn.execute(text('SELECT COUNT(*) FROM trades'))
    print(f'Trades remaining: {list(r)[0][0]:,}')
"

Restore from Backup (if needed):

# Decompress
gunzip -c backups/trades_archive_YYYYMMDD.csv.gz > trades_restore.csv

# Import (requires table recreation or COPY command)
# Contact DBA for large imports

Database Connection Pooling (Added Jan 20, 2026)

All services now use a shared connection pool to prevent connection leaks.

Check Connection Count:

ssh root@[hidden]
cd [hidden] && source venv/bin/activate && set -a && . ./.env && set +a
python3 << 'EOF'
import os
from sqlalchemy import create_engine, text
import ssl

db_url = os.environ["DATABASE_URL"]
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
engine = create_engine(db_url, connect_args={"ssl_context": ssl_context})

with engine.connect() as conn:
    result = conn.execute(text("""
        SELECT state, count(*) 
        FROM pg_stat_activity 
        WHERE datname = current_database() 
        GROUP BY state
    """))
    for row in result:
        print(f"{row[0] or 'unknown'}: {row[1]} connections")
EOF

Expected Values: | Metric | Normal | Warning | |--------|--------|---------| | Total connections | < 20 | > 30 | | Idle connections | < 15 | > 25 | | Long idle (>5min) | < 5 | > 10 |

Shared Connection Module: src/db/connection.py

If connection count exceeds normal values: 1. Check for service restarts that may have leaked connections 2. Restart affected services 3. Older idle connections will be recycled automatically (30 min)

7. Dashboard

Access

URL: http://[hidden]:8080

Tabs:
- Trading: Upcoming earnings, conviction, volume
- Database: Data coverage, gaps, backfill status
- System: API health, dependencies, service status
- Trading tab includes IBKR Reality Overlay (DB vs IBKR positions + stop orders)
- Testing Findings tab includes full post-audit log with filter + count

IBKR Snapshot Source (Updated Feb 2026; cadence Finding #472, Sep 26, 2026): - Dashboard no longer connects to IBKR directly. - Live IBKR data is read from the trading service snapshot file: - Path: IBKR_SNAPSHOT_PATH (default: logs/ibkr_snapshot.json) - Write + freshness: src/earnings/ibkr_snapshot.py (shared). Cadence = max(write interval, loop idle). IBKR_SNAPSHOT_INTERVAL_SEC (default 10s) is the write floor during 09:00–20:00 ET; off-hours floor 60s; weekend/holiday loop idle 60s. Overlay stale = cadence × IBKR_SNAPSHOT_STALE_SLACK_MULT (default 2). Do not treat age > 10s as disconnected.

Cold-start expectations (Updated Apr 26, 2026 — Finding #249): - Phase 1 — DB-only bootstrap (_bootstrap_cache_from_db()): completes ~1 s after restart. 📅 Upcoming Earnings widget shows DB rows immediately with mcap_tier from src/earnings/market_cap_tier.py for rows with a PM market (KNOWN_MCAPS + the shared cache, no network in this pass; unknown = Small — #450 review) and earnings_markets.market_cap_category for the rest (Finding #219). Company name + exchange may be empty until phase 2 enriches. - Phase 2 — full enrichment (_refresh_cache()): completes ~30 s–2 min after restart on a healthy day; up to ~5 min if yfinance is heavily rate-limited (Polygon 2nd-tier resolves ~98 % of misses at ~150 ms each). - Steady-state — 30 s background refresh loop. mcap cache is warm; refreshes are cheap. - 2-hour deep-refresh — clears _mcap_cache and re-fetches; expect a ~30 s–2 min stall every 2 hours. - If you see Live 0 rows · DB N rows for >5 min after a restart, check journalctl -u polymarket-dashboard | grep -E 'DB-bootstrap|Polygon fallback|Cache background refresh started' — the bootstrap log line should fire within seconds; absence means a bootstrap failure (check WARNING logs).

API Endpoints

Public dashboard defense in depth (Finding #467, Sep 26, 2026)

The dashboard (and its /docs/* pages) stay open to the internet by operator decision. Two controls keep secrets off it:

1. This repo's committed-secret scrub (#457, reliability/cred-scrub) — a guard test over tracked files, so a secret can't be re-committed into a doc.
2. The runtime response filter (earnings_dashboard/secret_redaction.py, registered with one line: install_secret_redaction(app) right after app = FastAPI(...)) — redacts secret VALUES (collected from os.environ at app startup — the same .env sourced by the systemd unit's EnvironmentFile=) from EVERY response: HTML, JSON, plain text, 404s, error pages, and unhandled exceptions. This is what catches a secret that slips into a doc, a log excerpt, or an error message even though nothing in that code path is "about" secrets. Never call os.environ secret VALUES in a log line — the module itself never does, and it never will regardless of what leaks into a response body.

Separately, earnings_dashboard/identifier_masking.py masks IBKR account/order ids, wallet addresses, and server hostnames/IPs/paths at each API payload's source (performance/position data stays visible). Both the main dashboard and its two modular subtrees (strategies/btc5m/, strategies/etfstack/) have their own copies of the small mask helpers — each subtree is "modular, separately-extractable" and must not import from earnings_dashboard.

FastAPI's own auto-docs (/openapi.json, /docs, /docs/oauth2-redirect, /redoc) are disabled via the FastAPI() constructor (not to be confused with the dashboard's own /docs/{doc_name} markdown route, which is unaffected). Three always-inert unauthenticated write endpoints were removed outright: POST /api/testing-prod/confirm, /reset, and /api/set-trading-mode — see "Change Trading Mode" above.

If you add a new secret-shaped env var: no action needed — the filter's name-matching rule (ends in KEY/TOKEN, contains SECRET/PASS/PASSPHRASE/PRIVATE_KEY) picks it up automatically at the next app restart. If you add a new IBKR/wallet/path field to a dashboard payload: mask it at the same call site per identifier_masking.py's pattern — the filter above is a backstop, not a substitute, for values that aren't in the current environment (like the btc5m/etfstack subtree's hardcoded wallet/host constants, which the filter also knows about explicitly — see secret_redaction._collect_known_identifier_literals()). Full test coverage: tests/unit/test_finding_467_dashboard_hardening.py.

Restart Dashboard

ssh root@[hidden]

# Kill existing dashboard
pkill -f uvicorn

# Restart
cd [hidden] && source venv/bin/activate
nohup uvicorn earnings_dashboard.app:app --host 0.0.0.0 --port 8080 \
  > /var/log/polymarket/dashboard.log 2>&1 &

# Verify
curl http://localhost:8080/api/health

Dashboard DB load and the read-only guard (Finding #445)

The dashboard is the heaviest DB reader. Its hypertable SQL is time-bounded, coverage is cached for 6 h, and each statement runs in its own short transaction with a 60 s statement_timeout (rules: docs/POSTGRES_OPERATIONS.md "Temp-file spill vs the managed disk guard"). If the managed Postgres starts flipping read-only again (25006 / 57P01 / network error in the trader, collector, live-trades or dashboard logs):

# Read-only probes (two reads, 5 s timeout). No session read-only override here:
# SHOW must report the server's default_transaction_read_only (on = guard fired).
ssh root@[hidden] 'cd [hidden] && U=$(grep -E "^DATABASE_URL=" .env | cut -d= -f2- | tr -d "\""); \
  PGOPTIONS="-c statement_timeout=5000" psql "${U/+pg8000/}" -X -At -c \
  "SELECT sessions_killed, temp_files, temp_bytes FROM pg_stat_database WHERE datname = current_database()" \
  -c "SHOW default_transaction_read_only"'
# Long dashboard transactions (acceptance: none older than 60 s)
#   SELECT pid, now()-xact_start, left(query,80) FROM pg_stat_activity
#   WHERE application_name <> 'deploy_migrate' AND xact_start < now() - interval '60 seconds';

• sessions_killed rising in steps every ~10–12 min means the guard is firing. Ops hold (operator approval; the trader does not need the dashboard): systemctl disable --now polymarket-dashboard. Lift it with systemctl enable --now polymarket-dashboard. A deploy also lifts it: the deploy restarts the dashboard.
• Dashboard-API consumers degrade while it is down: etfstack-signal writes account_value=1.0, leverage validation records nulls, and the Sun 12:00 paper reconnect exits 1.
• temp_bytes only moves when a spilling transaction commits. Compare it across a full stats pass (acceptance: < 100 MB growth per pass; the pre-#445 baseline was 5.2 GiB).

Durable DB-write spool (Finding #455)

The trader and live-trades collector spool DB writes that fail after an external fill/print under /var/lib/polymarket/db-write-spool (override: DB_WRITE_SPOOL_DIR). Per-kind subdirectories are position_open, position_close, ibkr_execution, reconciliation_log, and live_trade; quarantine/ holds corrupt/permanent failures. They contain no credentials.

• A pending/quarantined position_open or position_close is a live-money incident. _pending_fill_spool_blocks_entries() is the exact fail-close lever in both single and batch entry paths; reconciliation cannot reopen it. The IBKR position and hardware stop remain authoritative.
• Never delete, edit, git clean, or blindly replay spool files. The owning service retries under flock + Postgres advisory lock and removes a file only after DB commit and live-book hydration. Migration 039 unique keys cover a crash after commit but before ACK.
• Capacity is per kind; live_trade cannot consume the reserved position_open/position_close capacity. If a fill still cannot spool, an emergency marker + in-process sticky gate keeps admission closed. It does not use trading_block.txt; exits remain allowed.
• Transient DB errors retry indefinitely with exponential backoff and never quarantine. Quarantine means genuine poison/corrupt data and requires the audited operator release below.
• Reconciliation renders a spooled filled position PERSIST_PENDING, not EXTERNAL. Verify IBKR first, then both trading_positions and trade_audit_log.
• Integrity artifact:db_write_spool is CRITICAL immediately for position records/quarantine and when any other kind is older than five minutes.
• The deploy pre-creates this directory and write-tests it as the effective trader and live-trades service users before restarting anything.

Read-only triage:

ssh root@[hidden] '
  systemctl is-active earnings-trader polymarket-live-trades
  test ! -d /var/lib/polymarket/db-write-spool ||
    ls -la /var/lib/polymarket/db-write-spool
  for d in position_open position_close ibkr_execution reconciliation_log live_trade quarantine; do
    test ! -d "/var/lib/polymarket/db-write-spool/$d" ||
      ls -la "/var/lib/polymarket/db-write-spool/$d"
  done
  grep -E "position_persist|execution_persist|reconciliation_persist|live_trades_db_write|#455" \
    [hidden]/logs/earnings_trader.log /var/log/polymarket/live_trades.log 2>/dev/null || true
'

Sanctioned quarantine release (operator only). First verify the symbol's IBKR shares/executions and both DB tables. The release command prints the full record before moving it to released/; it never deletes it. Supply the reviewer's name and the exact reconciliation evidence:

cd [hidden]
PYTHONDONTWRITEBYTECODE=1 venv/bin/python3 scripts/db_write_spool_admin.py release \
  --record /var/lib/polymarket/db-write-spool/quarantine/position_close-<record>.json \
  --operator Marius \
  --reason "IBKR flat; trading_positions and trade_audit_log verified closed; exec <id>" \
  --confirm RELEASE-AFTER-VERIFICATION

Use list or show --record <path> for read-only inspection. If a fill could not be spooled and emergency_fill_block.json exists, reconcile it first, then use the same operator/reason/confirm fields with release-emergency and restart earnings-trader only in a safe window to clear the same-process sticky block. Never restart the Gateway for this. Never rm a pending, quarantined, emergency, or released record.

Migration-039 preflight (read-only; required before deploy):

SELECT ibkr_entry_exec_id, count(*)
FROM trade_audit_log
WHERE ibkr_entry_exec_id IS NOT NULL
GROUP BY 1
HAVING count(*) > 1;

Sep 26 result: 0 rows. The migration repeats this check and aborts on a duplicate. deploy_from_github.sh refuses direct invocation without the pre-pull SHA supplied by remote_deploy.sh. Its EXIT trap resets the checkout to that SHA after any failed pre-restart gate (host, needrestart, migration, conviction schema, spool write); no service has restarted at that point.

Rollback is patch-forward while any spool file exists: the fill spool itself blocks entries; stop only polymarket-live-trades if its retry is defective, copy the spool, fix the replay, and reconcile. Never revert migration 039. No Gateway restart. After the spool is empty, normal trader/live-trades code rollback is safe. The 98 Sep-25 missing prints require a separately approved bounded backfill after deploy; do not infer recovery from an empty spool because those failures predate #455.

8. Monitoring

Loss brakes (CLB-1 + A-6, Finding #366, Aug 20 2026)

The trading service has exactly two loss brakes; the consecutive-loss breaker (3/5/7 losses → 4h/24h/manual) is RETIRED and its counter deleted:

• Daily-loss breaker (sole loss brake): fires at day P&L ≤ −5% of capital (DAILY_LOSS_LIMIT). Day P&L is DB-anchored (trade_audit_log, ET session day, #331 net-of-commissions definition), so it survives service restarts — on startup the trader re-derives today's P&L and re-triggers the pause if the day is already past the limit (A-6, log line A-6 reconstruct (#366)). It resumes at the true next ET market open (market calendar, 9:30 ET — not a fixed UTC hour). No operator action needed; manual_resume does NOT clear it.
• Session stop breaker: 4 stop-exits in an ET day aborts new entries for the session (#312/#335.b — unchanged, already DB-derived/restart-safe).
• Sizing tiers (not a pause): day P&L ≤ −2.5% → new positions sized ×0.50; ≤ −4% → ×0.25 (DAILY_LOSS_SIZE_TIER_ELEVATED/HIGH, both in GUARD_REGISTRY).

If you see a consecutive losses - pause line in journals dated after the #366 deploy, that is a defect — the code path no longer exists.

Daily Checks

• [ ] Dashboard shows backfill processes as "Running"
• [ ] Trades backfill percentage increasing
• [ ] Price snapshots being collected (check /api/db-stats)
• [ ] No errors in logs

Weekly Checks

• [ ] Compare category coverage percentages
• [ ] Verify no new gaps appearing
• [ ] Check disk space: df -h
• [ ] Check memory: free -h

Log Files

Log retention (Finding #260/#274 + #380, Aug 21 2026)

Two mechanisms, both bounded to 14 days:

1. logrotate (/etc/logrotate.d/polymarket, deploy twin deploy/logrotate/polymarket): covers /var/log/polymarket/*.log AND — since #380 — [hidden]/logs/*.log (the trader's actual append: target, which had grown to 2.33 GB unrotated). Daily + maxsize 500M early-trigger, 14 days compressed, copytruncate (required — append: fds are never reopened).
2. journald (/etc/systemd/journald.conf.d/polymarket.conf, deploy twin deploy/journald/polymarket.conf): SystemMaxUse=2G + MaxRetentionSec=14day (#380 — journald had reached 3.9 GB with no bound). Install per the deploy twin's header; journalctl --disk-usage to verify.

Convention (binding, #380): any new service unit that writes a file log (StandardOutput=append: or an in-app file handler) must have its path added to deploy/logrotate/polymarket in the SAME commit. A log-size claim is only verified with the claim's path pinned via the unit's StandardOutput= target.

Health Check Commands

# Check all processes
ps aux | grep -E 'gamma_collector|backfill|uvicorn' | grep -v grep

# Check dashboard API
curl -s http://localhost:8080/api/health | python3 -m json.tool

# Check database connectivity
python3 << 'EOF'
from sqlalchemy import create_engine, text
import os, ssl
with open('.env') as f:
    for line in f:
        if line.startswith('DATABASE_URL='):
            db_url = line.split('=', 1)[1].strip().replace('?ssl=true', '')
            break
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
engine = create_engine(db_url, connect_args={'ssl_context': ssl_context})
with engine.connect() as conn:
    result = conn.execute(text('SELECT 1'))
    print('Database: OK')
EOF

# Check Polymarket data-api (replaces Dome — Finding #287/#288/#289 May 10, 2026)
curl -s "https://data-api.polymarket.com/trades?limit=1" \
  | python3 -m json.tool | head -20

# Check Polymarket Gamma API health
curl -s "https://gamma-api.polymarket.com/markets?limit=1" \
  | python3 -m json.tool | head -10

# Check Polymarket CLOB API health
curl -s "https://clob.polymarket.com/ok" | head -5

8.A Integrity Monitor (Apr 27, 2026 — Finding #258)

The integrity monitor (src/data_tracker/integrity_monitor.py) replaces the v2/v3 watchdogs and is the canonical source for "is the data pipeline healthy?" questions. See docs/DATA_INTEGRITY.md §2 for architecture.

Operator runbook

# Service status
systemctl is-active polymarket-integrity-monitor.service
systemctl status polymarket-integrity-monitor --no-pager | head -15

# Live status snapshot (no DB hit)
curl -s http://127.0.0.1:8080/api/integrity | jq '.summary'
# {
#   "total": N,     ← live rule count: GET /api/integrity → summary.total (src/data_tracker/integrity_monitor.py)
#                   (85 as of Sep 26, 2026 live probe; history: grew 28 → 34 → ~47 → ~51 with #383's archive rules)
#   "ok": 31,
#   "warn": 0,
#   "fail": 1,      ← polymarket-backup.service flagged (Apr 27 04:09 OOM kill, see Finding #267)
#   "unknown": 2
# }

# Direct DB query for non-OK checks
psql "$DATABASE_URL" -c "SELECT severity, status, check_name, message
FROM integrity_status WHERE status != 'ok'
ORDER BY severity, check_name"

# Live tail
tail -f /var/log/polymarket/integrity_monitor.log

# Force a single cycle (ad-hoc / CI)
sudo -E [hidden]/venv/bin/python \
  [hidden]/src/data_tracker/integrity_monitor.py --once

# Add a new rule: edit `_build_rules()` in integrity_monitor.py, append a
# Rule(name, category, severity, description, evaluator). On next deploy
# the new rule appears in /api/integrity automatically.

What "unknown" status means

Some NULL-rate rules window-scope their population (e.g. "trade_audit_log.conviction_at_exit on rows where exit_time > '2026-04-26 17:38:00+00'"). When that window has fewer than min_rows (typically 2-5) the rule returns status='unknown' rather than alerting on too-small batches. This is by design — unknown is silent, never sends Telegram.

When to expect Telegram alerts

• Monitor detects freshness:price_snapshots is fail (last write > 10 min) → CRITICAL Telegram fires within 5 minutes (one cycle).
• Monitor sees systemd:earnings-trader.service is inactive → CRITICAL Telegram fires within 5 minutes.
• Monitor sees null_rate:trade_audit_log:conviction_at_exit_post_fix go from unknown → fail (i.e. enough rows post-fix have NULL despite the Finding #237 writer fix being deployed) → HIGH Telegram fires.

Decommissioned

The v2 and v3 watchdog units (polymarket-watchdog.service and polymarket-watchdog-v3.service) were removed Apr 27, 2026. The data_gaps table is read-only legacy (all 804 rows tombstoned). Don't write new code that reads or writes either.

8.A1 Running a deploy — detached on the server (F-8, Sep 26, 2026)

Every deploy runs detached. Never run deploy_from_github.sh attached to an SSH session: on Sep 26, 2026 the client dropped mid-deploy and the remote script was killed right after the strategy regen (close-persist contract check, integrity-monitor restart, logrotate and the final dashboard restart + HTTP check never ran; the exit status was lost). From the operator machine:

bash deploy/remote_deploy.sh                    # start + poll until DEPLOY_EXIT
bash deploy/remote_deploy.sh --sha <40-hex>     # deploy exactly this reviewed commit
bash deploy/remote_deploy.sh --poll <id>        # resume if the local side dropped

• One pinned target (#476 Round 4): the unit fetches origin/master ONCE and resolves one commit (--sha must be on origin/master and a fast-forward of HEAD, else rc 7; a malformed --sha/--poll is rc 2). The crontab pre-flight reads that commit, the checkout moves with git merge --ff-only <sha> (no second fetch or pull), and deploy_from_github.sh gets DEPLOY_TARGET_SHA and stops before any gate unless HEAD equals it. A push landing mid-deploy is not deployed. Use --sha with the approved SHA when master may move.
• The script prints the unit name and the resume command first, then runs git fetch origin master → resolve target → crontab pre-flight → git merge --ff-only <target> && DEPLOY_TARGET_SHA=<target> bash deploy/deploy_from_github.sh as systemd-run --unit=polymarket-deploy-<id> --collect (<id> = UTC timestamp + random suffix; owned by PID 1, survives any client disconnect). The unit's output goes to /var/log/polymarket/deploy-<id>.log, which ends with DEPLOY_EXIT rc=N.
• One deploy at a time: the unit takes flock -n on /run/lock/polymarket-deploy.lock before git fetch and holds it for the whole deploy (children do not inherit the lock fd). A unit that cannot take it logs another deploy holds …, writes DEPLOY_EXIT rc=3 and touches nothing. The polymarket-deploy-* list check before the start is only a faster refusal.
• If the start call itself fails (e.g. the SSH channel drops during the handoff), the script checks that exact unit, its log and its journal: if it ran, it polls it; if not, it exits 6 (nothing deployed).
• Exit status = the deploy's rc. 3 = another deploy is running (refused, or the unit lost the lock). 4 = still running after the wait cap, or polls kept failing — the deploy continues on the server; resume with --poll <id>. 5 = the unit ended without a DEPLOY_EXIT line (read the log and journalctl -u polymarket-deploy-<id>). 6 = the start failed and the unit never ran.
• By hand on the server (same thing): systemctl is-active polymarket-deploy-<id>; grep DEPLOY_EXIT /var/log/polymarket/deploy-<id>.log.
• The deploy restarts the trader, dashboard, collector, live-trades, websocket and (F-7) earnings-conviction-tracker — the tracker only after conviction_tracker.py --check-schema (read-only catalog check) passes; a missing conviction_history column/index stops the deploy before any restart. Never the Gateway, Xvfb or x11vnc.

8.A2 Prod checkout: never run a broad git clean (Finding #468, Sep 26, 2026)

[hidden] holds live runtime state next to the code. Never run git clean in any form on prod: not -fd, not -x / -X, and not a -n dry run followed by the real one. Depending on the flags it would destroy:

Since #468 these paths (plus logrotate outputs logs/*.log.*, the generated scripts/phase3_* outputs, the SEC cache and .env.bak.*) are in .gitignore, so git status on prod stays clean. That makes git clean -x / -X the dangerous forms: they delete ignored files, i.e. exactly the state above. Plain git clean -fd deletes untracked files, which is how this state was exposed before #468.

Untracked files that do not belong in the checkout are moved, never cleaned, and each move is logged in the ops log:

• stale logs, archived state dumps, scratch copies → compressed archive or copy under /var/lib/polymarket/archive/;
• secret-bearing files (.env.bak.*) → a root-only directory (0700, files 0600) outside the checkout, deleted after the credential rotation that covers them;
• source-like files (e.g. an untracked strategies/ tree) → out of the checkout intact, then reviewed and committed through a normal branch. Never ignore source-like files: that hides drift.

Check before a move: git -C [hidden] status --porcelain lists the untracked residue; re-sample ls -l /proc/*/fd 2>/dev/null | grep [hidden] so nothing open is moved. The deploy's own pre-pull hygiene (git_hygiene_cleanup, Finding #303) removes only untracked files byte-identical to origin/master and never touches ignored files.

8.A3 Trader alert policy — expected IBKR events, one page per incident (Finding #451)

Code: src/earnings/alert_policy.py over B1's shared src/earnings/incident_ledger.py. The trader pages an incident once, when its deadline passes and it is still open; the resolve pages only if the incident page was delivered — Telegram answered HTTP 200 for it (a webhook-only success or a filtered alert is not delivered). Every trader page, operator action and resolve is sent by the trader's alert-dispatcher thread (on every change and every 15 s), so a failed send is retried until delivered; a failed send never arms an AlertManager cooldown, and cooldowns are scoped to the incident generation + purpose, so a new incident pages at once.

• Page text = information (operator decision Sep 27 2026: the trader self-heals; automatic Gateway restart stays on as a last resort): every trader IBKR page says what was detected, what the bot did automatically and how the outcome is reported (RECONNECTED, or a page if automatic recovery fails). :4001 up → reconnect in place (IB() reset, one attach-bounce #441), then the automatic last-resort Gateway restart (#438). :4001 closed → the trader keeps retrying and attaches when the port is back; a login completes once IBKR's 2FA push is approved. The manual lever appears only as "Fallback only: systemctl restart ibgateway-ibc → approve the 2FA push → the trader attaches itself" — never "restart the trader" alone while :4001 is down.
• Lifecycle: STARTUP / SHUTDOWN are silent when logs/planned_restart.json is fresh (≤15 min): written by the #441 attach-bounce (bounce_trader) and at SHUTDOWN while a polymarket-deploy-* unit is active. Otherwise unchanged (WARNING; ERROR when shutting down with open positions).
• Persistence: logs/alert_incidents.json — the trader's IncidentLedger (incidents with generation / page / escalations, resolves = owed resolve pages); logs/alert_deliveries.json — page texts, the last resolved generation and the OD6 delivery record. Both git-ignored. The #441 bounce's new process keeps the old deadline and generation, never re-pages a delivered page, and keeps retrying owed pages / resolves. If a file cannot be locked, read or written — or disappears — the process keeps its own copy and still pages (only cross-restart dedupe degrades).
• OD6: integrity trader:ibkr_request_path keeps evaluating and showing; its Telegram page is skipped (recorded as handled in the integrity ledger, no retry) only when logs/alert_deliveries.json deliveries["ibkr:<down|up>"] shows, within 30 min and for the same incident generation (down: the open incident; up: the last resolved one), a delivered trader page or handled_by: ibkr-monitor — for down only after the monitor's attested DOWN was delivered, for up only after the heartbeat's last_resolved shows the same monitor generation with resolve: delivered. Every DOWN / RESTORED send for a physical IBKR outage — trader, integrity or ibkr-monitor — first takes a claim in logs/alert_arbiter.json (one outage record: members, down, up; fcntl lock): only the claimant sends, a failed send releases it, a claim older than 150 s is taken over (ALERT_ARBITER_CLAIM_TTL_SEC, valid 120–180 s; anything else is ignored with an ERROR log and flagged by the guard-override rule). Operator actions (the 2FA phone step, a recycle that did not run, a recycle on hold) are action:<kind> phases of the same outage: each pages at most once per outage, never as a second DOWN. An open outage is never pruned; if the file is lost or reset mid-outage, the pagers' RESTORED claims share one reconstructed record (one RESTORED), and a DOWN claimed after the loss may repeat once. Inspect: python3 -m json.tool [hidden]/logs/alert_arbiter.json.
• Inspect: python3 -m json.tool [hidden]/logs/alert_incidents.json (and alert_deliveries.json); trader log lines alert_policy: dispatcher started, incident … opened, … resolved … without a delivered page — no resolve page, … down telegram_delivered=, ibkr-monitor is live — its gateway:live incident pages this outage.
• Clear a stuck incident (only if the trader is stopped or the condition is verified gone): remove the key from incidents (and any resolves entry) in alert_incidents.json, or delete both files; the next event reopens it.

8.B Post-deploy verification checklist (Apr 27, 2026 — Finding #262)

After ANY deploy that touches earnings_dashboard/app.py or earnings_dashboard/templates/dashboard.html, smoke-test:

# 1. Dashboard service active?
ssh root@[hidden] 'systemctl is-active polymarket-dashboard'

# 2. GET / returns 200? (NOT just /api/health — Jinja can fail silently)
ssh root@[hidden] 'curl -s -o /dev/null -w "GET / HTTP %{http_code}\n" http://127.0.0.1:8080/'

# 3. /api/integrity still serves?
ssh root@[hidden] 'curl -s http://127.0.0.1:8080/api/integrity | jq .summary'

If GET / returns 500 with Jinja UndefinedError, see Finding #262 — most likely cause is stale in-memory Jinja compiled-template cache from a previous process state. Force-restart with systemctl restart polymarket-dashboard.

deploy/deploy_from_github.sh now does this verification automatically at the end of every deploy and exits non-zero if GET / doesn't return 200. So a green deploy is now sufficient for "dashboard is healthy".

8.C Service exit-code semantics (Apr 27, 2026 — Finding #251)

Six services were updated to exit non-zero on per-script-defined failure thresholds so systemd correctly surfaces failures. Before this audit, several services exited 0 even when the run failed substantively, hiding issues from systemctl is-active.

systemctl is-active <unit> is the canonical "did this run succeed?" probe post-Phase 1 — but only while the unit is still in failed SubState. Once the next timer fires the unit flips through active(running) → inactive(dead) and the previous failure is hidden. The integrity monitor's systemd_last_run:<unit> rules (Finding #268, Apr 27 evening) read Result + ExecMainStatus + ExecMainExitTimestamp so a unit's most- recent exit is monitored even when the unit is currently inactive between runs. This is the canonical "did the last batch run succeed?" probe; see docs/DATA_INTEGRITY.md §2.2.

Latent gap surfaced Apr 27 evening (Finding #267 + #268): polymarket-backup.service had been OOM-killed three nights running (Apr 25 / 26 / 27) before anyone noticed because no rule was watching the batch-service exit code. Phase 5's streaming pg_dump fix shipped Apr 27 13:43 UTC; first proven success Apr 28 04:00 UTC. Going forward, the new systemd_last_run:polymarket-backup.service rule fires a HIGH Telegram alert on any future OOM/kill/exit≠0.

Same gap class, second occurrence (Finding #370, Aug 21, 2026 — OPS-5): polymarket-trade-reconcile-full.service sat in failed state from Aug 1 to Aug 21 with zero alerts — it was never added to BATCH_SERVICES, AND its unit had no EnvironmentFile so its own Telegram alert path was silently disabled. Three defenses now exist: (1) both trade-reconcile units carry EnvironmentFile=[hidden]/.env and are installed by deploy_from_github.sh; (2) explicit systemd_last_run: + systemd_timer: rules for reconcile-full (monthly) and flex-reconcile (weekly); (3) the services:systemd_unit_health rule (CRITICAL) dynamically sweeps systemctl list-units --state=failed across ALL owned unit patterns (polymarket-*, earnings-trader*, etfstack-*, ibgateway-*, leverage-*) — a unit that nobody remembered to register can no longer fail invisibly. Rule of thumb: any unit whose script can send alerts MUST have EnvironmentFile — the DB layer has a private .env fallback but alerts.py reads os.environ only, so a missing env file breaks alerting and NOTHING else, which is the worst possible failure mode.

8.D Derived-artifact freshness (Apr 27, 2026)

8.E Healthchecks.io dead-man (Finding #448 / OD7)

External dead-man that still pages when Telegram or a local process is dead. SSOT for Python callers: src/healthchecks.py. Marius sets one secret in prod .env: HEALTHCHECKS_PING_KEY (project ping key). Never commit it, never log it, never echo the URL. URLs are slug-based: https://hc-ping.com/{HEALTHCHECKS_PING_KEY}/{slug}.

Off when unset: no HTTP, host script exits 0, backup/trader/integrity do not warn. Integrity rule config:healthchecks_ping_key is INFO / warn on the dashboard — not Telegram.

Pings never block the caller: daemon thread, urlopen timeout default 2 s (GUARD HEALTHCHECKS_PING_TIMEOUT_SEC 1–10). Errors log WARNING once per slug per HEALTHCHECKS_PING_MIN_INTERVAL_SEC (30–300, default 60), with the key redacted.

Deploy: deploy_from_github.sh installs the host unit+timer and enable --nows it. Does not restart ibgateway-ibc, xvfb, or x11vnc. Trader/integrity pick up the helper on their existing restart. Backup pings start on the next 05:00 UTC run.

Do not bash -x the host script (it would print the URL). Check journalctl -u polymarket-healthchecks-host.service and /var/log/polymarket/healthchecks-host.log if curl fails.

8.F Paging policy: delivery, incidents, Gateway pager, cron (Findings #448 / #449 / #456 / #476)

Delivered means one thing everywhere: Telegram answered HTTP 200 for that alert (DeliveryResult.delivered, src/earnings/alerts.py). A webhook success never counts. A channel without credentials is "not configured": it is not attempted, not retried and not a breaker failure. Successful sends log Alert delivered via Telegram at INFO, and a breaker-OPEN skip logs at WARNING. telegram-last-send.json (#449) records the latest attempt from Python and bash. It is diagnostics only and never changes a send's result.

Incidents (src/earnings/incident_ledger.py) are shared by ibkr-monitor and the integrity monitor. One incident = one page + one resolve. Each incident has a generation (its opened_at), and a delivery for an older generation never covers a new one. A failed page or resolve is retried on the next cycle. A new outage always pages: re-opening a key mints a new generation and deletes any resolve debt left from the last one. A resolve whose page was in flight when the sender died is sent after 120 s (after a restart too); an undeliverable resolve is deleted after 24 h. If the ledger file cannot be written, the process keeps its in-memory copy and a fallback copy in /run/polymarket, and still pages.

Live Gateway pager (src/ibkr/gateway_pager.py, run by ibkr-monitor.service). It never starts, stops, restarts or kills anything (OD5). It reads :4001 and the newest IBC log. "Waiting on 2FA" means the latest Second Factor Authentication initiated / Login attempt: line comes after the latest Login has completed and is less than 20 min old. "The weekly login" needs evidence: IBC's scheduled Re-starting session inside Sun 06:00–06:30 UTC, a real Second Factor Authentication initiated 0–10 min after it (a bare Login attempt: is not enough), no login completed since, and the outage opened within 10 min of the restart.

Every page is information first: Detected: (what the monitor saw), Automatic: (the trader attaches the moment :4001 is back, #441; its #438 last-resort recycle only acts while :4001 is up, so it cannot fix a closed port; the text follows IBKR_GATEWAY_AUTO_RECYCLE), then Fallback if it does not come back: the memory.md lever systemctl restart ibgateway-ibc → approve the push on the phone (every restart is a fresh login, #430) → wait for Login has completed + :4001 → the trader attaches itself. RESTORED says no action is needed. With a live page it takes a read-only screenshot of :99 (import -display :99 -window root) into /var/log/polymarket/ibkr-screens/. Paper :4002 never pages: its listener closes at the Sunday rotation until the 12:00 reconnect, which has no Telegram by etfstack operator preference. The heartbeat /var/log/polymarket/ibkr-monitor-heartbeat.json (0644) is rewritten every 30 s with live_incident (open outage: generation, deadline, page state) and last_resolved (latest closed outage: generation, resolve time, RESTORED delivered/owed/dropped). Integrity alerting:ibkr_monitor (HIGH) pages if the unit is inactive or the heartbeat is older than 5 min, and while :4001 is closed unless the heartbeat attests the page delivered or not yet due (then integrity is the fallback page).

Integrity alert policy (IntegrityMonitor.run_once): - Per-rule limiter: each rule sends under category data_integrity:<rule>. - Hysteresis: systemd:*, services:systemd_unit_health, artifact:ibkr_snapshot, dashboard:api_reachable and dashboard:db_errors need 2 failing cycles to page and 2 ok cycles to resolve (Rule.min_consecutive). - Deferral: openings are deferred, never dropped, while /run/polymarket/deploy-window.json is valid. The deploy writes it for 20 min before the first restart, re-stamps it to +5 min at the end, and anything longer than 1 h is ignored. Openings are also deferred in INTEGRITY_MAINTENANCE_WINDOWS (default Sun 01:30-03:00 UTC, DigitalOcean maintenance). A condition still failing after the window pages then. - UNKNOWN is no information: it neither opens nor resolves. - DB writes are best-effort: the page decision does not need a DB write, so db:read_only still pages during a read-only latch. - Monitor blind: if the database is unreachable, DB rules are skipped (UNKNOWN), systemd/artifact/meta rules still run, and integrity:meta:db_unreachable pages CRITICAL after 2 cycles. - First run: the first cycle with no ledger file adopts rules already failing in integrity_status as paged. They resolve once and never re-page. - Planned restarts: trader:restart_tripwire reports every trader start in 24 h (deploys, #441 bounces) as starts_24h and pages only on NRestarts > 0.

Crontab (#476): root's crontab is deploy/cron/root.crontab. The deploy runs deploy/install_crontab.py --install right after the host gates, BEFORE migrations and restarts, and stops on any non-zero rc. The install is a transaction: - one installer at a time (fcntl lock /run/lock/polymarket-crontab.lock; a second exits rc 2); - validate every live line first: exact known legacy lines are retired, managed lines kept, any other line preserved byte-exact under a "preserved host lines" block (then --check shows it as EXTRA until it moves into the repo), and any line that invokes a retired script in any shell form (shlex-tokenized: //, quotes, cd && ./, bash -c, ../) refuses with rc 3 and nothing changed; - a new 0600 backup /var/backups/polymarket/crontab.pre-deploy-<ts>-<random>, then a re-read (a concurrent edit aborts rc 2, nothing written), then an atomic crontab <file> swap, then a multiset re-check; - on an install or verification failure the backup is re-installed and verified (rc 2); if even that fails, rc 4 names the backup to reinstall by hand; - only after a verified install, /etc/cron.d/monitor_23_45 is moved aside, and only if it is still exactly the Jan 28 one-shot at move time; otherwise it stays with a WARNING and --check reports it.

Recycle budget lock (B1 Round 6): the trader's last-resort Gateway recycle holds an exclusive flock on [hidden]/logs/ibkr_last_recycle.json.lock from its guard check through the restart; the stamp [hidden]/logs/ibkr_last_recycle.json carries state (reserved → restarted) and a claim. Never edit or delete the stamp while the trader runs; a reserved stamp that never became restarted means a recycle died mid-way — the budget is kept on purpose (no second 2FA), and the fallback is systemctl restart ibgateway-ibc + approve the push. Stray ibkr_last_recycle.json.tmp.* files are abandoned writes and are safe to delete.

IBC soft restart (#484, flag default off): when IBKR_GATEWAY_SOFT_RESTART=true (dated # GUARD-OVERRIDE-UNTIL only — do not enable in the #484 commit), the same lock also covers IBC loopback RESTART on 127.0.0.1:7462 and the token-reuse wait. That path does not write the last-resort stamp. Own stamp logs/ibkr_last_soft_restart.json (≤3/UTC-day, 20 min; written before send). Refuses Monday 13:30–20:15 UTC and weekday entry 14:25–14:40 / exit 19:45–20:05 UTC. No IBC RESTART from Sunday 01:00 ET until /opt/ibgateway/Jts/launcher.log (IBKR_LAUNCHER_LOG_PATH; paper Jts-paper) or this week's launcher.YYYYMMDD.log sibling shows this week's first initialTokenType=PWD and an IBC log later shows Login has completed — last-resort systemctl restart ibgateway-ibc still may run. Monday 06:00 AutoRestart rotates the live file (prod 28 Sep 2026 06:33 UTC: Sunday PWD only in launcher.20260927.log; live was Monday TST). TST does not count. Dated enable: confirm week_token_alive_reason is None against those live logs after the weekly login. Never send IBC STOP. A RESTART already on the wire is may-have-restarted, never systemctl.

Release conditions for the first managed-crontab deploy (B1 review Round 5): root's crontab is FROZEN for the whole deploy (no crontab -e, no cron.d edits); the automatic pre-flight must report 0 preserve lines (any preserved operator line is reviewed before deploying: relocation after the managed block can change its environment); after the deploy venv/bin/python3 deploy/install_crontab.py --check must print OK (this also proves the Jan 28 one-shot was retired). Moving managed jobs to /etc/cron.d later removes the residual crontab -e race.

Pre-flight before the pull (automatic, #476 Round 3; pinned, Round 4): deploy/remote_deploy.sh fetches once (fatal on failure), resolves the target commit, extracts <sha>:deploy/install_crontab.py and <sha>:deploy/cron/root.crontab into a temp dir and runs --dry-run against the live crontab while the retired scripts still exist. Do not edit root's crontab while a deploy runs (crontab has no compare-and-swap; the installer never restores over a crontab that is neither the old one nor its own output, rc 4). Operator lines appear in its output only as [operator job '<schedule>' runs <program> … sha256:<12>]. Non-zero → "crontab pre-flight (#476) failed rc=… — nothing pulled, nothing restarted". The same check by hand, read-only: cd [hidden] && git fetch origin && git show origin/master:deploy/install_crontab.py > /tmp/ic-preflight.py && git show origin/master:deploy/cron/root.crontab > /tmp/rc-preflight && venv/bin/python3 /tmp/ic-preflight.py --dry-run --managed /tmp/rc-preflight → rc 0 required.

scripts/ibkr_schedule_audit.sh (cron 04:00 and integrity cron:schedule_audit, WARNING dashboard-only) runs the installer's --check. deploy_ibkr_monitoring.sh is a read-only check from the Mac.

9. Troubleshooting

Common Issues

Repeated 2FA Requests (Fixed Feb 3, 2026)

Symptom: Getting 2FA requests from IBKR every 5 minutes.

Root Cause: Multiple IBC services running simultaneously, causing session conflicts. Each IBC instance tries to log in, invalidating the other's session.

Solution (Applied Feb 3, 2026):

# Check which IBKR services are running
ssh root@[hidden] "systemctl list-units --type=service | grep -iE 'ibkr|ibc|gateway'"

# Only ibgateway-ibc.service should be active
# The conflicting services have been RENAMED to prevent accidental re-enabling:
#   /etc/systemd/system/ibgateway.service.disabled
#   /etc/systemd/system/polymarket-ibc.service.disabled

# Verify only one IBC process is running
ssh root@[hidden] "ps aux | grep ibcstart | grep -v grep"
# Should show only ONE ibcstart.sh process

Prevention: - The conflicting service files were renamed with .disabled suffix - Never create additional IBC services - only ibgateway-ibc.service manages the Gateway - If you see multiple ibcstart.sh processes, kill all and restart ibgateway-ibc.service

Stop Loss System (UPDATED Feb 7, 2026 - Phase 4D)

The stop loss system now has three layers for maximum reliability:

Layer 0: Real-Time Fill Detection (Phase 4D - Feb 7, 2026) - IBKR orderStatusEvent callback fires when ANY order fills - _on_ibkr_fill() detects stop fills for tracked positions immediately - Publishes STOP_FILLED event via EventBus - Queues position cleanup on main thread (thread-safe) - Solves CBOE/MCHP problem where stop fills were missed during downtime

Layer 1: Hardware Stop Orders (IBKR) - UPDATED Feb 10, 2026 (Finding #179) - STP LMT orders for ALL exchanges (not pure STP) - Pure STP gets IBKR Warning 2109 on NASDAQ (outsideRth IGNORED, won't trigger after hours) - STP LMT with 2% buffer respects outsideRth and triggers in extended hours - outsideRth=True - active during pre/post market - tif='GTC' - persists overnight until triggered or cancelled - Executes even if our service is down

Layer 2: Software Backup (Python) - Runs every second in main loop - Checks current price vs stop price for all positions - NEW (Finding #109): Before executing, verifies: 1. Symbol not already in _exit_in_progress (prevents retry loops) 2. IBKR position is not flat (prevents race with hardware stop) 3. Position direction matches expected (prevents selling into SHORT) - If all checks pass AND no hardware stop active, executes backup exit - NEW (Feb 15, 2026): If stop is breached and hardware stop remains unfilled beyond timeout, escalate to instant_exit:stop_timeout_escalation - Logs all activity for debugging

Race Condition Prevention (Feb 3, 2026 - Finding #109):

The PFE incident showed a critical race condition: 1. Hardware stop fills (position goes flat) 2. Software backup runs 2 seconds later 3. _check_hardware_stop_status() returns False (stop no longer in openTrades) 4. Software executes ANOTHER sell, creating SHORT position

Fix: Software backup now calls _get_ibkr_position_shares() before executing. If position is flat, it cleans up via _handle_hardware_stop_fill() instead of executing another exit.

Scheduled Exit Timing (Feb 3, 2026 - Finding #110):

Previously, exits triggered 2 minutes early due to a <= 120 threshold bug. The PEP incident on Feb 3, 2026 showed this caused: 1. Exit at 9:30 AM instead of 9:32 AM (now 10:32 AM) 2. DB update failure (pg8000 type mismatch with slippage floats) 3. Reconciliation overwrote correct data with wrong values

Fix: Changed check_scheduled_exits() threshold from 120s to 10s. Now exits trigger at the actual scheduled time (with 10-second grace for processing delays).

If stop loss fails: 1. Check journalctl -u earnings-trader for errors 2. Look for "HARDWARE STOP" entries showing order placement 3. Look for "SOFTWARE STOP BACKUP" entries showing price checks 4. Look for "EXIT VERIFICATION FAILED" - indicates position needs manual review 5. Look for "exit already in progress" - normal, prevents retry loops 6. Check IBKR Gateway was connected during the critical period

# Check for stop loss related logs
ssh root@[hidden] 'journalctl -u earnings-trader --since "1 hour ago" | grep -i "stop"'

# Check if hardware stop orders exist in IBKR
ssh root@[hidden] 'cd [hidden] && source venv/bin/activate && python3 -c "
from ib_insync import IB
ib = IB()
ib.connect(\"127.0.0.1\", 4001, clientId=998, timeout=10)
for trade in ib.openTrades():
    print(f\"{trade.contract.symbol}: {trade.order.orderType} @ {trade.order.auxPrice}\")
ib.disconnect()
"'

Earnings Timing Issues (BMO/AMC)

If a stock shows wrong timing (e.g., SBUX showing AMC when it should be BMO):

Root Cause: Timing data may be stale or company changed their release time.

Solution: Use the centralized timing system:

# Check current timing and source
python scripts/check_earnings_timing.py --ticker SBUX

# Fix a timing with override (persists across restarts)
python scripts/check_earnings_timing.py --set-override SBUX BMO "Changed to BMO per NASDAQ"

# Check all timings against live sources
python scripts/check_earnings_timing.py --auto-fix

# View all current overrides
python scripts/check_earnings_timing.py --show-overrides

Files involved: - src/earnings/earnings_timing.py - Single source of truth - src/earnings/timing_data/overrides.json - Persistent corrections

After fix: Restart the dashboard to apply changes:

ssh root@[hidden] 'pkill -f earnings_dashboard; cd [hidden] && nohup [hidden]/venv/bin/python -m earnings_dashboard.app > /var/log/polymarket/earnings_dashboard.log 2>&1 &'

10. Position Reconciliation (Added Jan 11, 2026)

Automatic Reconciliation

The trading orchestrator automatically reconciles DB positions with IBKR on startup.

CRITICAL UPDATE (Feb 15, 2026): Reconciliation uses guarded automation: - Default (RECONCILIATION_AUTO_CLOSE=false): detection + alerts only. - Optional (RECONCILIATION_AUTO_CLOSE=true): execution-ledger auto-close allowed only when strict guardrails pass (side, share, and timing consistency).

Why? The auto-fix feature caused data corruption in the PFE and PEP incidents: - PFE: Auto-synced 12 shares to 108 (from erroneous exit loop SHORT position) - PEP: Auto-closed as "orphaned" with wrong P&L ($7.72 instead of $1.68) because DB update was pending

Current Behavior: - Duplicates: DETECTED and ALERTED, requires manual fix - Orphaned positions (in DB but not IBKR): execution ledger checked first; auto-close only if guardrails pass and enabled - Share mismatches: DETECTED and ALERTED, requires manual fix - External IBKR positions (in IBKR but not DB): DETECTED and ALERTED for manual review

Guardrail failure behavior: auto-close is blocked and reconciliation falls back to detect-and-alert (no destructive write).

Reconnect/heartbeat hardening (Feb 2026): - Reconciliation now captures and logs a reconciliation context (heartbeat age, reconnect mode/attempts, last reconnect error). - If trader heartbeat/session is stale or reconnect is still in progress, reconciliation is skipped fail-closed to prevent false orphan/manual alerts. - Before escalating an orphan as manual-only, reconciliation now checks both: - IBKR reqExecutions (live broker history), and - local ibkr_executions ledger fallback (strict side/shares/post-entry guardrails). - High-confidence stop-fill evidence can auto-close safely even when global auto-close is OFF.

New config: - RECONCILIATION_MAX_HEARTBEAT_STALE_SEC (default 120): max allowed age of last successful trader heartbeat for reconciliation to proceed. - RECONCILIATION_HIGH_CONFIDENCE_AUTO_CLOSE (default true): allows auto-close for strict, high-confidence stop-fill matches even if RECONCILIATION_AUTO_CLOSE=false.

Close persistence hardening (AS/42P18 recurrence fix): - Primary close persistence now uses type-safe notes append semantics to avoid pg8000 nullable-parameter ambiguity (42P18). - If the rich close update fails, a fallback close path executes a minimal hard-typed update for both trading_positions and trade_audit_log. - This ensures broker-confirmed closes are not left open in DB due to secondary field persistence failures.

Manual orphan reconciliation protocol (DB source-of-truth consistency): 1. Verify live IBKR position truth first. 2. If confirmed absent in IBKR, close BOTH trading_positions and trade_audit_log in a single transaction. 3. Add explicit forensic note in trade_audit_log.notes documenting: - disconnect/reconnect evidence, - why manual reconciliation was required, - pricing/PnL confidence level. 4. Re-check /api/state (missing_in_ibkr must be empty) and /api/trader-status after update.

When you receive a reconciliation alert, check IBKR activity log to understand what actually happened before making any fixes:

# View IBKR activity log for actual fill prices
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && python3 -c \"
from ib_insync import IB
from datetime import datetime, timedelta
ib = IB()
ib.connect('127.0.0.1', 4001, clientId=998, timeout=10)
for exec in ib.executions():
    print(f'{exec.time}: {exec.contract.symbol} {exec.side} {exec.shares} @ \${exec.price:.2f}')
ib.disconnect()
\""

View reconciliation log to see what was detected:

# View reconciliation log
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && \
  psql \$DATABASE_URL -c \"SELECT run_time, db_positions_checked, ibkr_positions_found, \
  mismatches_found, orphaned_db_positions, external_ibkr_positions \
  FROM position_reconciliation_log ORDER BY run_time DESC LIMIT 10\""

Manual Position Reconciliation

If reconciliation finds issues, use these commands to investigate:

# Check DB open positions
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && \
  psql \$DATABASE_URL -c \"SELECT symbol, direction, shares, shares_fractional, status \
  FROM trading_positions WHERE status = 'open'\""

# Check IBKR positions
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && python3 -c \"
from ib_insync import IB
ib = IB()
ib.connect('127.0.0.1', 4001, clientId=998, timeout=10)
for p in ib.positions():
    if p.position != 0:
        print(f'{p.contract.symbol}: {p.position} shares @ \${p.avgCost:.2f}')
ib.disconnect()
\""

External Trade Resolution

When external trades are detected:

# View external trades table
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && \
  psql \$DATABASE_URL -c \"SELECT * FROM external_trades WHERE resolved = FALSE\""

# Mark external trade as resolved after investigation
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && \
  psql \$DATABASE_URL -c \"UPDATE external_trades SET resolved = TRUE, \
  resolved_at = NOW(), resolution_notes = 'Manual investigation complete' WHERE id = <ID>\""

Orphaned Position Resolution

If a DB position has no matching IBKR position:

# Mark position as closed (if confirmed closed in IBKR)
ssh root@[hidden] "cd [hidden] && source venv/bin/activate && \
  psql \$DATABASE_URL -c \"UPDATE trading_positions SET status = 'closed', \
  exit_reason = 'orphaned_reconciliation', reconciliation_status = 'orphaned' \
  WHERE symbol = '<SYMBOL>' AND status = 'open'\""

Exit Verification Failures

If exit verification failed (position not flat after exit order):

1. Check IBKR for remaining shares
2. Close manually if needed: bash ssh root@[hidden] "cd [hidden] && source venv/bin/activate && python3 -c \" from ib_insync import IB, Stock, MarketOrder ib = IB() ib.connect('127.0.0.1', 4001, clientId=999, timeout=10) contract = Stock('SYMBOL', 'SMART', 'USD') ib.qualifyContracts(contract) # For LONG: SELL, for SHORT: BUY order = MarketOrder('SELL', SHARES) trade = ib.placeOrder(contract, order) ib.sleep(5) print(f'Status: {trade.orderStatus.status}') ib.disconnect() \""
3. Update DB after manual close: bash psql $DATABASE_URL -c "UPDATE trading_positions SET status = 'closed', \ exit_reason = 'manual_close_verification_failed', exit_verified = TRUE \ WHERE symbol = '<SYMBOL>' AND status = 'open'"

11. Recovery Commands

# Full restart of all services
ssh root@[hidden]
cd [hidden]

# Kill all
pkill -f "python.*gamma_collector"
pkill -f "python.*backfill"
pkill uvicorn

# Wait
sleep 5

# Restart
source venv/bin/activate
nohup python3 -m src.data_tracker.collectors.gamma_collector --interval 60 \
  > /var/log/polymarket/gamma_collector.log 2>&1 &
nohup python -u src/data_tracker/collectors/backfill_trades.py \
  > /var/log/polymarket/trades_backfill.log 2>&1 &
nohup uvicorn earnings_dashboard.app:app --host 0.0.0.0 --port 8080 \
  > /var/log/polymarket/dashboard.log 2>&1 &

# Verify
sleep 3
ps aux | grep -E 'gamma_collector|backfill|uvicorn' | grep -v grep

12. CI Verification & Telegram Alerts

The CI workflow runs scripts/verify_retry_backoff_and_block.py on every push/PR.

What it verifies: - Collector/backfill retry + backoff behavior on 429/5xx - IBKR error blocking policy split (simulated, no live trading): - 1100 connectivity loss = transient, no persistent block file - 1300 socket/connection closure = persistent block file + trading blocked - IBKR disconnect alert path remains valid (send_alert-based, no stale async helper import)

Telegram alerts on failure: - GitHub repo secrets required: - TELEGRAM_BOT_TOKEN - TELEGRAM_CHAT_ID - Workflow file: .github/workflows/verify_retry_backoff.yml

Repo & Flow: - Repo: https://github.com/eumarius/earnings-trading-bot-.git - Flow: local machine → GitHub → server - Server update: bash deploy/remote_deploy.sh from the operator machine (runs [hidden]/deploy/deploy_from_github.sh detached; §8.A1)

12.1 Dashboard Connectivity Semantics (Finding #190)

Dashboard now shows two distinct broker signals: - Trader Session: readiness of the trading process session (clientId=1) used for order management. - Gateway: API/gateway reachability telemetry.

Operational interpretation: - Gateway=Connected + Trader Session=Reconnecting/Disconnected means trading is not ready yet. - Use /api/trader-status for trader session truth, and /api/ibkr-status for gateway telemetry. - /api/state includes trading_connectivity (gateway_state, trader_session_state, effective_trading_state).

12.2 Trader Stall Auto-Recovery + Exit Reason Guardrails (Feb 2026)

To prevent missed exits caused by a live-but-stuck trader process: - TradingOrchestrator now tracks main-loop progress and starts a watchdog thread. - If no progress is observed for MAIN_LOOP_STALL_TIMEOUT_SEC (default 120s), the process exits with failure so systemd auto-restarts it. - The budget is MAIN_LOOP_STALL_TIMEOUT_SEC, re-read every poll (history: #453's Monday-28-Sep 300 s dated override, retired by Finding #470). - Finding #470: the signal scan marks progress per ticker (stage=signals:<bmo|amc>:<SYMBOL>) and pumps queued IB owner work between tickers; the latest orderbook / price-snapshot reads are bounded to CONVICTION_LATEST_LOOKBACK_DAYS (7) and run under CONVICTION_QUERY_TIMEOUT_MS (15000). A stall writes an all-thread dump to MAIN_LOOP_STALL_DUMP_DIR ([hidden]/logs/stall_dumps/stall-<UTC>-<pid>.txt) before os._exit; the CRITICAL log line and page name the file. - Poll frequency is controlled by MAIN_LOOP_WATCHDOG_POLL_SEC (default 5s).

To prevent DB persist failures on long exit_reason values: - Exit reasons are normalized to fit VARCHAR(50) before writing trading_positions / trade_audit_log. - The full original reason is preserved in trade_audit_log.notes with prefix exit_reason_truncated:. - This avoids broker/DB drift caused by overlong reason strings during critical exits.

12.3 Leverage Phase A (Read-Only Telemetry)

Leverage implementation starts in read-only mode (no entry/exit behavior change): - Risk engine now computes leverage telemetry from broker state: - equity_base = NetLiquidation - PDT_RESERVE_CAPITAL - gross_exposure = sum(abs(position_market_value)) - effective_leverage, margin_utilization, liquidity_buffer - Dashboard /api/state now includes a leverage object for monitoring.

Phase-A flags (defaults shown): - ENABLE_LEVERAGE_ENGINE=false - ENABLE_MARGIN_BLOCKERS_ENFORCED=false - ENABLE_FINANCING_LEDGER=false - ENABLE_LEVERAGE_DASHBOARD_METRICS=true - LEVERAGE_MODE=regt_2x

Operational note: - Enforcement remains OFF until later phases. If leverage metrics are stale/unavailable, treat them as informational only during Phase A.

12.4 Leverage Phase B (Schema + Ledger Foundation)

Phase B is now in place as additive schema only (no behavioral change): - New migration: src/data_tracker/migrations/020_leverage_margin_financing.sql - trade_audit_log now includes nullable leverage/financing fields: - leverage_used, margin_required, margin_used, borrowed_amount, margin_account_type - margin_interest_rate_apr, margin_interest_days, margin_interest_cost - short_borrow_rate_apr, short_borrow_cost - financing_cost_total, net_pnl_after_financing - New tables: - margin_ledger for daily financing accrual rows - margin_events for margin-threshold events and safety actions

Operational notes: - Historical rows stay NULL by design (no backfill). - Entry persistence now writes leverage snapshot fields when available; financing costs remain unset until Phase C accrual engine is enabled.

12.5 Leverage Phase C (Daily Financing Ledger)

Phase C introduces financing accrual and close-time aggregation: - Daily accrual writes one row per open position to margin_ledger. - Close paths aggregate ledger totals into trade_audit_log: - margin_interest_days, margin_interest_cost, short_borrow_cost - financing_cost_total, net_pnl_after_financing - Aggregation applies across all close paths (normal exit and reconciliation auto-close).

Phase-C flags and defaults: - ENABLE_FINANCING_LEDGER=false - FINANCING_MARGIN_INTEREST_APR_DEFAULT=0.065 - FINANCING_SHORT_BORROW_APR_DEFAULT=0.030 - FINANCING_DAY_COUNT_BASIS=360 - FINANCING_ACCRUAL_HOUR_ET=20 - FINANCING_ACCRUAL_MINUTE_ET=5

Operational notes: - Accrual is idempotent by (position_id, accrual_date). - Runs even on holidays/weekends when enabled. - If disabled, close-time aggregation still works and safely resolves to zero financing when no ledger rows exist.

12.6 Leverage Rollout Status (A-E)

• Phase A: read-only leverage telemetry - deployed.
• Phase B: schema + ledger/event tables - deployed.
• Phase C: financing accrual + close-time net PnL - deployed in code, validate live before enabling enforcement.
• Phase D: pre-trade margin blockers (fail-closed) - deployed in code and flag-controlled.
• Phase E: reporting/alerts/docs hardening - deployed in code, live validation pending.

12.7 Leverage Phase D (Pre-Trade Margin Gates, Fail-Closed)

Phase D enforces margin safety at entry points only: - Entry paths (execute_entry + batch entry) now evaluate projected margin state before order placement. - If checks fail and enforcement is ON, entries are blocked and exits remain allowed. - Every block/shadow event is written to margin_events for forensic review.

Phase-D flags and defaults: - ENABLE_LEVERAGE_ENGINE=false - ENABLE_MARGIN_BLOCKERS_ENFORCED=false - LEVERAGE_MAX_EFFECTIVE_LEVERAGE=2.0 - LEVERAGE_MAX_MARGIN_UTILIZATION=0.65 - LEVERAGE_MIN_LIQUIDITY_BUFFER=0.25 - LEVERAGE_PROJECTED_MARGIN_REQ_PCT=0.50

Fail-closed conditions (when enforced): - stale or unavailable margin state - unsupported leverage mode - projected leverage/margin utilization/liquidity buffer threshold breaches

12.8 Leverage Phase E (Reporting + Alerting Hardening)

Phase E provides backward-compatible reporting and proactive leverage alerting: - Dashboard trading stats now use financing-aware PnL: - COALESCE(net_pnl_after_financing, realized_pnl) - Trade history rows include: - pnl_realized, pnl_after_financing, financing_cost_total - Dashboard state includes financing summary: - financing.today_cost, financing.week_cost, financing.open_position_daily_projection - Trader loop emits leverage alert categories and records matching margin_events: - margin_state_stale, leverage_cap_breach, margin_utilization_high, margin_headroom_low

12.9 Upcoming Earnings Dashboard Fail-Safe (Feb 2026)

If external earnings calendar providers degrade (for example NASDAQ API parse failures), dashboard upcoming rows now fail over to the authoritative earnings_markets table.

• Primary source remains EarningsCalendar for broad external enrichment.
• Automatic fail-safe path: if calendar returns zero rows but earnings_markets has upcoming data, dashboard/API builds upcoming list from DB.
• This prevents false "No upcoming earnings" UI states during upstream API outages.
• Main dashboard table now refreshes in-page from /api/earnings-calendar?days=14 every 30s and displays a live row count/timestamp status line.
• Server-render path also re-fetches once when cached earnings list is empty, preventing sticky empty-cache first-render states.
• API response now includes DB truth counters (db_upcoming_14d, db_upcoming_30d, db_next_upcoming_date) so UI can display backend state explicitly.
• Client refresh now keeps last known-good rows when transient API empty/timeouts occur, and marks the subtitle as warning instead of dropping to false empty-state.

Upload Files from Local

# From local machine
scp /Users/marius/projects/polymarket-bot/earnings_dashboard/app.py \
  root@[hidden]:[hidden]/earnings_dashboard/

scp /Users/marius/projects/polymarket-bot/src/data_tracker/collectors/*.py \
  root@[hidden]:[hidden]/src/data_tracker/collectors/

Appendix: Environment Variables

# .env file contents
DATABASE_URL=postgresql+pg8000://doadmin:[REDACTED]@HOST:25060/defaultdb

# DOME_API_KEY removed Finding #289 (May 10, 2026 — Dome was acquired by
# Polymarket and EOL'd Apr 28, 2026). All consumers now hit the public
# Polymarket APIs (gamma-api / data-api / clob.polymarket.com — no key required
# for public endpoints).

# Optional
TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_CHAT_ID=your_chat_id

# PDT reserve — dollars EXCLUDED from position sizing (trading capital =
# net_liquidation − this). FUND-1/#366 (Aug 20, 2026): 25000 → 17450, giving
# ~$10k trading capital at ~$27.5k equity. The FINRA PDT floor is TOTAL
# equity ≥ $25k, not this value. GUARD_REGISTRY floors it at ≥ 17000.
PDT_RESERVE_CAPITAL=17450

This document consolidates: OPERATIONS_GUIDE.md, INFRASTRUCTURE.md, CLOUD_DEPLOYMENT.md, BACKFILL_PLAN.md, DATABASE_GAP_ANALYSIS.md

12.10 IBKR Reconnect/Reconciliation Alert Hardening (Feb 2026 - Finding #196)

Root issue addressed: - Reconciliation callback could run while IBKR reconnect flag was still set. - Fail-closed reconciliation skip was correctly blocking writes, but was escalated as a critical position-sync error and created alert noise.

Hardening patch: - IBKRClient now clears reconnect flags (reconnect_in_progress=false, reconnect_mode=idle) before firing reconnect callbacks. - ReconciliationManager now classifies fail-closed guard skips as warning-level reconciliation_guard alerts (not critical position_sync). - Guard skips now trigger bounded retry-on-ready: - RECONCILIATION_GUARD_RETRY_DELAY_SEC (default 5) - RECONCILIATION_GUARD_MAX_RETRIES (default 3) - Guard alert noise was deduped by RECONCILIATION_GUARD_ALERT_COOLDOWN_SEC (history: 300 s repeat-page cooldown; retired by Finding #451 — the guard now pages once per incident after RECON_GUARD_PAGE_SLA_SEC (1800) or RECON_GUARD_OPEN_POSITION_SLA_SEC (300) with an open position; see the trader alert policy runbook). - Main loop watchdog now includes stage-aware forensics and thread dump output in logs before forced restart.

Validation notes: - Unit coverage verifies reconnect flags are idle before reconnect callbacks execute. - Unit coverage verifies guard scheduler dedupe/bounds and warning-category guard alert path. - Existing critical alerts for real mismatches/orphans remain unchanged.

Last updated: February 15, 2026

12.11 Conviction Window Robustness + Unified Reconciliation Close Path (Feb 2026 - Finding #197)

Root issues addressed: - Conviction fallback logic could use a single price_snapshots point in noisy windows, allowing outlier-driven direction errors. - Reconciliation high-confidence override was effectively stop-order-specific, causing recurring EXEC_LEDGER_ONLY warnings for broker-flat positions with strong execution-ledger evidence. - Trading orchestrator still had a legacy close-persistence implementation separate from the hardened PositionManager path.

Hardening patch: - ConvictionCalculator now computes robust conviction from a window of nearby price_snapshots: - median estimator over bounded sample set, - minimum sample gate, - instability gate via max window range, - reliability metadata (sample_count, min/max/range, reliability_reason). - SignalGenerator fail-closes when conviction reliability is false and logs the reliability reason for traceability. - TradingOrchestrator close persistence now routes through PositionManager.persist_position_close_with_exit_details(...) (single hardened implementation). - ReconciliationManager high-confidence execution-ledger override now supports non-stop exits under strict guardrails (shares/side/time + exec_id evidence). - Reconciliation execution-ledger auto-close now uses the hardened position-manager close path and then stamps reconciliation metadata.

Verification notes (local): - Unit tests passed: - tests/unit/test_conviction_robustness.py - tests/unit/test_signal_reliability_gate.py - tests/unit/test_reconciliation_guard_retry.py - tests/unit/test_close_path_routing.py - No linter errors on modified files.

Production evidence (Feb 26, 2026): - ✅ EXEC_LEDGER_ONLY for SNOW eliminated after deploy. Was recurring every 5 min pre-deploy; zero post-deploy. - ✅ Reconciliation close metadata complete: SNOW#75 and ICUI#62 repaired with full exit_order_type, exit_order_id, ibkr_exit_exec_id fields. - ✅ Reconciliation all-clear: "DB and IBKR are in sync" on every post-deploy cycle. - ⏳ Conviction robustness evidence blocked — no new entry candidates since deploy.

SNOW forensic detail: - SNOW instant exit at $166.45 succeeded in IBKR but DB persist failed with 42P18 (pre-deploy). - Reconciliation then flagged SNOW as EXEC_LEDGER_ONLY every 5 min. - After the d3f23b3 deploy, the high-confidence override would have resolved this automatically. - Data was manually repaired from log evidence: exit corrected from stop_loss@$170.85 to instant_exit@$166.45, PnL from -$9.96 to -$1.16.

Alert noise fix (commit 6cb8830): - CLOSED TRADE EXEC METADATA INCOMPLETE check now deduplicates by row ID set with 1h cooldown.

12.12 Direction-Aware Duplicate Prevention + IBKR Hours Parser (Mar 2026 - Finding #199)

Root issue addressed: - Scheduled exit market fallback was blocked by duplicate prevention logic that used abs(shares), making it direction-blind. For EXIT orders covering a short, finding shares means the exit DIDN'T work — but the check concluded "likely filled" and skipped the fallback. - IBKR market hours parser crashed on YYYYMMDD:HHMM-YYYYMMDD:HHMM format, blocking all post-failure retry attempts.

Incident: GPRO SHORT -397 shares scheduled exit at 3:57 PM ET on Mar 6. LIMIT BUY at $0.77 didn't fill. Market fallback skipped (inverted duplicate check). Position left open. Closed manually after hours at $0.7752 (+$111.08).

Hardening patch: - order_executor.py: Duplicate check now compares position direction (p.shares sign) against order side. - client.py: _parse_ibkr_hours strips date prefixes from time components. - batch_order_manager.py: Same direction-aware fix applied (Section E same-pattern check).