IBKR Continuity - Weekly 2FA Architecture

Version: 3.13 | Updated: Sep 27, 2026 | Status: Finding #484 IMPLEMENTED-PENDING-DEPLOY (flag default off; do not enable in this commit): IBC loopback RESTART can reuse the session token (heal-1234 paper P4 + live L7). Finding #454 DEPLOYED Sep 26, 2026 23:29 UTC (deploy #13 00607c9; soak pending: next 04:13 reset and 06:00 AutoRestart): paper 4002 proved consecutive reqCurrentTime pacing caused the 0/37 in-process recovery result; probes are now spaced 1.1 s on the owner loop, with reset/latch/heartbeat truth fixes. Finding #441 1102 no nested util.run; attach-bounce after IB() reset DEPLOYED Sep 9, 2026. Finding #440 same-thread req* after 2FA DEPLOYED Sep 7, 2026. Finding #439 one last-resort 2FA per week DEPLOYED Sep 5, 2026.

Last-resort systemctl restart ibgateway-ibc by the trader only after an IB() reset failed, with the #435/#439 age, cooldown, weekly-budget and Sunday 06:00–18:00 UTC guards and never while IBC awaits a 2FA (#438 automatic recycle, default on; operator decision Sat 26 Sep 2026 20:39 UTC, #456; IBKR_GATEWAY_AUTO_RECYCLE=false is the dated hold). Finding #484 (IBKR_GATEWAY_SOFT_RESTART, default false) can try IBC loopback RESTART (token reuse, no phone 2FA; Monday 13:30–20:15 UTC freeze and weekday entry/exit waves refuse) before that last-resort; do not enable the flag in the #484 commit. Every systemctl restart ibgateway-ibc (the trader's last resort or by hand) is a fresh IBKR login and needs a 2FA approval on the operator's phone (#430); only IBC's in-process 06:00 AutoRestart keeps the session (Mon–Sat). Watchers (ibkr-monitor, integrity) are page-only. Weekday IBC AutoRestart reattaches IB() in-process. Sunday 06:00 is the planned weekly 2FA. Farm sticky and young Gateway do not recycle. SecondFactorAuthenticationExitInterval=1800. Live config.ini is not in git.

In-process session recovery contract (#454)

• All ib_insync req* work runs on the thread that called connect(). Heartbeat/reconnect workers marshal jobs to that owner; a timed-out queued job is cancelled and may not run later.
• reqCurrentTime probes are spaced by at least 2.0 seconds with ib.sleep, based on the read-only paper-4002 reproduction plus the review margin. The durable sequence remains three request successes including qualify SPY.
• Error 1100/1102 does not reset the one-reset budget. Durable health clears it; an explicit weekday IBC reattach opens one reset attempt.
• An owner thread that did not pump is a failed heartbeat, not a healthy heartbeat. A failed fresh-IB() health check sets request_path_ok=false before any #441 trader attach-bounce.
• The 2FA watcher classifies authentication already in flight from the complete IBC log tail. It never routes a Sunday/2FA login through the weekday in-process branch.
• Recovery failure may bounce earnings-trader; it does not restart ibgateway-ibc. Weekday Gateway restarts remain prohibited.

IBC Command Server RESTART (#484, flag default off)

• IBKR_GATEWAY_SOFT_RESTART default false. true needs # GUARD-OVERRIDE-UNTIL. Do not enable in the #484 commit.
• When on: loopback RESTART to 127.0.0.1:7462 (paper 7463) inside the B1 recycle lock, before last-resort systemctl restart ibgateway-ibc. Token reuse does not spend the weekly 2FA budget.
• Refuse Monday 13:30–20:15 UTC, weekday entry 14:25–14:40 UTC, weekday exit 19:45–20:05 UTC, and daily AutoRestart 05:45–06:30 UTC. Saturday 20:00 UTC is not an exit wave. No IBC RESTART from Sunday 01:00 ET until launcher.log or this week's launcher.YYYYMMDD.log shows initialTokenType=PWD and an IBC log later shows Login has completed (DST-aware; last-resort systemctl still may run — that path is the PWD login). TST does not count. Monday 06:00 AutoRestart rotates the live file.
• Never send STOP/EXIT. An accepted or transmitted-unacked RESTART never falls through to systemctl. Soft stamp is written before the send.

Executive Summary

IBKR Gateway now requires 2FA only ONCE per week (Sundays at 10:00 AM Abu Dhabi).

Exception (Finding #443): a weekday 2FA DID happen whenever unattended-upgrades' needrestart restarted the live xvfb.service, because ibgateway-ibc has Requires=xvfb.service (Sep 1 ×2, 2, 11, 22, 25, 2026). #443 excludes the live xvfb.service/x11vnc.service and checks coverage at every deploy.

RESOLVED (v3.3 - Jan 28, 2026): Root cause identified and fixed: - Gateway had internal auto-logoff set to 23:45 UTC (11:45 PM) - IBKR's default - IBC's settings are now properly applied when IBC starts the Gateway - Verified: Gateway now shows "Restart in progress" at 06:00 instead of "Shutdown progress"

Architecture

Service Configuration

Key Scripts

Retired (#476, Sep 26, 2026): gateway_restart_manager.sh (killed Gateway PIDs), ibkr_health_monitor.sh (paging moved to ibkr-monitor), ibkr_restart_warning.sh and the 05:55 duplicate reminder, the uninstalled --critical line, and the never-in-git check_trader_health.sh / exit_reminder.sh crontab lines. All of them had been dead ("Permission denied" or "No such file") since Feb 15, 2026, apart from the manager.

Canonical schedule sources: cron times in deploy/cron/root.crontab (managed, installed by the deploy); the IBC restart time in [hidden]/scripts/ibkr_schedule.env

IBC Configuration (/opt/ibc/config.ini)

# Credentials
IbLoginId=<redacted — see /opt/ibc/config.ini on prod>
IbPassword=***
TradingMode=live

# Auto-restart at 06:00 UTC (10:00 AM Abu Dhabi)
# CRITICAL: Format must be HH:MM AM/PM (not 24-hour)
AutoRestart=yes
AutoRestartTime=06:00 AM

# 2FA Handling
SecondFactorAuthenticationTimeout=180
ReloginAfterSecondFactorAuthenticationTimeout=yes

# Session Management
ExistingSessionDetectedAction=primary
StoreSettingsOnServer=no

# API Settings
OverrideTwsApiPort=4001
AcceptIncomingConnectionAction=accept

How It Works

Monday-Saturday (No 2FA)

06:00 UTC (10:00 AM Abu Dhabi): IBC AutoRestart — "Re-starting session" in the same JVM
           └── "Login has completed" ~20-40 s later, no 2FA (#437)

06:00-06:02 UTC: the trader watcher requests an in-process IB() reattach; durable
           owner-thread health succeeds without a trader/Gateway restart (#454);
           one attach-bounce only if that fails (#441). If the token is rejected
           or a 2FA line appears, the 2FA attach path applies.

If :4001 stays down >180 s: ibkr-monitor pages once with the lever
           (systemctl restart ibgateway-ibc → approve push → Login has completed + :4001)

History (retired #476): gateway_restart_manager.sh ran at 05:58, looked for a second Gateway process and killed the newer one. IBC's AutoRestart is in-process, so there was never a second process (0 kills in 15 logged runs, Sep 10-26), and a pattern matching the IBC JVMs would have seen live + paper and could kill the live Gateway.

Sunday (Weekly 2FA)

05:50 UTC (09:50 AM Abu Dhabi): the one Telegram reminder

06:00 UTC (10:00 AM Abu Dhabi): Gateway's internal AutoRestart triggers
           ├── IBKR resets all session tokens (weekly)
           ├── IBC enters credentials automatically
           └── IBKR sends 2FA push notification

06:00-06:03 UTC: User approves 2FA on IBKR app (3 min window)

06:03+ UTC: Gateway connected, new session token valid for 7 days
            └── no message when it came back inside the wait (no page was sent)

06:20 UTC: still unapproved → ibkr-monitor pages once "2FA NOT APPROVED" (#456)
IBC hang with no 2FA (Sep 13/20 'Gateway' dialog) → page at ~06:03 with the restart lever

Overnight Shutdown Issue & Fix (v3.1 → v3.2)

Root Cause Discovered (Jan 27, 2026 - v3.1):

The AutoRestartTime in IBC config had wrong format: - Wrong: AutoRestartTime=06:00 (24-hour format - IGNORED by IBC) - Correct: AutoRestartTime=06:00 AM (12-hour format with AM/PM)

v3.1 Fix: Changed format to 06:00 AM.

ROOT CAUSE ANALYSIS (Jan 28, 2026 - v3.3):

The Real Problem: Gateway had two separate timer mechanisms:

Evidence from logs showing the difference:

# OLD behavior (before fix) - LOGOFF mode:
Jan 27 23:45:00 detected dialog entitled: Shutdown progress  # ❌ BAD - full shutdown

# NEW behavior (after fix) - RESTART mode:  
Jan 28 06:00:00 detected dialog entitled: Restart in progress  # ✅ GOOD - restart with token

Timeline of the Issue:

Jan 27 23:40:31 UTC: "Exit Session Setting" dialog appeared (5-min warning)
Jan 27 23:45:00 UTC: "Shutdown progress" - Gateway did AUTO LOGOFF (not restart!)
Jan 27 23:47:02 UTC: ibc_smart_restart.sh started sleeping until 05:00 UTC  # legacy wrapper (since removed; live unit now runs /opt/ibc/gatewaystart.sh -inline)
Jan 28 04:06:37 UTC: ibkr-monitor tried auto-recovery (conflict!)
Jan 28 04:21:55 UTC: IBC started Gateway with AutoRestart settings applied
Jan 28 05:55:30 UTC: "Exit Session Setting" dialog appeared (5-min warning)
Jan 28 06:00:00 UTC: "Restart in progress" - Gateway did AUTO RESTART ✅

Why It's Fixed Now:

When IBC starts the Gateway, it applies the AutoRestart=yes and AutoRestartTime=06:00 AM settings to the Gateway's internal configuration. This changes the Gateway from "Auto Logoff" mode to "Auto Restart" mode.

Exported Gateway settings confirm: autoLogoffTime="06:00,AM" (was previously 11:45 PM)

The Fix Has Three Components:

Monitoring: A cron job at 23:30 UTC will verify Gateway stays running tonight.

Verification: To confirm the fix is working, check the logs at 06:00 UTC:

# This command shows what happened at the restart time
journalctl -u ibgateway-ibc.service --since "06:00" --until "06:10" | grep -E "Restart|Shutdown"

# GOOD output: "Restart in progress"
# BAD output: "Shutdown progress"

Fallback (if shutdown still occurs) — HISTORICAL (v3.2 era): the legacy ibc_smart_restart.sh wrapper delayed overnight restarts as shown below. That wrapper has since been removed — the live ibgateway-ibc.service now runs /opt/ibc/gatewaystart.sh -inline directly and relies on IBKR's internal 06:00 AutoRestart (managed by IBC). The narrative is preserved for context only:

23:45 UTC (if Gateway still shuts down): IBC exits
           ├── systemd triggers Restart=always
           └── ibc_smart_restart.sh starts  # legacy — removed

23:47 UTC: Smart wrapper detects hour is outside 05:00-22:00 UTC
           ├── Sends Telegram: "Will restart at 06:00 UTC (10:00 AM Abu Dhabi)"
           └── Sleeps until 05:00 UTC

05:00 UTC (09:00 AM Abu Dhabi): Wrapper wakes up, starts IBC
           └── 2FA notification sent

ibkr-monitor off-hours handling (v3.2 — HISTORICAL; references the now-removed ibc_smart_restart.sh):

# In gateway_monitor.py - _attempt_auto_recovery()
if self._is_off_hours():  # 22:00-05:00 UTC
    logger.info("⏰ Off-hours - deferring to ibc_smart_restart.sh")
    return False  # Don't attempt recovery

if self._is_ibc_smart_restart_sleeping():
    logger.info("🔄 ibc_smart_restart.sh is handling restart")
    return False  # Don't interfere

Crontab Configuration

The root crontab is deploy/cron/root.crontab (Finding #476), installed by deploy/deploy_from_github.sh via deploy/install_crontab.py --install. The installer backs the old crontab up to /var/backups/polymarket/, and it refuses (leaving the crontab unchanged) if the host has a line it does not know. Check it with venv/bin/python3 deploy/install_crontab.py --check or bash deploy_ibkr_monitoring.sh (read-only):

50 5 * * 0 /bin/bash [hidden]/scripts/sunday_2fa_reminder.sh >> /var/log/polymarket/sunday_2fa_reminder.log 2>&1
0 4 * * * /bin/bash [hidden]/scripts/ibkr_schedule_audit.sh >> /var/log/polymarket/ibkr_schedule_audit.log 2>&1

Scripts run through /bin/bash, so a lost exec bit cannot silence them again.

systemd Service Configuration

[Unit]
Description=IBC Gateway Controller (handles login/2FA)
After=network.target xvfb.service
Wants=xvfb.service

[Service]
Type=simple
User=root
Environment=DISPLAY=:99
WorkingDirectory=/opt/ibc

# Live unit runs the IBC entry-point directly (legacy ibc_smart_restart.sh removed)
ExecStart=/opt/ibc/gatewaystart.sh -inline

# Always restart - IBC + IBKR's internal 06:00 AutoRestart handle timing
Restart=always
RestartSec=120

# Prevent rapid restart loops
StartLimitIntervalSec=600
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

Troubleshooting

Gateway Shuts Down at 23:45 UTC (03:45 AM Abu Dhabi)

Cause: Gateway was in "Auto Logoff" mode instead of "Auto Restart" mode

Key Diagnostic: Check the log message at shutdown time: - Shutdown progress = ❌ AUTO LOGOFF mode (bad - full shutdown, requires 2FA on restart) - Restart in progress = ✅ AUTO RESTART mode (good - session token preserved)

Symptoms: - Logs show: detected dialog entitled: Shutdown progress (not "Restart in progress") - Gateway shuts down even though IBC has AutoRestartTime=06:00 AM - Smart wrapper sleeps until 05:00 UTC, requiring 2FA at 09:00 AM Abu Dhabi

Solution: IBC must start the Gateway to apply the AutoRestart settings. If Gateway was started manually or by another process, the settings may not be applied correctly. Ensure IBC is the only process managing the Gateway lifecycle.

Root Cause: IBC's AutoRestartTime does NOT configure Gateway's internal exit time. These are separate settings.

Solution: 1. Connect via VNC using RealVNC/TigerVNC Viewer to [hidden]:5901 (macOS Finder vnc:// Screen Sharing does NOT work with x11vnc). Note: Sunday 2FA is primary via IBKR Mobile phone-push; VNC is the fallback. 2. In Gateway: Configure → Lock and Exit 3. Set auto-restart to 06:00 AM or disable auto-logoff 4. Click OK/Apply

Gateway Stuck at Login After Overnight Restart

Cause (v3.2-era, historical): ibkr-monitor tried auto-recovery while the legacy ibc_smart_restart.sh wrapper was sleeping (wrapper now removed; live ibgateway-ibc.service runs /opt/ibc/gatewaystart.sh -inline)

Solution (v3.2): ibkr-monitor now checks for off-hours and defers to smart wrapper:

journalctl -u ibkr-monitor | grep "Off-hours\|deferring"

Gateway Shows "Attempt N: connection error"

Cause: Waiting for 2FA approval Action: Check phone for IB Key notification and approve

Gateway Shows "UNRECOGNIZED USERNAME OR PASSWORD"

Cause: Credentials incorrect or account locked Action: 1. Verify credentials at IBKR Portal 2. Update /opt/ibc/config.ini if password changed 3. Check for account lockout emails

Gateway Down During Market Hours

Immediate Actions: 1. Check phone for pending 2FA notification 2. VNC to server ([hidden]:5901) if login screen visible 3. Use IBKR Mobile app as fallback for position management

Session Invalidated (2FA Required Unexpectedly)

Common Causes: - Gateway process killed externally (kill, systemctl stop) - Logged in from IBKR Portal, TWS, or Mobile app (same username) - Server rebooted without IBC restart - IBKR's internal shutdown at ~23:45 UTC (handled by smart restart now)

Fix: Approve 2FA on phone, then investigate what invalidated the session

Emergency Procedures

Gateway Down, Need to Trade NOW

1. Use IBKR Mobile App
2. All order types available
3. Manage positions directly
4. Independent of Gateway

Manual Gateway Restart

# SSH to server
ssh root@[hidden]

# Restart IBC (will trigger 2FA!)
systemctl restart ibgateway-ibc.service

# Approve 2FA on phone within 3 minutes
# Monitor logs
journalctl -u ibgateway-ibc.service -f

Check Current Status

# Port check
ss -tlnp | grep 4001

# Service status
systemctl status ibgateway-ibc.service

# IBC Gateway log (legacy ibc_smart_restart.log removed)
tail -20 /var/log/polymarket/ibgateway-ibc.log

# Screenshot
DISPLAY=:99 import -window root /tmp/gateway.png

Historical Context

Previous Issues (Resolved)

Key Learning

IBC 3.19 and Gateway 10.37 have a UI mismatch - IBC cannot reliably detect the login dialog. The solution is:

1. Let IBC/Gateway handle the internal AutoRestart and re-login (preserves the session token; it is in-process)
2. Reattach the trader in-process after Login has completed and :4001 (#454)
3. Only on Sundays does IBKR force token expiry, requiring 2FA; classify a true 2FA login from the IBC log and take the attach-bounce path
4. (history, retired #476) gateway_restart_manager.sh killed "new" processes; never needed, now removed
5. Smart restart wrapper handles overnight shutdowns gracefully

Monitoring

Health Check Endpoints

Logs

Note (history, retired #476): ibkr_health_monitor.sh persisted health state outside /tmp and classified active 2FA windows as login_pending rather than healthy.

Last Updated: May 31, 2026 (mechanism corrected: legacy ibc_smart_restart.sh removed; live ibgateway-ibc.service runs /opt/ibc/gatewaystart.sh -inline)