IBKR Gateway Setup & Monitoring

Version: 2.0 | Last Updated: May 10, 2026 | Status: ✅ PRODUCTION (live) · ⏳ READY-TO-WIRE (paper, Phase 0 of ETF Stack Strategy)

What changed in v2.0 (May 10, 2026): Architecture sections rewritten to match the actual current production setup — the v1.x doc described a TigerVNC-based two-display setup (vncserver :1/:2) that no longer exists. Reality: the live Gateway runs in a single Xvfb :99 X server with x11vnc bridging to RFB port 5901. The paper Gateway is being added in this same version as a parallel Xvfb :100 + x11vnc → port 5902 setup, fully isolated and managed by IBC the same way live is.

Specific corrections vs v1.x: - Section 2 architecture diagram: TigerVNC :1/:2 ⇒ Xvfb :99/:100 + x11vnc - Section 3 VNC access: VNC password is in /root/.vnc/passwd (set Jan 5, 2026 via vncpasswd); it is NOT the IBKR account password as v1.x claimed - Section 3: macOS Finder Screen Sharing does not work with x11vnc — use RealVNC Viewer or TigerVNC Viewer instead (caught May 10, 2026 by operator) - Section 4: Gateway is IBC-managed, not manually launched. The configuration lives in /opt/ibc/config.ini (live) and /opt/ibc/config-paper.ini (paper), and the GUI settings page in the v1.x §4 is rarely-touched - New §8: paper Gateway setup runbook (template-with-placeholders pattern, credentials never enter git or chat)

Capital (live): trading capital = IBKR net_liquidation − Config.PDT_RESERVE_CAPITAL (default 17,450, FUND-1/#366 — SoT: src/earnings/config.py, docs/STRATEGY.md §7.40). Do not freeze net-liq in docs; Aug 21, 2026 deploy probe: NL $27,504.38 → TC $10,054.38 (FINDINGS #366). History: v1.1 claimed ~$27,450 / $25k reserve / ~$2,450 trading; the $25,000 deposit/reserve began Feb 2, 2026 (Finding #104), not at Jan 7 launch (~$2.7k balance, legacy #37).

Table of Contents

1. Overview
2. Server Architecture
3. VNC Access
4. Gateway Configuration
5. Monitoring
6. Troubleshooting
7. Daily Operations
8. Paper Gateway Setup Runbook

1. Overview

What's Running

The earnings trading bot connects to IBKR via IB Gateway running on the DigitalOcean server. As of v2.0, both live and paper Gateway instances coexist on the same host, each in its own X display managed by Xvfb + x11vnc + IBC.

Key Accounts

The original paper account DU…91 referenced in v1.x was never reconfirmed after the long live-only period; the actual paper account ID will be visible in /api/ibkr-status.paper.account_id once the paper Gateway is up.

2. Server Architecture

The architecture is two independent X-display stacks, one per Gateway, fully isolated. Each stack is a chain of three systemd services:

┌──────────────────────────────────────────────────────────────────────┐
│  DigitalOcean Server  [hidden]  (Ubuntu, ufw inactive)        │
│                                                                       │
│  ╔══════════════════════════════════════════════════════════════╗    │
│  ║  LIVE STACK (always-on)                                       ║    │
│  ║                                                                ║    │
│  ║  xvfb.service                                                  ║    │
│  ║    └─ /usr/bin/Xvfb :99 -screen 0 1024x768x24                 ║    │
│  ║       (creates /tmp/.X99-lock; X display socket)              ║    │
│  ║                                                                ║    │
│  ║  x11vnc.service  (After=xvfb.service Requires=xvfb.service)   ║    │
│  ║    └─ /usr/bin/x11vnc -display :99 -rfbport 5901              ║    │
│  ║       -rfbauth /root/.vnc/passwd                              ║    │
│  ║       (bridges X display :99 ⇄ TCP port 5901 with VNC auth)   ║    │
│  ║                                                                ║    │
│  ║  ibgateway-ibc.service  (DISPLAY=:99)                         ║    │
│  ║    └─ /opt/ibc/gatewaystart.sh -inline                        ║    │
│  ║       reads /opt/ibc/config.ini (live creds, TradingMode=live,║    │
│  ║       OverrideTwsApiPort=4001)                                ║    │
│  ║       └─ Java IB Gateway → listens 0.0.0.0:4001               ║    │
│  ║          → trader connects with client_id=1                   ║    │
│  ╚══════════════════════════════════════════════════════════════╝    │
│                                                                       │
│  ╔══════════════════════════════════════════════════════════════╗    │
│  ║  PAPER STACK (Phase 0 of ETF Stack — installed, not enabled) ║    │
│  ║                                                                ║    │
│  ║  xvfb-paper.service                                           ║    │
│  ║    └─ /usr/bin/Xvfb :100 -screen 0 1024x768x24                ║    │
│  ║                                                                ║    │
│  ║  x11vnc-paper.service  (After/Requires=xvfb-paper)            ║    │
│  ║    └─ /usr/bin/x11vnc -display :100 -rfbport 5902             ║    │
│  ║       -rfbauth /root/.vnc/passwd  (SAME password file as live)║    │
│  ║                                                                ║    │
│  ║  ibgateway-ibc-paper.service  (DISPLAY=:100)                  ║    │
│  ║    └─ /opt/ibc/gatewaystart-paper.sh -inline                  ║    │
│  ║       reads /opt/ibc/config-paper.ini (paper creds,           ║    │
│  ║       TradingMode=paper, OverrideTwsApiPort=4002)             ║    │
│  ║       uses TWS_SETTINGS_PATH=/opt/ibgateway/Jts-paper         ║    │
│  ║         (separate from live's settings — IBC requirement)     ║    │
│  ║       └─ Java IB Gateway → listens 0.0.0.0:4002               ║    │
│  ║          → etfstack connects with client_id=200               ║    │
│  ╚══════════════════════════════════════════════════════════════╝    │
│                                                                       │
│  ┌──────────────┐                                                    │
│  │ Operator     │ ─── RealVNC Viewer ────▶ 5901 (live GUI)           │
│  │ Mac          │ ─── RealVNC Viewer ────▶ 5902 (paper GUI)          │
│  │              │                                                    │
│  │              │ ─── ssh ──────────────▶ shell + log access         │
│  │              │                                                    │
│  │              │ ─── HTTPS browser ────▶ :8080 dashboard            │
│  └──────────────┘                                                    │
│                                                                       │
│  Trading processes (connect to IBKR via localhost only):              │
│    earnings-trader.service  (client_id=1)   → 127.0.0.1:4001 (live)  │
│    etfstack-execute.service (client_id=200) → 127.0.0.1:4002 (paper) │
│      [Phase 2+ — see strategies/etfstack/docs/ARCHITECTURE.md]       │
│                                                                       │
└──────────────────────────────────────────────────────────────────────┘

Key points about this architecture: - Both stacks are independent: a crash in one (e.g. paper Gateway hang) cannot affect the other - The shared /root/.vnc/passwd lets the operator use one VNC password for both displays - Both API ports (4001/4002) listen on 0.0.0.0 but the trader/etfstack always connect via 127.0.0.1; the cloud firewall (DigitalOcean) is the actual external boundary, NOT ufw (which is inactive) - IBC handles 2FA automatically via the ReloginAfterSecondFactorAuthenticationTimeout=yes + SecondFactorAuthenticationTimeout=180 settings; on first login (and on the daily auto-restart) the operator approves the push notification on their phone

3. VNC Access

⚠️ macOS Finder Screen Sharing does NOT work

x11vnc does not speak the Apple-Remote-Desktop variant that macOS's built-in Screen Sharing client expects. The handshake fails before login with a generic "Unable to communicate" error and nothing appears in /var/log/x11vnc.log (the connection never reaches RFB negotiation). This bit the operator on May 10, 2026 — confirmed it's a client-side incompatibility, server is healthy.

Use RealVNC Viewer or TigerVNC Viewer instead

VNC password

The password is stored at /root/.vnc/passwd (8 bytes — VNC's obfuscated format). It is NOT the IBKR account password (v1.x of this doc was wrong about that). The current value was set Jan 5, 2026 via vncpasswd. If lost, reset via SSH:

ssh root@[hidden]
x11vnc -storepasswd <newpass> /root/.vnc/passwd
systemctl restart x11vnc.service
# If paper stack is up too:
systemctl restart x11vnc-paper.service

The same password file is used by both x11vnc.service (live) and x11vnc-paper.service (paper) — one password unlocks both.

Connecting

Connection flow (RealVNC Viewer): 1. Open RealVNC Viewer 2. Enter [hidden]:5901 (live) or :5902 (paper) in the address bar 3. Enter the VNC password from /root/.vnc/passwd when prompted 4. You'll see the Gateway's Java window (or a black-screen X server if Gateway hasn't been started yet)

4. Gateway Configuration

The Gateway is IBC-managed — IBC (Interactive Brokers Controller) is a third-party automation wrapper that auto-logs-in, dismisses dialogs, handles 2FA, and auto-restarts. The operator does not normally touch the Gateway GUI directly except for first-time 2FA approval and emergency manual recovery.

Configuration files

Key IBC settings (both configs)

When the GUI matters

The Gateway GUI is normally hidden (window minimized via MinimizeMainWindow=yes), but the operator may need to interact with it via VNC in these cases:

• First-time login: approve 2FA push notification on phone (no GUI interaction needed)
• 2FA timeout: if the push notification was missed, IBC will restart and prompt again — operator approves on phone
• Settings drift: if the API checkbox somehow gets unchecked (rare; never observed in this project), reach via Configure → Settings → API → Settings; verify Enable ActiveX and Socket Clients=ON, Read-Only API=OFF

For the paper Gateway specifically, see §8 Paper Gateway Setup Runbook.

5. Monitoring

Dashboard Integration

The dashboard shows IBKR status in the top bar: - 🟢 Green dot = Connected - 🔴 Red dot = Disconnected - 🟡 Yellow dot = Unknown/Connecting

API Endpoint

curl http://[hidden]:8080/api/ibkr-status

Response:

{
  "status": {
    "live": {
      "state": "connected",
      "account_id": "U2…13",
      "net_liquidation": 27450.00,
      "buying_power": 2450.00,
      "open_positions": 0,
      "uptime_pct": 99.5
    },
    "paper": {
      "state": "connected",
      "account_id": "DU…91",
      "net_liquidation": 1001575.80,
      "open_positions": 0
    },
    "is_healthy": true,
    "alerts": []
  }
}

Health Checks

The Gateway Monitor (src/ibkr/gateway_monitor.py) provides: - Continuous health checks every 30 seconds - Alert generation when gateways disconnect - Uptime tracking - Account balance monitoring

6. Troubleshooting

Gateway Won't Start

1. Check the live display stack is running (Xvfb + x11vnc, NOT vncserver): bash ssh root@[hidden] 'systemctl status xvfb.service x11vnc.service'

2. Restart the display stack if needed (live display :99 → x11vnc on 5901): bash ssh root@[hidden] 'systemctl restart xvfb.service x11vnc.service'

3. Start Gateway (IBC-managed — do NOT launch ./ibgateway manually): bash # ibgateway-ibc.service runs /opt/ibc/gatewaystart.sh -inline on DISPLAY=:99 ssh root@[hidden] 'systemctl restart ibgateway-ibc.service'

Gateway Disconnected

1. Connect via VNC
2. Check if Gateway window is showing error
3. If logged out, re-login with credentials
4. Approve 2FA on phone

API Connection Refused

1. Check Gateway is running: bash ssh root@[hidden] 'ss -tlnp | grep -E "4001|4002"'

2. Verify Gateway API settings (via VNC)

3. Check firewall: bash ssh root@[hidden] 'ufw status | grep -E "4001|4002|5901|5902"'

7. Daily Operations

Morning Checklist

1. ☐ Check dashboard for IBKR status (green indicators)
2. ☐ Verify /api/ibkr-status shows connected
3. ☐ Confirm account balances match expected

If Disconnected

1. Sunday 2FA is primary via IBKR Mobile phone-push approval; VNC is the fallback — connect to the appropriate port (5901 for live, 5902 for paper)
2. IBC auto-enters credentials from /opt/ibc/config.ini (no inline credentials needed)
3. Approve 2FA notification on phone
4. Verify status turns green on dashboard

Weekly Maintenance

IBKR restarts Gateway for maintenance on: - Sunday nights (~midnight ET)

After maintenance: 1. Gateway may need manual re-login 2. Connect via VNC and verify connection 3. Approve 2FA if prompted

8. Paper Gateway Setup Runbook

First-time setup of the paper IB Gateway after a fresh deploy of the xvfb-paper / x11vnc-paper / ibgateway-ibc-paper units. After this runbook, the paper Gateway is permanent — it survives reboots, auto-restarts daily at 06:00 UTC (same as live), and is monitored via /api/ibkr-status.paper.

Prerequisites

• ✅ deploy/deploy_from_github.sh has been run with the paper unit files in repo (since May 10, 2026 commit; verify ls /etc/systemd/system/xvfb-paper.service)
• ✅ /opt/ibc/config-paper.ini exists with placeholder credentials (the deploy script copied it from deploy/ibc/config-paper.ini.template)
• ✅ /opt/ibc/gatewaystart-paper.sh exists and is executable (deploy script copied it from deploy/ibc/gatewaystart-paper.sh)
• ✅ Operator has paper IBKR credentials (separate DU* login)
• ✅ Operator has RealVNC Viewer (or equivalent) installed locally
• ✅ Operator knows the VNC password from /root/.vnc/passwd

Step 1 — Fill in paper credentials on prod (one-time)

ssh root@[hidden]
ls -la /opt/ibc/config-paper.ini   # should exist with 0600 perms
nano /opt/ibc/config-paper.ini

# Find these two lines:
#   IbLoginId=PAPER_USERNAME_HERE
#   IbPassword=PAPER_PASSWORD_HERE
# Replace with actual paper-account credentials. Save + exit.

# Verify perms are still 0600 (root-only-readable):
ls -la /opt/ibc/config-paper.ini

The gatewaystart-paper.sh script refuses to start if the placeholders are still present, so this step is a hard prerequisite for §3.

Step 2 — Start the three paper services in order

# Order matters: xvfb-paper must be up before x11vnc-paper or ibgateway-ibc-paper
# (declared via After=/Requires= in the unit files, but explicit start is clearer)

systemctl enable --now xvfb-paper.service
sleep 2
systemctl status xvfb-paper.service | grep Active   # expect: active (running)
ls /tmp/.X100-lock                                   # expect: file exists

systemctl enable --now x11vnc-paper.service
sleep 2
ss -tlnp | grep 5902                                 # expect: x11vnc listening on 5902

systemctl enable --now ibgateway-ibc-paper.service
sleep 5
systemctl status ibgateway-ibc-paper.service         # expect: active (running)
tail -20 /var/log/polymarket/ibgateway-ibc-paper.log # expect: "Logging in to IB Gateway..."

If ibgateway-ibc-paper.service exits immediately with Error: paper IBC process is already running — that means a previous start attempt is still in the process table; kill it first: pkill -f 'java.*config-paper.ini' && systemctl restart ibgateway-ibc-paper.service

Step 3 — VNC in to approve 2FA + verify Gateway window

RealVNC Viewer → connect to [hidden]:5902
Password: <contents of /root/.vnc/passwd>

You should see the Java IB Gateway window appear (may take 30-90s after the service start). When it shows the 2FA prompt: - Approve via IBKR Mobile push notification on your phone (preferred), OR - Type the SMS / hardware-token code into the GUI via VNC

After 2FA succeeds, the Gateway window will minimize itself (per the MinimizeMainWindow=yes setting). The X display is still running but the Java window is iconified — that's normal.

Step 4 — Verify API is up + dashboard sees it

ssh root@[hidden] "ss -tlnp | grep 4002"
# expect: LISTEN 0  50  *:4002  *:*  users:(("java",pid=...))

curl -s http://[hidden]:8080/api/ibkr-status | jq '.status.paper'
# expect: state="connected", account_id starts with "DU",
#         net_liquidation > 0, open_positions=0 (or whatever's there)

If /api/ibkr-status.paper.state shows disconnected more than 90s after Gateway start, check: - journalctl -u ibgateway-ibc-paper.service -n 50 for IBC error messages - tail -50 /var/log/polymarket/ibgateway-ibc-paper.log for Java errors - Via VNC: is the Gateway window still alive, or did it crash? Is there a blocking dialog?

Step 5 — Confirm daily auto-restart works (passive — wait for 06:00 UTC)

After 24h, verify the daily auto-restart cycle:

journalctl -u ibgateway-ibc-paper.service --since "06:00 UTC today" -n 100
# expect: a clean exit + restart sequence around 06:00 UTC

Unlike live, paper credentials do NOT require fresh 2FA on auto-restart (different IBKR security tier — confirmed empirically 2026-05-24 by Sunday reconnect smoke-test). This is what makes Step 6 (automated weekly reconnect) possible. If your account ever changes behavior here (e.g., IBKR enables 2FA on paper), the ibgateway-paper-weekly-reconnect.service smoke-test will fail and surface in /api/integrity within ~26h via the systemd_last_run rule + the dashboard integrity-fail banner within ≤30s of that.

Step 6 — Automated weekly reconnect (Sun 12:00 UTC)

Added 2026-05-24 (Finding etfstack-006) after the Sun 2026-05-17 → Mon 2026-05-26 silent failure: IBKR's weekly user-session rotation around Sun 06:00 UTC silently closed the paper API socket on port 4002 while leaving the Java process alive and systemd reporting active. etfstack-execute.service failed every weekday for ~9 days until manual recovery.

The ibgateway-paper-weekly-reconnect.{service,timer} pair (installed by deploy/deploy_from_github.sh, auto-enabled because no credentials needed) runs scripts/ibgateway_paper_weekly_reconnect.sh every Sun at 12:00 UTC. The script:

1. systemctl restart ibgateway-ibc-paper.service (unconditional — the failure mode shows a healthy-looking port that's actually about to fail, so a "check first" optimization would create false negatives)
2. Polls ss -tln for up to 120s until port 4002 LISTEN
3. Polls /api/ibkr-status.paper.state for up to 60s until connected
4. Exits 0 on success, 1 on failure → systemd_last_run integrity rule
5. dashboard banner surface failures within ~30s

Verify timer is active:

systemctl list-timers ibgateway-paper-weekly-reconnect.timer --no-pager
# expect: NEXT field = next Sun 12:00:00 UTC

journalctl -u ibgateway-paper-weekly-reconnect.service --since "last sunday" -n 50
# OR, for the structured log written by the script itself:
tail -30 /var/log/polymarket/ibgateway-paper-weekly-reconnect.log
# expect: "=== Paper Gateway weekly reconnect complete ===" each Sunday

Manually trigger (safe to run any time — full cycle is ~20s):

systemctl start ibgateway-paper-weekly-reconnect.service
echo "exit=$(systemctl show ibgateway-paper-weekly-reconnect.service --property=ExecMainStatus --value)"

If this Step 6 automation ever stops being viable (e.g., IBKR enables 2FA on paper accounts, breaking the unattended cycle), revert to the manual revival recipe in strategies/etfstack/docs/RUNBOOK.md §5 and disable the timer: systemctl disable --now ibgateway-paper-weekly-reconnect.timer.

Phase 2 wiring (after this runbook completes)

Once the paper Gateway is up and stable, the ETF Stack Strategy's execution_runner.py (Phase 2 deliverable) will connect on client_id=200 and start placing paper orders. See strategies/etfstack/docs/RUNBOOK.md and strategies/etfstack/docs/STRATEGY_SPEC.md §11 for the safety architecture.

Decommission / reset (if needed)

To wipe paper Gateway state and start over (e.g., to switch paper accounts):

systemctl stop ibgateway-ibc-paper.service x11vnc-paper.service xvfb-paper.service
systemctl disable ibgateway-ibc-paper.service x11vnc-paper.service xvfb-paper.service
rm -rf /opt/ibgateway/Jts-paper      # session state
rm -rf /opt/ibc/logs-paper            # IBC logs
rm /opt/ibc/config-paper.ini          # credentials
# Then re-run deploy/deploy_from_github.sh to re-install the template,
# refill credentials in /opt/ibc/config-paper.ini, and start over from §1.

Appendix A: Port Reference

ufw is currently inactive on this host — the actual external boundary is the DigitalOcean cloud firewall layer (managed via the DO web console / API), not Ubuntu's ufw.

Appendix B: File Locations

Last updated: May 10, 2026 — v2.0 architecture rewrite (TigerVNC :1/:2 was historical fiction; reality is Xvfb :99/:100 + x11vnc bridges) + paper Gateway setup runbook for the ETF Stack Strategy.