Documentation

Brokers

Unified Broker Platform

The brokers package is a single source of truth for broker connections, credentials, and token audit so the trading bot and (later) the portfolio sync share one account store instead of scattering secrets across .env and SQLite. Phase A1 introduced the store and Dhan trading; Phase A2c wires Zerodha (KiteConnect) as a fully tradeable second broker on the broker-agnostic TradingBroker contract. Phase A3 wires the SYNC side: SyncBroker adapters for Dhan/Zerodha holdings sync, replacing the portfolio-app mirror hack.

Database schema

Migration db/011_brokers_schema.sql creates the brokers Postgres schema with three tables:

brokers.broker_connections — one row per account. broker_name (DHAN | ZERODHA | …), account_id, credentials (Fernet-encrypted JSON), and role (TRADING | SYNC | TRADING+SYNC). A partial unique index guarantees at most one active trading account (roles TRADING / TRADING+SYNC).
brokers.broker_token_events — append-only audit of every token refresh attempt: connection_id, outcome (success | failed), source (renewal | regeneration | oauth | manual), error, obtained_at.
brokers.broker_sync_states — per-connection holdings-sync state (last_success_date, next_retry_at, failure_count, error fields). Reserved for the SYNC wiring in A3.

A second migration, db/012_zerodha_instruments.sql, adds brokers.zerodha_instruments — a broker-data cache (descriptor_key primary key → instrument_token + tradingsymbol, plus lot_size/tick_size) populated from the daily Kite NFO instruments dump and indexed on tradingsymbol.

The `arb_bot/brokers/` package

Module	Role
`brokers/__init__.py`	Registry. `get_active_broker()` builds the active trading broker from the first `TRADING`/`TRADING+SYNC` row — a `ZERODHA` row builds `ZerodhaTradingBroker`, a `DHAN` row builds `DhanTradingBroker`. The connection's `broker_name` is the source of truth (a ZERODHA connection is never built with the Dhan adapter); `Config.BROKER` is a compat selector when `broker_name` is blank. `get_broker(name)` looks one up by name. Raises `BrokerConfigError` for an unknown broker. `get_sync_broker(broker_name, account_id)` / `get_sync_brokers()` (A3) build `DhanSyncBroker`/`ZerodhaSyncBroker` from a SYNC-role row; return `None`/`[]` when none exists. IBKR/MO/CAS are not built here.
`brokers/base.py`	`BaseBroker` lifecycle base (holds the `connection`, exposes `name` and `refresh_client()`).
`brokers/capabilities.py`	`TradingBroker` / `SyncBroker` ABCs. `TradingBroker` declares the normalized trading surface — `place_order`, `cancel_order`, `get_order_status`, `get_positions`, `get_fund_limits`, `get_option_chain`, `get_expiry_list`, `get_ltp`, `margin_for_order`. Each concrete adapter adapts itself via `as_trading_broker()`. `SyncBroker` (A3) declares one method — `fetch_holdings(self) -> pandas.DataFrame` — read-only, no orders.
`brokers/instruments.py`	`InstrumentDescriptor` (A2b) — broker-neutral identity `(underlying, expiry, strike, option_type, exchange, product)` with a stable `.key` used to resolve broker-specific IDs; `from_key()` rehydrates it (the inverse of `.key`).
`brokers/models.py`	Normalized value objects shared by both adapters: `OrderRequest`, `OrderResult`, `OrderStatus`, `OptionChain`, `OptionRow`, `FundSnapshot`, `MarginSnapshot`, `PositionSnapshot`.
`brokers/credentials.py`	Fernet `encrypt()`/`decrypt()` using `BROKER_SECRET_KEY` (falls back to `PORTFOLIO_SECRET_KEY`). `BROKER_ALLOW_DEV_SECRET=1` generates an ephemeral key for local dev only.
`brokers/config_repo.py`	Data access. `BrokerConnectionInfo` dataclass; `list_connections()`, `get_connection()`, `get_active_trading_connection()`, `upsert_connection()`, `update_credentials()`, `log_token_event()`, and `persist_token()` (merges a patch into the encrypted creds then audits). `get_sync_connections()` / `get_sync_connection(broker_name, account_id)` (A3) select rows with role `SYNC` or `TRADING+SYNC` — the latter prefers an exact account match, then any SYNC row for that broker.
`brokers/errors.py`	`BrokerConfigError` and `TokenExpired`.
`brokers/tokens/base.py`	`TokenProvider` ABC — `is_valid()`, `refresh()`, `accept_request_token()` (OAuth brokers override this; Dhan does not).
`brokers/tokens/dhan.py`	`DhanTokenProvider` ports `arb_bot/token_manager.py`. Same renew-then-regenerate semantics (try `/v2/RenewToken` first, fall back to PIN+TOTP), but reads/writes creds via `config_repo`. `MAX_TOKEN_AGE_HOURS=23`, `RENEW_HOURS_BEFORE_EXPIRY=4`.
`brokers/zerodha/token.py`	`ZerodhaTokenProvider(TokenProvider)` — daily OAuth access-token lifecycle. Backend TOTP refresh (`/api/login` → `/api/twofa` → follow the redirect chain → capture `request_token` → `generate_session()`) and direct OAuth callback handling via `accept_request_token()`. Credentials are read/written through `config_repo` — never `Config`/`.env`. Refreshes before midnight IST (`REFRESH_BEFORE_MIDNIGHT_HOURS=4`); `accept_request_token()` completes OAuth from a request token.
`brokers/tokens/service.py`	`TokenService.refresh_due()` walks every connection and refreshes any token that is not `is_valid()`. `DhanTokenProvider` and `ZerodhaTokenProvider` are both registered, so either broker's token is refreshed; each broker is wrapped in try/except so one failure cannot crash the loop.
`brokers/zerodha/mapping.py`	`ZerodhaMapping` — Postgres cache (`brokers.zerodha_instruments`) mapping `InstrumentDescriptor.key` to Kite `instrument_token` + `tradingsymbol`. `populate_from_csv()` parses the daily NFO instruments dump (keeps `NFO-OPT` CE/PE rows for the underlying); `is_stale(max_age)` flags a stale/empty cache; `rows_for_expiry()` feeds option-chain construction.
`brokers/zerodha/client.py`	`ZerodhaTradingBroker(TradingBroker)` — descriptor-native (no legacy methods; `as_trading_broker()` → `self`). Resolves tradingsymbols via `ZerodhaMapping`. `get_ltp()` uses batched `kite.quote()` (max 500 instruments/call); `get_option_chain()` is built from batched quotes (IV left at `0.0` — the iv_sampler computes it) since Zerodha has no native chain API; `get_order_status()` maps Kite status strings → `OrderStatus`; `get_fund_limits()` reads the `kite.margins()` equity block.
`brokers/dhan/trading.py`	`DhanTradingBroker` — a superset of legacy `arb_bot/client.DhanClient`. Built from a `BrokerConnectionInfo` + `DhanTokenProvider` so credentials come from the DB. It keeps the legacy method bodies and a `_DhanTradingAdapter` that normalizes them onto the `TradingBroker` contract (via `as_trading_broker()`); `PART_TRADED_AND_CANCELLED` is mapped to `OrderStatus.TRADED`.
`brokers/dhan/sync.py`	`DhanSyncBroker(SyncBroker)` (A3), `name="dhan-sync"` — read-only holdings fetch via `/v2/holdings` and current-day tradebook fetch via `/v2/trades`. Creds (`client_id`, `access_token`) come from `self.connection.credentials`. Holdings return with `Portfolio` left blank — `ConnectionSyncService` stamps it, ensures the default portfolio row, and imports tradebook rows without recalculating broker-sourced holdings.
`brokers/zerodha/sync.py`	`ZerodhaSyncBroker(SyncBroker)` (A3), `name="zerodha-sync"` — read-only equity + mutual-fund holdings fetch via Kite Connect. Creds (`api_key`, `access_token`) come from `self.connection.credentials`. `Portfolio` left blank, same as Dhan.

Config knobs

The active broker is selected by Config.BROKER (env BROKER, default "dhan"); set BROKER=zerodha to trade via Zerodha. Zerodha rate-limit / caching knobs (all env-overridable): Config.ZERODHA_QUOTE_COOLDOWN (min seconds between kite.quote() batches), Config.ZERODHA_ORDER_DELAY_SEC (post-order sleep), and Config.ZERODHA_INSTRUMENT_REFRESH (max age in seconds before the instruments cache is considered stale). Credentials are never read from Config/.env for an active broker — they live Fernet-encrypted in brokers.broker_connections.

Bootstrap & `.env` fallback

ArbitrageBot builds its trading client via _construct_client() in arb_bot/bot.py, then resolves the normalized view through get_active_trading_broker():

get_active_broker() ├── ZERODHA connection → ZerodhaTradingBroker + ZerodhaTokenProvider ├── DHAN connection → DhanTradingBroker + DhanTokenProvider (unified path) │ blank broker_name → Config.BROKER selector ("dhan" is the safe default) └── no TRADING row → legacy DhanClient(TokenManager()) from .env (bootstrap/fallback — Dhan keeps trading)

To move an account onto the unified store, run the one-time seed script, which copies the live Dhan (or Zerodha) credentials into an encrypted brokers.broker_connections row:

python -m scripts.seed_broker_connection --broker DHAN --account-id $CLIENT_ID --role TRADING

Deleting that row (or running with none) restores the .env fallback — there is no go-live cliff.

Engine fill flow

ExecutionEngine._poll_fill() is now broker-agnostic. When a TradingBroker view exists it confirms fills through get_order_status() returning the OrderStatus enum (enum-to-enum comparisons); otherwise it falls back to the legacy Dhan-string API. Dhan's PART_TRADED_AND_CANCELLED (an IOC that partially fills, leaving a real position that must be tracked) is preserved — the Dhan adapter maps it to OrderStatus.TRADED so the engine treats the partial fill as filled (filled=true). The Zerodha adapter has no equivalent partial-IOC state.

Generalized token refresher

token_refresher.py is broker-agnostic. Its run_once() prefers the unified TokenService.refresh_due() whenever any broker_connections row exists; otherwise it falls back to the legacy TokenManager().refresh_if_due() for bootstrap. The standalone service loop (5-minute interval, 3-consecutive-failure Telegram alert) is unchanged. See Token Management for the renew/regenerate details.

Operator surfaces

Telegram /broker_status — lists every connection (broker, account id, role) with a live token-validity check via each broker's TokenProvider.is_valid(); Zerodha rows append the token date.
Telegram /refresh_zerodha [request_token] — with a request_token argument it completes OAuth via accept_request_token(); without an argument it triggers the auto-TOTP refresh (and posts the manual login URL to Telegram if TOTP fails). No-op when the active broker is Dhan.
Dashboard GET /zerodha/callback — unauthenticated Kite OAuth redirect target. Exchanges ?request_token=... for an access token via ZerodhaTokenProvider.accept_request_token() and persists it. No JWT is enforced because Kite cannot authenticate against our session; returns a generic error on failure.
Dashboard Broker connections card — BrokerStatusCard.tsx reads the authenticated /api/broker/status endpoint, which now emits a ZERODHA block (token_valid, token_date) alongside the Dhan row. /api/broker/health exposes the longer-running broker-health samples.
Portfolio Connections tab — creates and edits encrypted brokers.broker_connections rows directly. Operators choose the row role (SYNC, TRADING, or TRADING+SYNC). Dhan rows require access_token, pin, and totp_secret (with client_id derived from account id when omitted). Zerodha rows require api_key, api_secret, user_id, password, and totp_secret. access_token is optional on first setup because Zerodha generates it from OAuth; set the Kite app redirect URL to https://app.portfolioplanner.online/zerodha/callback.
Bot scheduling — when Config.BROKER is "zerodha" the bot schedules a Zerodha token refresh at 20:00 and 22:00 IST and runs a non-fatal startup validity check that refreshes an expired/due token before entering the loop. The Dhan schedule is unchanged.

Secrets stay encrypted, never in code. Broker credentials live Fernet-encrypted in brokers.broker_connections; the encryption key is BROKER_SECRET_KEY (or PORTFOLIO_SECRET_KEY), supplied via environment. The dashboard and /api/broker/status never return token values.

SYNC wiring (A3). Dhan/Zerodha portfolio-holdings sync now goes through SyncBroker adapters backed by brokers.broker_connections instead of a portfolio-local credential mirror plus standalone fetch_dhan_holdings / fetch_zerodha_holdings calls. The brokers.broker_connections row is now authoritative for Dhan/Zerodha SYNC creds — create/manage it via scripts.seed_broker_connection, the dashboard, or broker commands. The dashboard Connections tab, portfolio Telegram broker commands, trading startup, token refresh, and holdings sync all use brokers.broker_connections. IBKR/MO/CAS are stored there as SYNC rows too; their adapters remain read-only and portfolio-specific, but credentials live in the same broker store. arb_bot/client.py and arb_bot/token_manager.py stay as the bootstrap/.env fallback for trading until every operator has moved to a DB connection, after which they retire.

Dhan Statement sync

DhanSyncBroker imports holdings from /v2/holdings and portfolio transactions from the paginated Statement Trade History endpoint, /v2/trades/{from-date}/{to-date}/{page}. It carries ISIN/security id metadata so portfolio sync can normalize Dhan trade-history display names back to holding symbols. The current-day order tradebook endpoint /v2/trades is not sufficient for portfolio transaction backfill.

Dashboard broker connections

The Portfolio Connections tab writes directly to brokers.broker_connections.account_id is the routing key used by sync, so edits to Account ID are persisted through the connection update API. Dhan client_id and Zerodha user_id continue to be derived from Account ID when they were originally derived that way.

Dhan and Zerodha sync lookup now requires an exact Account ID match when syncing a specific connection; the sync path will not fall back to another same-broker row. Zerodha setup can be saved before an access_tokenexists, but holdings sync requires a generated access token from the backend refresh flow first. The dashboard refresh action uses backend refresh, stores a short-lived signed state for that exact connection, and the callback consumes that state so multiple Zerodha rows are handled independently.

IBKR setup is Activity Flex only: save query_id andtoken on the connection. Activity Flex is end-of-day statement data, so IBKR rows are treated as once-daily sync sources and repeated same-day syncs are skipped after a successful import. Configure the Flex query with both Open Positions andTrades for the required historical date range: the workspace imports long stock/ETF positions and stock/ETF trade rows; cash, options, shorts, and unsupported rows are intentionally skipped. A successful sync with no supported trades explicitly reports zero transactions. A transaction-fetch or transaction-persistence failure is reported as a failed sync (holdings already persisted by that run remain intact) so it cannot be mistaken for a complete ledger import. Failed automatic IBKR imports retry hourly until a sync succeeds.

Automatic portfolio sync is driven by the deployed token-refreshermaintenance service. The container still polls every five minutes for due token work, but Dhan and Zerodha holdings imports are gated to one run every 30 minutes per sync connection. Dhan token rotation keeps the existing 23-hour maximum token age; Zerodha token rotation uses the storedapi_key, api_secret, user_id,password, and totp_secret to complete backend OAuth refresh when the access token is due.