Decision Observability

Decision Observability (Phase 3, Task 1–11, observe-only)

Architecture boundary

arb_bot/decision_observability/ is import-safe: importing it never initializes a broker, loads tokens, connects to a database, starts a scheduler, or initializes an LLM. It has no import of arb_bot.client, arb_bot.token_manager, or arb_bot.execution.engine. Producer modules (the Council orchestrator, scanners, RiskEnvelope, GoLiveManager) emit typed events into this package after they have already made their own deterministic decision — observability wiring is strictly additive and never sits upstream of a trading decision.

Strict contracts (Task 1)

contracts.py defines DecisionEvent, DecisionEvidence, DecisionAlternative, DecisionContradiction, and DecisionRuleEvaluation — every model uses ConfigDict(extra="forbid", frozen=True), rejects non-finite numbers, requires upper-snake-case reason codes, and preserves None exactly rather than substituting zero. ids.py provides UUID-based trace_/evt_/run_ identifiers and a canonical-JSON content_hash that is stable across key ordering. sanitizer.py recursively rejects mapping keys matching a deny-list (token, secret, password, authorization, cookie, totp, api_key, access_key, refresh_key, session) and redacts Authorization/ Bearer values inside free-text error strings.

Append-only persistence and outbox (Task 2–3, corrected)

db/054_decision_observability.sql adds decision_traces, decision_events, decision_evidence, decision_event_evidence, decision_alternatives, decision_contradictions, decision_rule_evaluations, agent_workflow_runs/agent_workflow_steps, decision_outcomes/decision_counterfactuals, and decision_observability_outbox — all append-only, with (trace_id, sequence) uniqueness enforced by the repository's transactional sequence allocation. DecisionEventRepository.append_event rejects a parent_event_id that belongs to a different trace. A 2026-07-19 correction pass hardened three further gaps a review found in the original Task 2–3 implementation:

• Strict idempotency. append_event, append_evidence, and outbox.enqueue originally treated any reused event_id/ evidence_id/idempotency key as unconditionally idempotent — a genuine duplicate (safe retry) and a reused identifier carrying different content (a bug) were indistinguishable. All three now compare canonical business content and raise IdempotencyConflictError when a reused identifier's content actually differs, while still returning the persisted row silently for a true duplicate.
• Atomic dispatch. DecisionOutboxDispatcher originally committed the canonical event insert and the dispatched_at marker as two separate transactions — a crash between them left an event durably persisted with its outbox row still pending. The dispatcher now claims one row via FOR UPDATE SKIP LOCKED, appends the event, and marks it dispatched in one transaction with exactly one commit; it refuses to construct if the outbox and event repositories do not share one connection, since atomicity cannot be guaranteed across two.
• Sanitize-before-persist. outbox.enqueue originally persisted the raw event payload before sanitization. It now runs the payload through sanitize_payload before the row is ever inserted. A second independent review pass over the rebased branch found the same gap in append_evidence's value/metadata — dormant in practice (no production code constructs DecisionEvidence yet), but now closed the same way: sanitized before both the insert and the duplicate-idempotency comparison.

Council lineage (Task 4)

OptionsCouncilOrchestrator accepts an optional event_sink (defaulting to NullDecisionEventSink, so existing call sites are unaffected) and optional trace_id/run_id parameters on run(). It emits STARTED/COMPLETED/FAILED events for the canonical COUNCIL_CONTEXT → COUNCIL_DEBATE → COUNCIL_RISK → COUNCIL_MANAGER stage order — the exact four-call sequence, six-role ordering, and prompt/model/orchestrator pinning are unchanged; a sink exception is caught and discarded so a broken sink can never fail or falsely succeed a Council run.

Runtime wiring (2026-07-19 correction). The scheduled Council runner's _default_orchestrator_factory originally passed neither event_sink nor code_sha to the orchestrator, so every scheduled run silently used NullDecisionEventSink — Task 4's lineage code existed but was never durably recorded in production. Two config flags now gate this, defaulting to the same off-by-default posture as every other *_ENABLED flag in config.py: DECISION_OBSERVABILITY_ENABLED (default False) and DECISION_OBSERVABILITY_CODE_SHA (unset by default). When disabled, behavior is byte-for-byte unchanged. When enabled with a configured code SHA, the factory wires OutboxDecisionEventSink (council_sink.py), which enqueues Council stage events into the transactional outbox. When enabled without a configured code SHA, the factory logs a warning and falls back to the Null sink rather than persisting a misleading "unknown" code identity as if it were real. Neither the sink choice nor the code SHA can affect a Council decision.

Dispatch scheduling (2026-07-22 correction). OutboxDecisionEventSink.emit() enqueues a row into decision_observability_outbox (and upserts the parent decision_traces row) — it does not itself insert into decision_events. DecisionOutboxDispatcher.dispatch_once() drains an outbox row into decision_events and marks it dispatched. ArbitrageBot._setup_schedules() now registers _dispatch_decision_observability_outbox() on a DECISION_OBSERVABILITY_DISPATCH_INTERVAL_MIN-minute interval (default 5) whenever DECISION_OBSERVABILITY_ENABLED is true, tagged decision_observability_dispatch. Each tick opens its own PostgreSQL connection, drains up to 100 pending rows, and swallows any connection/dispatch failure into a warning log line rather than raising — a broken dispatch tick never affects the scan cycle, and undispatched rows are simply retried on the next tick. Before this correction, nothing in production ever called dispatch_once() outside tests, so outbox rows accumulated with dispatched_at IS NULL indefinitely and never reached the /api/options/v3/decision-traces query API.

Scanner evaluation (Task 5, DC scanner pilot)

Per ADR-3, DoubleCalendarScanner.evaluate() records every named entry-condition check (duplicate guard, DTE window, chain availability, ATM strike, leg security IDs, IV floor and ceiling, far-IV spread, near-straddle floor, realized range) in evaluation order, and scan() becomes return self.evaluate(open_trades).candidate — a characterization test proves the returned signal dict is byte-for-byte identical to the pre-refactor implementation. The remaining strategy scanners (DCS, DCS_SKEW, DDC, IC, MDC) and Decision Battle wiring are deferred to a follow-up increment; this page will be updated when they land.

Risk and mode policy evaluation (Task 6)

RiskEnvelope.evaluate() preserves the exact pre-refactor branch order (active-pause continuation → lifetime drawdown → daily kill switch → monthly drawdown → weekly drawdown → loss streak) and early-return semantics; check_envelope() is now a thin projection of evaluate()'s aggregate fields. A Pydantic model validator on RiskEnvelopeEvaluation proves the aggregate status is reconstructible purely from the ordered child checks. GoLiveManager.evaluate_mode() is strictly read-only — it never calls transition() or persists state — and reports the confidence gate and (when a balance is supplied) the margin gate as named checks.

Transition observability (2026-07-19 correction). evaluate_mode() only ever described the current mode — its requested_mode field was set equal to effective_mode, so it could never answer "why was a request to move from SHADOW to LIVE allowed or blocked?", and its fingerprint covered only the strategy name despite check_confidence()/ margin_ok() consulting several config thresholds and strategy aliases. GoLiveManager.evaluate_transition(strategy, requested_mode=...) is a new, equally read-only method that mirrors transition()'s exact eligibility branches (current-state guard, validation gate, confidence gate, AI-promotion gate for AI_MARKET) and reports ALLOWED/BLOCKED/NOOP/ INVALID_TRANSITION with the real block reason — never calling transition(), _save(), or _emit_transition(). Both methods' fingerprints now cover every consulted policy input (confidence thresholds, strategy aliases, requested mode, margin buffer only when margin was supplied) instead of just the strategy name.

Sanitized execution DTOs (Task 7)

ExecutionLegTrace and ExecutionEvaluation are safe DTOs assembled from already-normalized local values — never a raw broker SDK response. Because the schema itself has no field for tokens, account numbers, or auth headers, no secret-bearing value can survive construction. The 3,000+ line live ExecutionEngine is not yet wired to emit these DTOs; that wiring is deferred given its size and live-order-placement risk.

GET-only query API (Task 8)

GET /api/options/v3/decision-traces and GET /api/options/v3/decision-traces/{trace_id}/timeline are authenticated, GET-only, and read-only — no broker, LLM, or scanner call. Pagination uses an opaque base64 cursor encoding (started_at, trace_id), bounded to [1, 100] rows per page. The remaining seven routes from the specification (evidence, alternatives, rules, outcome, counterfactuals, scorecards, single-trace detail) are deferred to a follow-up increment.

Cockpit wiring (Task 9)

frontend/src/option_cockpit/api/contracts.ts and validators.ts mirror the two Task 8 response shapes with parse-or-throw validators (parseDecisionTraceListResponse, parseDecisionTraceTimelineResponse). ALLOWED_GET_ENDPOINTS gained /api/options/v3/decision-traces and a new ALLOWED_GET_ENDPOINT_PREFIXES list covers the dynamic /api/options/v3/decision-traces/ detail path — isAllowed() checks both. The visual Decision Explorer components (DecisionExplorerSection, DecisionTimeline) and Playwright coverage are deferred to a follow-up increment.

Outcomes, look-ahead boundary, and counterfactuals (Task 10)

DecisionOutcome is a versioned, append-only economics record — a recalculation inserts a new row with supersedes_outcome_id rather than mutating the prior one. ReplayDataset.validate_point_in_time() raises LookAheadViolation for any evidence or outcome timestamped after as_of. CounterfactualEvaluator.evaluate_no_trade() always returns a proven zero (data_quality="COMPLETE") — this is deliberately distinct from an UNAVAILABLE counterfactual for a strategy with insufficient historical pricing data, so a reviewer can never mistake "we know it would have been zero" for "we don't know." Deferred: outcome repository persistence, versioned-outcome integration with live traces, a general strategy counterfactual engine (beyond the no-trade case above), and a background outcome-scoring scheduler are not part of Task 10 — only the contracts, the look-ahead guard, and the no-trade counterfactual are implemented today.

Durable native workflow (Task 11)

Per ADR-4, NativeCouncilWorkflowEngine extends the existing bounded four-call orchestrator with durable step checkpoints instead of migrating to LangGraph. graph_signature() is a content-addressed hash of the workflow's shape (roles, prompt/orchestrator versions, model, authority, snapshot schema version) — it deliberately excludes transient fields like a generation timestamp. A step resumes only when its checkpointed graph_signature and (when supplied) input_hash both match the current request; any mismatch re-executes the stage from scratch. A provider failure raises WorkflowProviderError and is recorded as a FAILED step — there is no silent cross-provider failover. Status: prototype/scaffold, not runtime-wired. This engine is a standalone, fully unit-tested component backed only by InMemoryWorkflowRepository — there is no PostgreSQL-backed workflow persistence yet, and it is not called from OptionsCouncilOrchestrator or OptionsCouncilRunner. No scheduled Council run uses this engine today; describing it as "durable" refers to its checkpoint/resume design, not to any current production behavior.

Incomplete-lineage tracking (2026-07-19 correction)

BufferedDecisionEventSink.emit() returns False when its bounded queue is full, but no production call site previously did anything with that result — a dropped event never marked decision_traces.lineage_complete = FALSE, so a reviewer had no way to tell a trace with a silent gap from a genuinely complete one. The sink now accepts an optional lineage_registrar callback; DroppedTraceRegistry records a dropped event's trace_id in an in-memory bounded set (no blocking database write on the scanner/risk hot path) and a separate drain step persists it via the new DecisionEventRepository.mark_lineage_incomplete() — idempotent, and it can never flip a trace back to complete once marked.

Rollback

Every Task 1–11 addition, and every 2026-07-19 correction above, is purely additive: existing call sites (scan(), check_envelope(), OptionsCouncilOrchestrator.run() without a sink) are behavior-preserving compatibility wrappers, and the correction pass changed no public method's signature in a backward-incompatible way. Rollback disables emitters (pass no event_sink, leave DECISION_OBSERVABILITY_ENABLED at its default False) and the two v3 query routes; the append-only observability tables may remain dormant without affecting trading. No migration touches an existing table.