12.4 Systems Tuning, Hardware & Risk Gates

Let's look at the systems side of the runway: risk gates, reservations, recovery, rate limits, order and session state machines, CPU pinning, NUMA, networking choices, timestamps, and logging. This is where clever C++ still loses to bad operational design.

Think of pre-trade risk as a locked blast door. It must be fast, and when unsure it must fail closed. These checks are not optional strategy flavoring.

Common checks include:

• Maximum order quantity
• Maximum order notional
• Maximum position
• Maximum gross exposure
• Maximum open orders
• Price collars
• Order-rate limits
• Credit or margin limits
• Instrument trading status
• Kill-switch state
• Duplicate client-order-ID protection
• Session state and exchange connectivity

A simple quantity and notional check:

enum class RiskResult {
    allowed,
    kill_switch_active,
    invalid_quantity,
    quantity_limit_exceeded,
    notional_limit_exceeded,
    position_limit_exceeded,
    arithmetic_overflow
};

struct RiskLimits {
    std::int64_t max_order_quantity;
    std::int64_t max_order_notional_ticks;
    std::int64_t max_absolute_position;
};

struct RiskState {
    bool kill_switch_active{false};
    std::int64_t current_position{0};
};

bool multiplication_would_overflow(std::int64_t a, std::int64_t b) noexcept;

RiskResult check_order(
    const RiskLimits& limits,
    const RiskState& state,
    std::int64_t price_ticks,
    std::int64_t quantity,
    std::int64_t signed_quantity
) noexcept {
    if (state.kill_switch_active) {
        return RiskResult::kill_switch_active;
    }

    if (quantity <= 0) {
        return RiskResult::invalid_quantity;
    }

    if (quantity > limits.max_order_quantity) {
        return RiskResult::quantity_limit_exceeded;
    }

    if (price_ticks < 0 || multiplication_would_overflow(price_ticks, quantity)) {
        return RiskResult::arithmetic_overflow;
    }

    const auto notional = price_ticks * quantity;

    if (notional > limits.max_order_notional_ticks) {
        return RiskResult::notional_limit_exceeded;
    }

    const auto projected_position =
        state.current_position + signed_quantity;

    if (projected_position > limits.max_absolute_position ||
        projected_position < -limits.max_absolute_position) {
        return RiskResult::position_limit_exceeded;
    }

    return RiskResult::allowed;
}

This simplified example still omits important production concerns:

• Multi-currency valuation
• Partial fills and replace races
• Exchange acknowledgement delays
• Concurrent order paths
• Market-specific self-trade prevention
• Regulatory controls
• Persistence across restarts

The next sections cover outstanding exposure reservations. Keep the blast-door principle in mind:

A risk check must be authoritative, deterministic, observable, and unable to be silently bypassed.

A simple filled-position check is not enough for real pre-trade risk.

Multiple live orders can each look valid on their own while their combined exposure blows past the intended limit. If a strategy fires several buys before any fills arrive, the filled position can still look safe while the outstanding book is already too large.

A better model reserves exposure when an order is accepted locally. Then risk is measured against filled position plus outstanding reservations.

struct RiskReservation {
    std::int64_t reserved_buy_quantity{0};
    std::int64_t reserved_sell_quantity{0};
};

A projected exposure helper can include both filled position and reserved quantity:

std::int64_t projected_buy_exposure(
    std::int64_t filled_position,
    std::int64_t reserved_buy_quantity
) noexcept {
    return filled_position + reserved_buy_quantity;
}

The key idea is simple:

If the order is allowed to exist, it must already be counted in the risk picture.

Reservations should be adjusted when the order is accepted, rejected, cancelled, replaced, partially filled, or fully filled.

Remember: a cancel request is not a cancel confirmation. Until the venue confirms the cancel, keep the reservation. Hope is not a risk release.

When an order event arrives, update reservations immediately and consistently.

A practical rule set is:

• New accepted order: reserve full remaining quantity
• Partial fill: reduce reserved quantity by executed amount
• Full fill: clear remaining reservation
• Cancel acknowledgement: clear remaining reservation
• Reject: clear reservation
• Replace: update reservation to match the new working quantity

Centralize this logic. Spreading exposure arithmetic across strategy code, order handlers, and risk listeners is how double counting appears.

struct PositionState {
    std::int64_t filled_position{0};
    std::int64_t reserved_buy{0};
    std::int64_t reserved_sell{0};
};

Then projected exposure comes from one source of truth.

The goal is not to over-engineer the model. The goal is to stop the system inventing extra buying power just because two asynchronous events arrived in the wrong order.

An order is not merely “sent” or “filled.” Its life is an asynchronous state machine with opinions.

A simplified state model:

A compact representation:

enum class OrderState : std::uint8_t {
    new_order,
    pending_new,
    live,
    partially_filled,
    pending_cancel,
    cancelled,
    filled,
    rejected
};

An order record might contain:

struct Order {
    std::uint64_t internal_id;
    std::uint64_t exchange_order_id;
    std::int64_t price_ticks;
    std::int64_t original_quantity;
    std::int64_t leaves_quantity;
    std::int64_t cumulative_quantity;
    OrderState state;
};

Transitions must be explicit and validated:

bool can_request_cancel(OrderState state) noexcept {
    return state == OrderState::live ||
           state == OrderState::partially_filled;
}

Do not assume a cancel request means the order is dead. Until the venue confirms it, that order may still execute and embarrass your position math.

This matters operationally:

Your position and exposure accounting must remain correct in every branch.

The session itself has states:

enum class SessionState : std::uint8_t {
    disconnected,
    connecting,
    logon_sent,
    active,
    recovery_required,
    logout_sent,
    stopped
};

Important session events may include:

• TCP or UDP transport loss
• Login accepted or rejected
• Heartbeat timeout
• Sequence gap
• Trading halt
• Protocol-level reject
• Exchange maintenance state
• Recovery snapshot completion
• Manual kill switch

A strategy should not confuse “the gateway process is alive” with “it is safe to trade.” Alive and tradable are different states.

At minimum, it needs a well-defined answer to:

bool may_send_orders(
    SessionState session,
    bool market_data_synchronized,
    bool risk_system_healthy,
    bool kill_switch_active
) noexcept {
    return session == SessionState::active &&
           market_data_synchronized &&
           risk_system_healthy &&
           !kill_switch_active;
}

Production systems usually need more conditions, but the principle stands: trading permission should be derived from explicit state, not vibes.

When a trading system disconnects, the hard part is often not reconnecting. The hard part is knowing what happened while you were away.

An order gateway must reconcile its local view with the venue's view after a disconnect or session reset.

That means answering questions like:

• Did the order reach the venue?
• Was it acknowledged before the disconnect?
• Was it partially filled?
• Was it cancelled, rejected, or replaced?
• Did the venue process an action the local system never saw?
• Is the local position still correct?

A practical recovery flow may look like this:

1. Detect session loss or sequence failure.
2. Stop sending new risk-bearing orders.
3. Mark affected instruments or strategies as not tradable.
4. Request recovery data from the venue, if supported.
5. Rebuild order state from execution reports, open-order snapshots, or drop copies.
6. Compare local order records with venue reality.
7. Resolve unknown or ambiguous states conservatively.
8. Resume trading only when the system is synchronized and validated.

A useful internal representation may include an explicit recovery state:

enum class RecoveryState : std::uint8_t {
    none,
    pending,
    reconciling,
    synchronized,
    failed
};

A gateway should not assume that "no acknowledgement received" means "order not accepted." Network loss, processing delay, and venue-side buffering can all produce ambiguity.

A safer rule is:

If order status is uncertain, treat it as live until proven otherwise.

That does not mean leaving all exposure open forever. It means reconciliation drives the next action, not guesswork. During uncertainty, restrict new risk and prioritize reconciliation and cancellation as appropriate.

Rate limiting is not only an exchange compliance checkbox. It is also a safety control.

A strategy that can generate too many messages too quickly can overload the venue, the gateway, internal queues, recovery logic, audit logging, and risk handling.

A practical gateway usually separates limits for:

• New orders
• Cancels
• Replaces
• Session messages
• Recovery requests

One simple approach is a token bucket.

struct TokenBucket {
    std::int64_t tokens{0};
    std::int64_t capacity{0};
    std::int64_t refill_per_second{0};
    std::uint64_t last_refill_ns{0};
};

A refill function may look like this:

void refill(TokenBucket& bucket, std::uint64_t now_ns) noexcept {
    if (now_ns <= bucket.last_refill_ns) {
        return;
    }

    const auto elapsed_ns = now_ns - bucket.last_refill_ns;
    const auto added =
        static_cast<std::int64_t>(
            (elapsed_ns * static_cast<std::uint64_t>(bucket.refill_per_second))
            / 1'000'000'000ULL
        );

    if (added > 0) {
        bucket.tokens = std::min(
            bucket.capacity,
            bucket.tokens + added
        );
        bucket.last_refill_ns = now_ns;
    }
}

And a consume step:

bool try_consume(TokenBucket& bucket, std::int64_t amount) noexcept {
    if (bucket.tokens < amount) {
        return false;
    }

    bucket.tokens -= amount;
    return true;
}

This is intentionally simple. Production code may separate buckets by order type or instrument class, and it must use a monotonic clock for now_ns.

The important policy decision is what to do when the bucket is empty:

• Reject the request locally
• Prioritize cancel messages over new orders
• Queue only if the queue is bounded and the delay is acceptable
• Fail closed for safety-critical paths

A good rule is:

A rate limit should protect the system, not quietly defer danger.

Not all messages have the same urgency.

Under normal conditions, a gateway may process everything in order. Under stress, it should prefer safety-critical work.

A practical priority order may be:

1. Risk controls
2. Cancels
3. Session recovery
4. Execution report handling
5. New orders
6. Telemetry
7. Diagnostics

That ordering is not universal, but the idea is important: if the system is overloaded, the first thing to preserve is the ability to reduce risk.

For example, a cancel request may beat a new quote, an execution report may beat a diagnostic log, and a kill switch may beat an analytics update.

This is why overload behavior must be designed before a market move designs it for you.

A general-purpose operating system can move a thread between CPU cores. Migration can disturb cache locality and add jitter.

On Linux, a thread may be pinned to a logical CPU:

#include <pthread.h>
#include <sched.h>

bool pin_current_thread_to_cpu(int cpu_id) noexcept {
    cpu_set_t set;
    CPU_ZERO(&set);
    CPU_SET(cpu_id, &set);

    return pthread_setaffinity_np(
        pthread_self(),
        sizeof(set),
        &set
    ) == 0;
}

Pinning is not a complete latency spa day. A disciplined deployment may also consider:

• Reserving isolated CPU cores
• Avoiding unrelated workloads on those cores
• Binding memory allocations to the local NUMA node
• Managing interrupt affinity
• Setting appropriate power-management policies
• Disabling or controlling frequency scaling where operationally justified
• Avoiding oversubscription
• Maintaining realistic failover behavior

Be cautious with hyperthreading or simultaneous multithreading. Two logical CPUs may share execution resources on the same physical core. Whether it helps depends on workload and hardware.

Always validate operational changes with infrastructure and security teams. Low-latency tuning that wrecks stability or monitoring is not a win. It is a fancy outage.

On a multi-socket server, memory is often Non-Uniform Memory Access (NUMA).

A CPU core can usually access memory attached to its own socket faster than memory attached to another socket.

A latency-sensitive thread pinned to a core on socket $0$ should ideally allocate and access its hot data on NUMA node $0$.

Bad pattern:

Potential consequences:

• Higher memory latency
• More interconnect traffic
• Increased tail latency
• Less predictable performance

First-touch allocation can matter: on many systems, physical memory is assigned to the NUMA node of the thread that first writes to it.

That means initialization placement can affect runtime latency.

NUMA behavior is platform-specific. Measure it using the deployment environment, not a laptop benchmark.

Normal networking often uses interrupts: the network interface receives a packet and the operating system schedules work.

This saves CPU when traffic is low, but can introduce variable wake-up latency.

Busy polling means a dedicated thread repeatedly checks for work:

while (running) {
    if (socket_has_packet()) {
        process_packet();
    }
}

Potential benefits:

• Reduced wake-up latency
• Fewer scheduler interactions
• More predictable timing
• Lower jitter under sustained load

Costs:

• Consumes a full CPU core
• Can worsen system-wide contention
• May increase heat and power use
• Requires careful deployment isolation
• Is wasteful during quiet periods

Busy polling is a trade-off, not a personality trait. It is often appropriate only for carefully isolated latency-critical processes.

Traditional sockets involve several layers:

Kernel-bypass frameworks can allow an application to interact more directly with network-interface queues, often through shared memory and polling.

Potential benefits include:

• Fewer copies
• Fewer system calls
• Less kernel scheduling variability
• More direct packet handling
• Better control over receive/transmit queues

But kernel bypass increases complexity:

• Driver and hardware dependencies
• Operational monitoring challenges
• Different security and isolation considerations
• More responsibility for packet handling
• More difficult upgrades and debugging
• Possible incompatibility with existing tooling

Use kernel bypass only when measured end-to-end gains justify the operational cost. Complexity is not free just because it is fashionable.

A trading system needs several different notions of time:

A simple latency decomposition:

And:

But timestamps are only meaningful if their clocks are understood.

Potential issues:

• Clock drift
• Clock steps caused by time synchronization
• Different clock domains
• Unstable cycle counters on old or virtualized systems
• Cross-core timestamp consistency
• Exchange timestamps representing different events than assumed

Use a monotonic clock for measuring intervals:

#include <chrono>

const auto start = std::chrono::steady_clock::now();

// Work

const auto end = std::chrono::steady_clock::now();
const auto elapsed = end - start;

For operational latency analysis, hardware timestamping and carefully synchronized clocks can be necessary. The exact approach depends on infrastructure and regulatory requirements.

A trading system often sees several timestamps for the same event. Those timestamps are not interchangeable.

Distinguish at least:

• Exchange timestamp
• NIC receive timestamp
• Application receive timestamp
• Decision timestamp
• Send timestamp (handed to kernel or NIC)
• Acknowledgement receive timestamp

Use a monotonic local clock for elapsed-time measurements.

Do not subtract an exchange timestamp from a local NIC or application timestamp unless the clocks are synchronized and the semantics are known. Mixing clock domains invents a fictional latency number with confidence.

The safe mental model is:

Use one clock for durations, and treat cross-domain timestamps as separate observations.

If the infrastructure supports precision time protocols such as PTP, analysis can improve. Synchronization still does not erase the need to understand what each timestamp actually means.

This is the classic Sync Log Pitfall:

logger.info(
    "Received order {} at price {} quantity {}",
    order_id,
    price,
    quantity
);

It may allocate, format strings, grab locks, write buffers, wake another thread, or do I/O. Congratulations: you just invited a novelist onto the runway.

Prefer stuffing compact binary telemetry into a bounded buffer:

struct LatencyEvent {
    std::uint64_t timestamp;
    std::uint64_t correlation_id;
    std::uint32_t event_type;
    std::uint32_t value;
};

Then pass it to an asynchronous telemetry thread.

Even asynchronous logging must have defined overload behavior. If telemetry is overloaded, decide what happens:

• Drop low-priority diagnostics
• Sample events
• Aggregate counters
• Preserve risk and audit events
• Trigger an alert
• Never block the matching or order-send path unexpectedly

Telemetry and audit are related, but they are not the same thing.

Telemetry is for observing the system. Audit is for proving what happened.

That distinction matters because telemetry can often be sampled, aggregated, or dropped under pressure. Audit records usually cannot be silently lost.

The practical lesson is:

Do not let a normal logger become the only record of a critical control event.

If the event affects money, compliance, or safety, the design should say so explicitly. Do not accidentally put compliance-grade durability requirements into an ordinary best-effort logger.

Modern CPUs predict branches. A frequently executed branch with predictable outcomes is cheap. An unpredictable branch can cause a pipeline flush.

if (event.type == EventType::trade) {
    process_trade(event);
} else {
    process_other(event);
}

If event types follow a stable pattern, this may be efficient. If the pattern is highly unpredictable, it may be costly.

Do not turn every branch into branchless arithmetic as a fashion statement. Branchless code can add instructions, reduce clarity, and perform worse.

Useful principles:

• Keep common paths simple
• Put rare error handling out of the primary path
• Order conditionals by likely outcome when known
• Use profiles to identify genuine hotspots
• Benchmark with representative event distributions

Compiler-specific branch hints exist, but they should be applied only after profiling:

if (__builtin_expect(condition, 1)) {
    // Common path
}

Such hints are not portable and can bite you when the workload changes its mind.

Fail closed at the risk gate, reserve outstanding exposure, reconcile after disconnects, rate-limit with cancel priority, pin and place memory deliberately, keep clocks honest, and never confuse best-effort telemetry with audit.

Practice: write may_send_orders for your own imaginary gateway and list three session events that must flip it false immediately.