Cache Optimization for Low-Latency C++

TL;DR

Cache optimization is the single highest-ROI performance technique for HFT. Fix false sharing first (cheap, massive impact), then align hot data to cache lines, then prefetch for predictable access patterns. Always measure with perf stat -e cache-misses before and after.

Cache hierarchy refresher

Level	Size (typical)	Latency	Shared?
L1	32–64 KB	4 cycles (~1ns)	Per core
L2	256 KB–1 MB	12 cycles (~4ns)	Per core
L3	8–64 MB	40 cycles (~12ns)	Shared
RAM	GBs	200 cycles (~60ns)	Shared

A cache line is 64 bytes on x86. The CPU always loads and invalidates 64-byte chunks.

False sharing occurs when two threads modify different variables that happen to live in the same 64-byte cache line. Every write by thread A invalidates the cache line in thread B, even though they're accessing different variables.

cpp

Godbolt

// BUG: counter[0] and counter[1] share a cache line
struct BadCounters {
    std::atomic<int64_t> producer_count;  // offset 0
    std::atomic<int64_t> consumer_count;  // offset 8 — same cache line!
};

// Thread 1 writes producer_count → invalidates cache line in Thread 2
// Thread 2 writes consumer_count → invalidates cache line in Thread 1
// → Cache line ping-pong between cores → 40-200 cycles per write

Fix: put each hot variable on its own cache line.

cpp

Godbolt

struct alignas(64) GoodCounters {
    alignas(64) std::atomic<int64_t> producer_count;  // cache line 0
    alignas(64) std::atomic<int64_t> consumer_count;  // cache line 1
};

// Or use padding explicitly
struct GoodCounters2 {
    std::atomic<int64_t> producer_count;
    char _pad1[64 - sizeof(std::atomic<int64_t>)];
    std::atomic<int64_t> consumer_count;
    char _pad2[64 - sizeof(std::atomic<int64_t>)];
};

The performance difference is dramatic: from ~200 cycles per write (false sharing) to ~5 cycles (own cache line). In an HFT queue, this is the difference between 200ns and 5ns throughput.

Cache-line-aligned data structures

cpp

Godbolt

// Order book entry — pad to 64 bytes so each entry is exactly one cache line
struct alignas(64) PriceLevel {
    int64_t  price;       // 8 bytes
    int64_t  quantity;    // 8 bytes
    uint32_t order_count; // 4 bytes
    uint32_t flags;       // 4 bytes
    // 40 bytes padding to fill cache line
    char _pad[40];
};
static_assert(sizeof(PriceLevel) == 64);
static_assert(alignof(PriceLevel) == 64);

// Array of PriceLevels — each level on its own cache line
// Accessing level[i] never loads level[i+1] into cache
std::array<PriceLevel, 1024> book;

When to align to cache lines:

Objects accessed by multiple threads (prevents false sharing)
Objects in hot inner loops where you want exactly one cache miss per access
SIMD buffers (often need 32-byte or 64-byte alignment)

Struct packing and layout

cpp

Godbolt

// Poorly laid out — padding wastes space, misses cache-friendly access
struct BadLayout {
    char   flag;         // 1 byte
    // 7 bytes padding!
    double price;        // 8 bytes
    char   side;         // 1 byte
    // 7 bytes padding!
    int64_t quantity;    // 8 bytes
    // Total: 32 bytes (could be 18)
};

// Better — group by access pattern, not by "logical" grouping
struct GoodLayout {
    double  price;       // 8 bytes — most accessed
    int64_t quantity;    // 8 bytes
    int32_t order_id;    // 4 bytes
    char    side;        // 1 byte
    char    flag;        // 1 byte
    char    _pad[2];     // explicit padding
    // Total: 24 bytes
};

Use -Wpadding (GCC/Clang) to find implicit padding. Use offsetof() to verify layout:

cpp

Godbolt

static_assert(offsetof(GoodLayout, price) == 0);
static_assert(offsetof(GoodLayout, quantity) == 8);
static_assert(sizeof(GoodLayout) == 24);

Hot/cold data splitting

cpp

Godbolt

// Together: hot and cold data pollute each other's cache lines
struct OrderFull {
    // Hot (accessed on every tick)
    int64_t price;
    int64_t qty;
    uint32_t order_id;
    // Cold (accessed on fill/cancel only)
    char client_id[32];
    char symbol[12];
    char strategy[16];
    uint64_t timestamp_ns;
};

// Split: hot data accessed without loading cold data
struct OrderHot {
    int64_t  price;
    int64_t  qty;
    uint32_t order_id;
    uint32_t cold_idx;  // index into cold array
};
struct OrderCold {
    char     client_id[32];
    char     symbol[12];
    char     strategy[16];
    uint64_t timestamp_ns;
};

std::vector<OrderHot>  hot_orders;
std::vector<OrderCold> cold_orders;
// hot_orders is ~60% smaller → more fits in L1/L2

Software prefetching

When access patterns are predictable, tell the CPU to prefetch ahead:

cpp

Godbolt

// Prefetch the next iteration's data while processing current
for (size_t i = 0; i < n; ++i) {
    if (i + 8 < n)
        __builtin_prefetch(&data[i + 8], 0, 1);  // prefetch for read, L2

    process(data[i]);
}

// __builtin_prefetch(addr, rw, locality)
// rw: 0=read, 1=write
// locality: 0=no temporal locality (streaming), 1=L2, 2=L1, 3=L1 (keep)

For sequential access, the hardware prefetcher usually does this automatically. Manual prefetching helps for strided or indirect access:

cpp

Godbolt

// Indirect access — prefetch the pointed-to data
for (size_t i = 0; i + 4 < n; ++i) {
    __builtin_prefetch(pointers[i + 4], 0, 1);
    process(*pointers[i]);
}

Measuring cache performance

bash

# Cache miss rate
perf stat -e L1-dcache-loads,L1-dcache-load-misses,LLC-loads,LLC-load-misses ./app

# Detailed cache event breakdown
perf stat -e \
  cache-references,cache-misses,\
  L1-dcache-loads,L1-dcache-load-misses,\
  l2_rqsts.miss \
  ./app

# Find which functions cause most cache misses
perf record -e LLC-load-misses:pp ./app
perf report --sort=symbol

# Cachegrind simulation (no hardware PMU needed)
valgrind --tool=cachegrind ./app
cg_annotate cachegrind.out.<pid> src/hot_path.cpp

NUMA awareness

On multi-socket servers, accessing memory on a remote NUMA node adds ~100ns:

cpp

Godbolt

#include <numa.h>

// Allocate on local NUMA node
void* buf = numa_alloc_local(size);

// Pin thread to NUMA node 0
numa_run_on_node(0);

// Interleave allocation across all nodes (for shared data)
void* shared = numa_alloc_interleaved(size);

Check NUMA topology: numactl --hardware. In HFT, pin the critical path to a single NUMA node.