C++
Language
since C++11
Advanced

C++ Memory Model and Atomics

C++ memory model — happens-before, memory orders, std::atomic operations, data races, and lock-free patterns. Deep reference for C++11 and beyond.

Memory Modelsince C++11

The C++ memory model defines the legal orderings of memory operations across threads, specifying precisely when a write by one thread becomes visible to another and enabling formally correct concurrent programs that are portable across all hardware architectures.

Overview

Before C++11, C++ had no notion of threads. Concurrent programs depended entirely on platform-specific guarantees — x86's total store order, ARM and POWER's weaker models — with no portable abstraction. C++11 introduced a formal memory model built on two primitives:

  • Sequenced-before: within a single thread, earlier statements are sequenced before later ones.
  • Synchronizes-with: an atomic release-store synchronizes with an acquire-load that reads the stored value.

Happens-before is derived transitively from these. If A happens-before B, every write by A is visible to B. Any concurrent access where the write is not ordered via happens-before is a data race — undefined behavior in the formal sense, not merely a bug with surprising output. The compiler and CPU are free to reorder, elide, or invent reads as long as the abstract machine behavior is preserved for a single thread.

Data Races

A data race exists when two threads access the same memory location concurrently, at least one access is a write, and neither happens-before the other. Even bool is not exempt — a single-byte write is not guaranteed atomic by the C++ abstract machine.

cpp
int counter = 0;
// Thread 1        Thread 2
counter++;     //  counter++;  // UB: data race — torn read/write, optimizer aliasing

// Fix: promote to atomic
std::atomic<int> counter{0};  // C++11
counter++;  // atomic RMW, safe from any thread, seq_cst by default

The undefined behavior here is not just stale data. An optimizer that proves two threads share a non-atomic variable may cache it in a register, hoist it out of a loop, or eliminate stores entirely.

std::atomic Operations

cpp
#include <atomic>  // C++11

std::atomic<int> x{0};

// Load / store with explicit ordering
int val = x.load(std::memory_order_acquire);
x.store(42, std::memory_order_release);

// Read-modify-write — atomic swap
int old = x.exchange(10, std::memory_order_acq_rel);

// Arithmetic — integral and pointer types only, returns old value
x.fetch_add(1, std::memory_order_relaxed);
x.fetch_sub(1);
x.fetch_and(0xFF);
x.fetch_or(flags);
x.fetch_xor(bits);

// Operator shorthands — always seq_cst, no way to override
x++;  x--;  x += 5;  x &= mask;

// Compare-and-swap — the foundation of lock-free algorithms
int expected = 0;
bool ok = x.compare_exchange_strong(expected, 1);
// x == expected → x = 1, returns true
// x != expected → expected = current x, returns false

// Weak CAS: may spuriously fail even when x == expected; always use in a loop
int cur = x.load(std::memory_order_relaxed);
while (!x.compare_exchange_weak(cur, cur * 2,
                                std::memory_order_release,
                                std::memory_order_relaxed));

// Lock-free checks
static_assert(std::atomic<int>::is_always_lock_free);  // C++17, compile-time
x.is_lock_free();  // runtime check for platform-sized types

Use compare_exchange_strong when you need a definitive yes/no answer without retrying. Use compare_exchange_weak inside retry loops — it maps directly to LL/SC instructions on ARM and POWER without a spurious-failure wrapper.

C++20 additions: x.wait(old) blocks the calling thread until the value changes (OS-assisted, not spinning). x.notify_one() and x.notify_all() wake blocked waiters.

cpp
// C++20: atomic wait/notify replaces spin-wait in many cases
std::atomic<int> phase{0};

// Consumer — blocks until phase != 0, no hot spin
phase.wait(0, std::memory_order_acquire);  // C++20

// Producer
phase.store(1, std::memory_order_release);
phase.notify_all();  // C++20

Memory Orders

OrderConstraint
memory_order_relaxedAtomicity only — no ordering relative to other operations
memory_order_consumeLoad-dependent ordering; avoid — no compiler implements it correctly; treat as acquire
memory_order_acquireNo load or store in this thread may be reordered before this load
memory_order_releaseNo load or store in this thread may be reordered after this store
memory_order_acq_relAcquire + release combined; for RMW operations
memory_order_seq_cstTotal global ordering across all seq_cst operations — default

memory_order_consume was designed for pointer-chasing on POWER (cheaper than acquire for dependent loads), but every major compiler promotes it to acquire. It is marked for revision in future standards. Do not use it.

seq_cst — Total Order

seq_cst imposes a single total order on all seq_cst atomic operations across all threads. Every thread agrees on the sequence in which they occurred.

The IRIW litmus test demonstrates the guarantee that only seq_cst provides:

cpp
std::atomic<int> x{0}, y{0};
int r1, r2, r3, r4;

// Thread A           Thread B           Thread C                 Thread D
x.store(1);          y.store(1);        r1=x.load();             r3=y.load();
                                         r2=y.load();             r4=x.load();

// With seq_cst: r1==1 && r2==0 && r3==1 && r4==0 is impossible.
// Threads C and D must agree on whether x's or y's store happened first.
// With only acquire/release: this outcome IS permitted on ARM/POWER.

Cost on x86 (TSO): seq_cst stores emit MFENCE or XCHG. On ARM: full DMB ISH barriers. Acquire-loads and release-stores, by contrast, are plain MOV on x86 and LDAR/STLR on ARMv8 — significantly cheaper.

acquire/release — Efficient Handoff

A release-store synchronizes with any acquire-load that reads the stored value. Everything the releasing thread wrote before the release is visible to the acquiring thread after the acquire.

cpp
// Spinlock
class Spinlock {
    std::atomic_flag flag_{};  // C++11; ATOMIC_FLAG_INIT deprecated in C++20
public:
    void lock() {
        while (flag_.test_and_set(std::memory_order_acquire)) {
            flag_.wait(true, std::memory_order_relaxed);  // C++20: OS wait, no hot spin
        }
    }
    void unlock() {
        flag_.clear(std::memory_order_release);
        flag_.notify_one();  // C++20
    }
};

// Message passing
struct Payload { int value; double coeff; };
std::atomic<Payload*> msg{nullptr};

// Thread 1 (producer)
auto* p = new Payload{42, 3.14};
msg.store(p, std::memory_order_release);  // all writes to *p are visible after acquire

// Thread 2 (consumer)
Payload* p = nullptr;
while (!(p = msg.load(std::memory_order_acquire)));
// p->value == 42, p->coeff == 3.14 — guaranteed by the release/acquire pair

relaxed — Atomicity Only

No synchronization, no ordering — only the guarantee that the operation is indivisible. Correct when the value is read only after an independent happens-before edge (e.g., thread join) provides the ordering.

cpp
std::atomic<uint64_t> bytes_in{0}, bytes_out{0};

void record_transfer(size_t n) {
    bytes_in.fetch_add(n, std::memory_order_relaxed);   // no ordering needed
    bytes_out.fetch_add(n, std::memory_order_relaxed);
}

// After joining all worker threads — join establishes happens-before:
std::printf("in=%llu out=%llu\n",
    bytes_in.load(std::memory_order_relaxed),
    bytes_out.load(std::memory_order_relaxed));

Never use relaxed to signal state that other threads access non-atomically — you get no happens-before relationship to make those accesses safe.

Happens-Before in Detail

The happens-before relation is derived from three rules:

  1. Sequenced-before: A appears before B in the same thread's execution.
  2. Synchronizes-with: A is a release-store and B is an acquire-load that reads A's value (or a later value from the same atomic written with release).
  3. Transitivity: A hb→ X and X hb→ B implies A hb→ B.
cpp
std::atomic<int> flag{0};
int data = 0;

// Thread 1
data = 100;                                // (i)  sequenced before (ii)
flag.store(1, std::memory_order_release);  // (ii) release-store

// Thread 2
while (flag.load(std::memory_order_acquire) == 0); // (iii) reads (ii)'s value
assert(data == 100);  // safe: (i) sb (ii), (ii) sw (iii), so (i) hb (iii)

The acquire-load only synchronizes with the thread that performed the matching release. Reading a value written by a relaxed store establishes no synchronization even if the load uses acquire.

Lock-Free Patterns

Treiber Stack (C++11)

cpp
template<typename T>
class TreiberStack {
    struct Node { T value; Node* next; };
    std::atomic<Node*> head_{nullptr};

public:
    void push(T val) {
        Node* node = new Node{std::move(val), head_.load(std::memory_order_relaxed)};
        // CAS loop: node->next updated by CAS on failure
        while (!head_.compare_exchange_weak(node->next, node,
                                            std::memory_order_release,
                                            std::memory_order_relaxed));
    }

    std::optional<T> pop() {
        Node* top = head_.load(std::memory_order_acquire);
        while (top) {
            if (head_.compare_exchange_weak(top, top->next,
                                            std::memory_order_acquire,
                                            std::memory_order_acquire)) {
                T val = std::move(top->value);
                // WARNING: bare delete is unsafe in production — ABA problem.
                // Another thread may free and reallocate this node between our
                // load and CAS. Use hazard pointers (C++26 std::hazard_pointer)
                // or epoch-based reclamation in production code.
                delete top;
                return val;
            }
        }
        return std::nullopt;
    }
};

The ABA problem: thread 1 reads head_ == A. Thread 2 pops A, pops B, pushes A back. Thread 1's CAS on head_ succeeds (still A) but A->next now points to freed memory. Hazard pointers (std::hazard_pointer, C++26) solve this by deferring reclamation until no thread holds a reference.

std::atomic_ref (C++20)

Applies atomic operations to elements of a plain array or struct without changing the type:

cpp
#include <atomic>  // C++20

std::vector<int> histogram(256, 0);

void tally(std::span<const uint8_t> data) {  // std::span: C++20
    for (uint8_t b : data)
        std::atomic_ref<int>{histogram[b]}.fetch_add(1, std::memory_order_relaxed);
}
// Thread-safe per-bucket updates; separate buckets never conflict.
// Requirement: histogram[i] must be aligned to atomic_ref<int>::required_alignment.

C++20 also added specializations std::atomic<std::shared_ptr<T>> and std::atomic<std::weak_ptr<T>>, enabling lock-free atomic updates to reference-counted pointers.

Common Pitfalls

cpp
// WRONG: relaxed store/load — no release/acquire, no synchronization
std::atomic<bool> ready{false};
int data = 0;
// Thread 1:
data = 42;
ready.store(true, std::memory_order_relaxed);  // no release semantics
// Thread 2:
while (!ready.load(std::memory_order_relaxed)); // no acquire semantics
// data may read 0 — no happens-before between threads

// RIGHT
ready.store(true, std::memory_order_release);
while (!ready.load(std::memory_order_acquire));

// WRONG: check-then-act is not atomic even with seq_cst
if (x.load() == 0) x.store(1);  // another thread may store between load and store
// RIGHT: express the entire operation atomically
int expected = 0;
x.compare_exchange_strong(expected, 1);

// WRONG: mixing seq_cst and weaker orders breaks the total-order guarantee
x.store(1);                                     // seq_cst
int v = x.load(std::memory_order_relaxed);      // relaxed load opts out of total order
// seq_cst guarantees hold only when ALL participating operations use seq_cst.

// WRONG: non-atomic double-checked locking (classic pre-C++11 bug)
if (!instance) {              // non-atomic read — data race
    std::lock_guard lk{mu};
    if (!instance) instance = new T{};  // non-atomic write — data race
}
// RIGHT: use std::call_once or a local static (C++11 guarantees thread-safe init)
static T& get() {
    static T instance{};  // C++11: initialization is thread-safe
    return instance;
}

Best Practices

  • Default to seq_cst while developing. Weaken only after correctness is established and profiling shows the cost matters.
  • Acquire/release for producer-consumer: the most common pattern, zero cost on x86.
  • relaxed only for independent accumulators where thread join provides the eventual happens-before.
  • Annotate every atomic operation with which side of the synchronization it represents — the code is not self-documenting.
  • Avoid memory_order_consume — treat it as acquire; no production compiler implements it as specified.
  • Test on ARM — acquire/release and especially seq_cst have measurable cost on weakly-ordered hardware; x86 masks the difference during development.
  • Never roll a lock-free data structure in production without understanding reclamation. Prefer std::mutex unless you have profiled evidence that lock contention is the bottleneck.

std::atomic_thread_fence — Standalone Fences

Fences decouple ordering from a specific atomic operation. A standalone std::atomic_thread_fence imposes ordering between all preceding or following memory operations in the calling thread, without being tied to a single load or store. This is useful when you want to batch several relaxed atomics and pay for synchronization exactly once.

cpp
// C++11
#include <atomic>

std::atomic<int> a{0}, b{0}, c{0};
int result_a, result_b, result_c;

// Thread 1 — publish three values with a single release fence
a.store(1, std::memory_order_relaxed);
b.store(2, std::memory_order_relaxed);
c.store(3, std::memory_order_relaxed);
std::atomic_thread_fence(std::memory_order_release);  // one fence covers all stores above

// Thread 2 — acquire fence pairs with the release fence above
std::atomic_thread_fence(std::memory_order_acquire);
result_a = a.load(std::memory_order_relaxed);  // sees 1
result_b = b.load(std::memory_order_relaxed);  // sees 2
result_c = c.load(std::memory_order_relaxed);  // sees 3

A fence-based release synchronizes with a fence-based acquire only if the acquire-side thread has observed a store that was sequenced after the release fence. The pairing rule is stricter than per-operation release/acquire — you must ensure a relaxed atomic carries the "happened" signal across threads (e.g., a flag that Thread 2 polls). Prefer per-operation ordering when in doubt; fences are primarily an optimization for throughput-sensitive code on weakly-ordered hardware (ARM, POWER) where batching barriers saves real cycles.

cpp
// C++11 — typical fence pattern with a flag to carry the signal
std::atomic<bool> published{false};
int shared_data = 0;

// Thread 1
shared_data = 99;
std::atomic_thread_fence(std::memory_order_release);
published.store(true, std::memory_order_relaxed);  // relaxed — fence already ordered

// Thread 2
while (!published.load(std::memory_order_relaxed));  // spin on flag
std::atomic_thread_fence(std::memory_order_acquire);
assert(shared_data == 99);  // safe: fence pair + the flag's relaxed store/load bridge them

memory_order_consume — Dependency-Chain Ordering

memory_order_consume was designed to be cheaper than acquire on POWER/ARM by exploiting hardware data-dependency tracking. The idea: if a thread loads a pointer P with consume, any subsequent dereference of P (or any value computed from P) is guaranteed to see the writes the producing thread sequenced before its release-store — without emitting a full memory barrier. Only operations in the dependency chain rooted at the loaded value are ordered; unrelated loads are not.

cpp
// C++11 — intended consume usage (theoretical; do not use in production)
struct Config { int threshold; double factor; };
std::atomic<Config*> cfg_ptr{nullptr};

// Thread 1 (producer)
Config* cfg = new Config{10, 2.5};
cfg_ptr.store(cfg, std::memory_order_release);

// Thread 2 (consumer) — consume: cheaper barrier, only cfg and *cfg are ordered
Config* p = cfg_ptr.load(std::memory_order_consume);  // dependency chain starts here
int t = p->threshold;  // safe: p depends on the consume load
double f = p->factor;  // safe: same dependency chain

In practice, every production compiler — GCC, Clang, MSVC — promotes memory_order_consume to memory_order_acquire. The specification requires compilers to track dependency chains through function calls, casts, and arithmetic, which proved impossible to implement correctly without banning common optimizations. The standard itself (since C++17) explicitly notes the issue and recommends treating consume as acquire until a revision arrives. Use memory_order_acquire directly. The performance gap between acquire and consume on real POWER hardware is narrow once you account for the compiler's inability to avoid the barrier anyway.

std::atomic_ref — Atomics Without Changing the Type

std::atomic_ref<T> (C++20) lets you apply atomic operations to an ordinary object without changing its declared type. This is invaluable when the object lives in a legacy struct, a C array, or a memory-mapped region where you cannot declare the field as std::atomic<T>.

cpp
// C++20
#include <atomic>
#include <vector>
#include <span>
#include <cstdint>

// Legacy struct from a C API — cannot modify the layout
struct SensorFrame {
    uint32_t sequence;
    float    values[16];
};

void update_sequence(SensorFrame& frame, uint32_t next) {
    // Treat frame.sequence as atomic for this call only
    std::atomic_ref<uint32_t> seq_ref{frame.sequence};
    seq_ref.store(next, std::memory_order_release);
}

uint32_t read_sequence(const SensorFrame& frame) {
    // atomic_ref<const T> is not valid; cast away const and use acquire
    std::atomic_ref<uint32_t> seq_ref{const_cast<uint32_t&>(frame.sequence)};
    return seq_ref.load(std::memory_order_acquire);
}

atomic_ref has two key constraints. First, the referenced object must be aligned to atomic_ref<T>::required_alignment, which may exceed alignof(T) on some platforms — allocate with alignas. Second, all concurrent accesses to the object must go through atomic_ref instances; mixing direct non-atomic access with an atomic_ref in flight is still a data race.

cpp
// C++20 — per-bucket histogram over a plain array, multiple threads
alignas(std::atomic_ref<int>::required_alignment) int histogram[256] = {};

void tally_chunk(std::span<const uint8_t> chunk) {
    for (uint8_t byte : chunk)
        std::atomic_ref<int>{histogram[byte]}.fetch_add(1, std::memory_order_relaxed);
    // Buckets are independent: relaxed is sufficient, join provides final happens-before
}

std::atomic<std::shared_ptr<T>> and std::atomic<std::weak_ptr<T>> were also added in C++20 as full specializations, replacing the deprecated free functions std::atomic_load / std::atomic_store from C++11.

Testing for Data Races with ThreadSanitizer

ThreadSanitizer (TSan) is the fastest way to find data races before they corrupt production data. It instruments every memory access at compile time and tracks happens-before dynamically, reporting races with full stack traces for both the racing accesses. Enable it with a single compiler flag — no source changes needed.

cpp
// C++11 — example with a real race that TSan will catch
#include <thread>

int shared = 0;  // not atomic, not protected

int main() {
    std::thread t1([] { shared = 1; });  // write
    std::thread t2([] { shared = 2; });  // concurrent write — data race
    t1.join();
    t2.join();
}

Build and run with TSan:

cpp
# Clang or GCC — same flag
clang++ -std=c++17 -fsanitize=thread -g -O1 -o race_demo race_demo.cpp
./race_demo

TSan output for the example above looks like:

cpp
WARNING: ThreadSanitizer: data race (pid=12345)
  Write of size 4 at 0x... by thread T2:
    #0 main::{lambda()#2}::operator()() race_demo.cpp:8
    #1 ...

  Previous write of size 4 at 0x... by thread T1:
    #0 main::{lambda()#1}::operator()() race_demo.cpp:7
    #1 ...

  Location is global 'shared' of size 4 at 0x...
SUMMARY: ThreadSanitizer: data race race_demo.cpp:8

TSan reports the two conflicting accesses with their stack traces and the variable involved. Common findings include relaxed counters used for signaling (fix: use release/acquire), non-atomic flags checked outside a mutex (fix: std::atomic<bool>), and iterator invalidation in concurrent containers (fix: external locking or a concurrent container).

Three practical notes. TSan has roughly 5–10× runtime overhead and 5–10× memory overhead — run it in CI on your test suite, not in production. It requires that all code linked together is instrumented; mixing instrumented and uninstrumented .so files produces false negatives. Finally, TSan understands std::mutex, std::atomic, and pthread_* primitives natively, so it correctly suppresses false positives from properly synchronized code.

See Also

  • std::mutex / std::condition_variable — higher-level synchronization built on the memory model
  • Coroutines — cooperative multitasking, no shared-memory races by default
  • RAII — deterministic resource release, works correctly across threads when paired with proper synchronization