C++
Language
since C++11
Advanced

C++ Performance Optimization

Cache efficiency, branch prediction, SIMD, move semantics, PMR, profiling, and compiler flags — the full C++ performance toolkit.

Performance Optimizationsince C++11

Systematic reduction of execution time and memory usage through algorithmic selection, data layout, and low-level tuning — validated by profilers, not intuition.

Overview

Performance comes in layers: algorithm complexity first, data layout second, micro-level tuning last. The order matters because a cache-friendly O(n²) is still O(n²). Profilers expose where time is actually spent; everything else is speculation.

The Memory Hierarchy

cpp
L1 cache     ~4 cycles      32–64 KB    per core
L2 cache    ~12 cycles     256 KB–1 MB  per core
L3 cache    ~40 cycles      4–64 MB     shared
DRAM       ~100+ cycles     GBs         main memory

A single L3 miss to DRAM wastes ~100 instructions of compute time on a 4-IPC machine. Sequential access triggers hardware prefetch automatically; random pointer chasing does not. Most performance failures trace back to this single fact.

Examples

Cache-Friendly Data Layout

Blocked matrix traversal

cpp
constexpr int N = 1024;
float A[N][N], B[N][N];

// Naive: B[j][i] strides N*sizeof(float) per write — one cache miss per element
void transpose_naive() {
    for (int i = 0; i < N; ++i)
        for (int j = 0; j < N; ++j)
            B[j][i] = A[i][j];
}

// Blocked: each tile fits in L1; sequential access within the tile — 10–20× faster
void transpose_blocked(int block = 32) {
    for (int i = 0; i < N; i += block)
        for (int j = 0; j < N; j += block)
            for (int ii = i, ie = std::min(i + block, N); ii < ie; ++ii)
                for (int jj = j, je = std::min(j + block, N); jj < je; ++jj)
                    B[jj][ii] = A[ii][jj];
}

Struct of Arrays vs Array of Structs

cpp
// AoS — default layout; loads 32 bytes per particle, uses 24 (flags rarely touched)
struct Particle { float px, py, pz, vx, vy, vz, mass; uint32_t flags; };
Particle particles[100'000];  // digit separators: C++14

// SoA — 100% cache utilisation for position/velocity loops; trivially auto-vectorised
struct ParticlePool {
    float px[100'000], py[100'000], pz[100'000];
    float vx[100'000], vy[100'000], vz[100'000];
    float mass[100'000];
    uint32_t flags[100'000];  // cold — in its own cache lines
};

// Compiles to AVX2 on -O2 -march=native; processes 8 floats per instruction
void update_positions(ParticlePool& pool, float dt) {
    for (int i = 0; i < 100'000; ++i) {
        pool.px[i] += pool.vx[i] * dt;
        pool.py[i] += pool.vy[i] * dt;
        pool.pz[i] += pool.vz[i] * dt;
    }
}

Branch Prediction

Mispredicts drain the pipeline for ~15–20 cycles. Unsorted or randomly-valued data causes ~50% mispredict on a threshold branch.

cpp
// Branchless: arithmetic replaces the conditional; no pipeline flush
void process_branchless(int* data, int n) {
    for (int i = 0; i < n; ++i)
        data[i] *= 1 + (data[i] > 128);  // adds 0 or 1 to multiplier
}

// C++20: annotate cold paths so the compiler places hot code inline
int parse(const char* input) {
    if (!input) [[unlikely]] throw std::invalid_argument{"null"};
    return std::atoi(input);
}

[[likely]] and [[unlikely]] (C++20) affect code layout, not runtime speculation. They are most valuable on paths with known, extreme probabilities — error handlers, rarely-taken forks.

Move Semantics and Allocation Avoidance

C++11 move semantics eliminate copies on temporaries. NRVO constructs the return value directly in the caller's stack frame — zero copies, zero moves. Since C++17, copy elision for prvalues is mandatory.

cpp
// NRVO — v is constructed in-place at the call site (C++11+; mandatory elision C++17)
std::vector<int> make_range(int n) {
    std::vector<int> v(n);
    std::iota(v.begin(), v.end(), 0);  // C++11
    return v;
}

// Sink idiom: caller pays one copy (lvalue) or one move (rvalue) — never two
class Index {
    std::vector<int> keys_;
public:
    explicit Index(std::vector<int> keys)  // C++11
        : keys_(std::move(keys)) {}
};

Avoid heap allocation in hot paths by reusing buffers:

cpp
class Processor {
    std::vector<int> scratch_;  // member; capacity is never released

    void run(std::span<const int> input) {  // std::span: C++20
        scratch_.clear();                   // O(1) — capacity preserved
        scratch_.reserve(input.size());
        for (int x : input)
            if (x > 0) scratch_.push_back(x);
    }
};

C++17 std::pmr::monotonic_buffer_resource provides arena allocation — allocate from stack memory, free everything at once with zero per-object cost:

cpp
#include <memory_resource>  // C++17

void batch_process(std::span<std::string_view> inputs) {  // std::span: C++20
    std::array<std::byte, 64 * 1024> buf;
    std::pmr::monotonic_buffer_resource arena{buf.data(), buf.size()};
    std::pmr::vector<std::pmr::string> tmp{&arena};  // C++17
    for (auto sv : inputs)
        tmp.emplace_back(sv, &arena);  // no operator new for small strings
}  // entire arena freed here in one shot

SIMD and Auto-Vectorisation

SoA loops like update_positions above auto-vectorise with -O2 -march=native. Verify with:

bash
g++ -O2 -march=native -fopt-info-vec-optimized -fopt-info-vec-missed main.cpp

For explicit SIMD control when the compiler fails, use intrinsics (x86-only; requires -mavx2):

cpp
#include <immintrin.h>  // AVX2 intrinsics — not portable across ISAs

void saxpy_avx2(float* __restrict__ y, const float* __restrict__ x,
                float a, int n) {
    __m256 va = _mm256_set1_ps(a);
    int i = 0;
    for (; i + 8 <= n; i += 8)
        _mm256_storeu_ps(y + i,
            _mm256_fmadd_ps(va, _mm256_loadu_ps(x + i), _mm256_loadu_ps(y + i)));
    for (; i < n; ++i) y[i] += a * x[i];  // scalar tail
}

Prefer auto-vectorisation. Hand-written intrinsics tie you to an ISA, generate fragile code, and are rarely faster than what a modern compiler produces with proper data layout and alignment hints.

Profiling Workflow

bash
# Compile with frame pointers — required for accurate call stacks
g++ -O2 -g -fno-omit-frame-pointer -o myapp main.cpp

# Sample-based profiling (Linux)
perf record -g -- ./myapp
perf report --call-graph dwarf

# Cache miss breakdown
perf stat -e cycles,instructions,cache-misses,cache-references ./myapp

# Flame graph visualisation
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

For micro-benchmarks, Google Benchmark prevents dead-code elimination and accounts for statistical noise:

cpp
#include <benchmark/benchmark.h>

static void BM_transpose_blocked(benchmark::State& state) {
    for (auto _ : state)
        transpose_blocked();
}
BENCHMARK(BM_transpose_blocked)->Unit(benchmark::kMicrosecond);
BENCHMARK_MAIN();

Compiler Flags

bash
-O2                    # safe; production default
-O3                    # aggressive loop transforms; benchmark before committing
-march=native          # all features of the build host — not portable
-march=x86-64-v3       # SSE4.2 + AVX2; safe for post-2015 x86 deployments
-flto=thin             # cross-TU inlining (LLVM ThinLTO); use -flto for GCC
-fprofile-generate     # pass 1: instrument binary; run representative workload
-fprofile-use          # pass 2: recompile with PGO data — 10–30% gain on real code
-ffast-math            # allows FP reassociation; breaks strict IEEE 754 semantics
-fno-omit-frame-pointer # profiler-friendly; ~1% overhead

Profile-Guided Optimization is the highest-leverage single change available for long-running services. The compiler inlines and lays out hot paths based on actual execution frequencies, not static heuristics.

Best Practices

  1. Profile before touching anything — hotspots are rarely where intuition points; profilers are the only ground truth
  2. Algorithm first — O(n log n) beats a perfectly cache-tuned O(n²) past a few thousand elements
  3. Reserve early — vector::reserve at construction eliminates all reallocation; check with capacity() after a run
  4. Separate hot and cold fields — rarely-accessed members (flags, debug state) should be at the end of a struct or in a separate allocation
  5. SoA for data-parallel loops — array-per-field enables auto-vectorisation and eliminates wasted cache bandwidth
  6. Return by value — NRVO and mandatory copy elision (C++17) make return local_var zero-cost; don't use output parameters to avoid copies
  7. LTO in production builds — -flto=thin is low-risk and consistently improves cross-module call sites with no source changes
  8. PGO for high-traffic binaries — two-pass build pays for itself in reduced CPU budget within weeks

Common Pitfalls

False sharing in multithreaded code. Cache lines are 64 bytes. Two threads writing to adjacent variables in the same cache line cause constant invalidation across cores, serialising what should be parallel work:

cpp
// BAD: counters[0] and counters[1] share a 64-byte cache line
std::atomic<int> counters[2];  // C++11

// Fix: pad each counter to its own cache line
struct alignas(64) PaddedCounter {  // alignas: C++11
    std::atomic<int> value;
};
PaddedCounter counters[2];

-ffast-math in numerics code. The flag allows the compiler to reorder floating-point operations and eliminate NaN/Inf checks. This breaks Kahan summation, makes std::isnan unreliable, and can silently corrupt numerically sensitive algorithms. Enable only after validating output against a reference.

Over-inlining. Marking every small function [[clang::always_inline]] or [[gnu::always_inline]] inflates the instruction cache footprint. Inlining saves nanoseconds per call but wastes kilobytes of I-cache. Use only after the profiler identifies the call site as a measured hotspot.

Premature branchless conversion. Branchless arithmetic eliminates pipeline flushes but adds latency on predictable branches and prevents short-circuit evaluation. On sorted or otherwise predictable data, the original branchy code is often faster. Profile both.

Container constant-factor blindness. std::unordered_map is O(1) amortised but has high constant factors: hash computation, modulo, and pointer chasing. For maps under ~20 entries, a sorted std::array plus std::lower_bound is often faster due to cache locality. Always benchmark before choosing a container for a hot path.

Branch prediction and likely/unlikely (C++20)

Mispredicts are expensive pipeline flushes, not free conditional skips. Modern out-of-order CPUs speculatively execute down one branch; when they guess wrong they must discard ~15–20 cycles of work and restart. Annotating extreme-probability branches lets the compiler place hot code on the fall-through path and cold code out of line, reducing instruction-cache pressure.

cpp
// C++20 — [[likely]] / [[unlikely]] on branches
#include <stdexcept>

double safe_sqrt(double x) {
    if (x < 0.0) [[unlikely]]      // error path: almost never taken
        throw std::domain_error{"negative input"};
    return std::sqrt(x);
}

// Combine with switch statements: common case first
int categorise(int value) {
    if (value > 0) [[likely]] {    // C++20
        return 1;
    } else if (value < 0) {
        return -1;
    } else [[unlikely]] {          // zero is rare in this workload
        return 0;
    }
}
cpp
// C++20 — std::assume hint tells the optimiser a condition is always true
// (no runtime check; undefined behaviour if violated)
#include <utility>  // std::assume (P1774 — C++23 in most STLs; __builtin_assume on GCC/Clang)

void process_aligned(float* p, int n) {
    __builtin_assume(n % 8 == 0);          // Clang/GCC extension
    __builtin_assume(reinterpret_cast<uintptr_t>(p) % 32 == 0);
    for (int i = 0; i < n; ++i) p[i] *= 2.0f;  // now auto-vectorises without scalar tail
}

The annotations are hints only — the CPU's dynamic branch predictor still learns at runtime. Reserve [[unlikely]] for genuinely exceptional paths (error handlers, validation failures) where the static hint matches reality.

Cache-friendly data layout

The choice of data structure determines whether every hot loop runs at compute speed or memory-bandwidth speed. Cache lines are 64 bytes on all modern x86 and ARM chips; any access that crosses a line boundary or touches a line that holds no useful data wastes fetch bandwidth.

cpp
// C++11 — alignas forces a struct onto its own cache line
struct alignas(64) HotCounter {   // C++11
    std::atomic<long> value{0};
    // 56 bytes of padding implicit via alignas(64)
};

// sizeof(HotCounter) == 64; adjacent counters never share a line
HotCounter thread_counters[16];
cpp
// Prefer flat arrays over node-based containers in hot loops
#include <vector>
#include <list>

// std::list: each node is a separate heap allocation; pointer-chasing kills prefetch
std::list<int> node_list(1'000'000);          // C++14 digit separator

// std::vector: elements contiguous in memory; hardware prefetch is effective
std::vector<int> flat_vec(1'000'000);

// Benchmark typical: flat_vec iteration is 5–15× faster at 1M elements
cpp
// Cold/hot field splitting: keep per-frame data together, archive data separate
struct Entity {
    // Hot: accessed every frame
    float x, y, z;
    float vx, vy, vz;

    // Cold: accessed rarely (editor/save/serialisation only)
    // Move to a parallel array or a separate heap allocation
    char  name[64];
    int   editor_id;
    bool  locked;
};

Prefer std::vector and contiguous containers by default. Switch to tree or hash containers only when the operation mix (frequent lookups vs. iteration) justifies the indirection cost, and benchmark the decision.

Copy elision and RVO/NRVO

Since C++17, returning a prvalue from a function is guaranteed zero-copy — no move, no copy, no destructor call. Named Return Value Optimisation (NRVO) extends this to named local variables and is applied by all major compilers at -O1 and above, though it remains technically optional.

cpp
// C++17 — mandatory copy elision (prvalue)
struct BigBuffer {
    int data[4096];
    BigBuffer() { /* expensive initialisation */ }
};

BigBuffer make_buffer() {
    return BigBuffer{};   // C++17: no copy/move — constructed directly in caller's frame
}

BigBuffer buf = make_buffer();  // zero extra construction
cpp
// C++11 — NRVO on named locals (compiler-applied, not guaranteed by the standard)
#include <vector>

std::vector<double> build_primes(int limit) {
    std::vector<double> result;   // NRVO: most compilers elide the move out of this function
    result.reserve(limit / 2);
    for (int i = 2; i < limit; ++i)
        if (/* is_prime(i) */ true) result.push_back(i);
    return result;                // do NOT write std::move(result) — defeats NRVO
}
cpp
// C++17 — guaranteed elision with factory functions and in-place construction
#include <optional>

std::optional<std::vector<int>> try_load(bool ok) {
    if (!ok) [[unlikely]] return std::nullopt;   // C++20 annotation; C++17 optional
    return std::vector<int>(1024, 0);            // guaranteed elision into the optional
}

Never write return std::move(local_var) — it suppresses NRVO and forces a move instead of zero-cost in-place construction. Let the compiler apply elision; add std::move only when returning a function parameter or a member that is no longer needed.

Small string optimization (SSO)

SSO eliminates heap allocation for strings that fit within the object's own footprint — typically 15 bytes on 64-bit platforms. All major standard-library implementations (libstdc++, libc++, MSVC STL) use SSO; short strings never touch the allocator.

cpp
#include <string>
#include <iostream>

// C++11 — SSO in action
std::string short_str = "hello";          // no heap allocation (≤15 chars on most impls)
std::string long_str  = "this string is definitely longer than fifteen characters";
                                          // heap allocation: 1 alloc, 1 free

std::cout << sizeof(std::string) << '\n'; // typically 24 or 32 bytes on 64-bit
cpp
// C++17 — std::string_view for read-only access: zero allocation, zero copy
#include <string_view>

void log_prefix(std::string_view sv) {   // C++17
    // sv never allocates; works with string literals, std::string, char arrays
    if (sv.starts_with("ERR")) [[unlikely]]  // C++20
        /* escalate */ ;
}

// Prefer string_view over const std::string& in function parameters:
// avoids forced heap allocation when a string literal is passed
cpp
// SSO threshold benchmark pattern — confirm your impl's threshold
#include <string>
#include <benchmark/benchmark.h>  // google/benchmark

static void BM_sso_short(benchmark::State& state) {
    for (auto _ : state) {
        std::string s = "hello!";          // SSO — no alloc
        benchmark::DoNotOptimize(s);
    }
}

static void BM_sso_long(benchmark::State& state) {
    for (auto _ : state) {
        std::string s = "this_string_exceeds_sso_limit_definitely";  // heap alloc
        benchmark::DoNotOptimize(s);
    }
}
BENCHMARK(BM_sso_short);
BENCHMARK(BM_sso_long);

Avoid converting std::string_view to std::string inside tight loops. Prefer string_view throughout call chains to keep strings on the SSO fast path or avoid allocation entirely.

Inlining and __attribute__((always_inline))

Inlining eliminates function-call overhead and opens the door to cross-boundary optimisations like constant folding and dead-code elimination. The compiler inlines automatically based on code size and call-site frequency heuristics; force-inlining overrides those heuristics and should be reserved for measured hotspots.

cpp
// C++11 — inline hint (compiler may ignore)
inline int add(int a, int b) { return a + b; }   // C++11

// Compiler-specific force-inline (GCC/Clang/MSVC)
[[gnu::always_inline]] inline int fast_add(int a, int b) { return a + b; }  // GCC/Clang
// __forceinline int fast_add(int a, int b) { return a + b; }               // MSVC
cpp
// C++20 — [[likely]]/[[unlikely]] combined with inlining for accessor patterns
class Counter {
    int value_ = 0;
public:
    [[gnu::always_inline]] inline void increment() noexcept { ++value_; }  // C++11 noexcept
    [[gnu::always_inline]] inline int  get()       const noexcept { return value_; }
};
cpp
// Verify inlining decisions with compiler explorer or nm
// nm -C myapp | grep fast_add     -> no symbol = inlined everywhere
// nm -C myapp | grep fast_add     -> symbol present = at least one call site not inlined

// Prevent inlining (e.g., for profiling landmarks or cold error paths):
[[gnu::noinline]] void cold_error_path() {   // GCC/Clang attribute
    throw std::runtime_error{"unexpected state"};
}

Force-inlining large functions bloats the instruction cache. The I-cache is typically 32 KB per core; filling it with inlined copies of a 200-instruction function leaves less room for the actual hot loop body. Profile I-cache misses (perf stat -e L1-icache-load-misses) before and after applying always_inline to confirm the change helps.

Profiling-guided optimization (PGO) workflow

PGO feeds real execution frequencies back into the compiler so it can make evidence-based decisions about inlining, branch layout, and register allocation. Typical gains on server applications are 10–30% throughput improvement with no source changes — the highest-leverage single build flag available.

bash
# Step 1: instrument binary (GCC)
g++ -O2 -fprofile-generate -o myapp_instr main.cpp

# Step 2: run with representative workload (production traffic replay or load test)
./myapp_instr --workload production_sample.txt

# Step 3: recompile with collected profiles
g++ -O2 -fprofile-use -fprofile-correction -o myapp_pgo main.cpp
# -fprofile-correction: handles profile data from multiple runs / merged profiles
bash
# LLVM/Clang equivalent (recommended for ThinLTO + PGO combination)
clang++ -O2 -fprofile-generate=./pgo_data -o myapp_instr main.cpp
./myapp_instr --workload production_sample.txt
llvm-profdata merge -output=pgo.profdata ./pgo_data/*.profraw
clang++ -O2 -fprofile-use=pgo.profdata -flto=thin -o myapp_pgo main.cpp
bash
# Verify PGO is being applied — compiler warns if profile data is missing/stale:
# warning: profile data may be out of date [-Wprofile-instr-out-of-date]
# Regenerate profiles whenever code structure changes significantly

PGO profiles are workload-specific. A profile captured from a test suite that exercises different hot paths than production will produce a binary optimised for the wrong thing. Always profile with representative traffic, and refresh profiles after major code changes.

Compiler flags reference

Flag selection is the cheapest performance lever — it requires no source changes and consistently yields 10–40% improvement over the default -O0 debug build. Start with -O2 and measure; add flags one at a time and benchmark each change.

bash
# Safe production baseline
-O2                      # inlining, loop unrolling, dead-code elimination
-DNDEBUG                 # disables assert(); removes bounds-check overhead in STL
-fno-omit-frame-pointer  # required by perf, gdb, and sanitisers; ~1% overhead
bash
# Aggressive optimisation (benchmark before enabling in production)
-O3                      # auto-vectorisation, aggressive loop transforms
-march=native            # all CPU features of the build host; not cross-compilable
-march=x86-64-v3         # SSE4.2 + AVX2 + FMA; safe for post-2015 x86 deployments
-march=armv8.2-a+fp16    # ARM equivalent for Neoverse N1/N2 class hardware
bash
# Link-Time Optimisation (cross-TU inlining and dead-code elimination)
-flto=thin               # LLVM ThinLTO: parallel, incremental; use with clang++
-flto                    # GCC LTO: slower link but same quality; use -flto=auto for parallel
bash
# Profile-Guided Optimisation (two-pass build)
-fprofile-generate       # pass 1: instrument — run representative workload after this step
-fprofile-use            # pass 2: apply collected data — 10–30% gain on real workloads
bash
# Floating-point (enable only if numerics correctness is not critical)
-ffast-math              # reassociation, no NaN/Inf checks; breaks Kahan, std::isnan
-ffp-contract=fast       # fused multiply-add (FMA) generation without -ffast-math

# Sanitiser builds (development only — not for production)
-fsanitize=address,undefined   # ASan + UBSan: catches memory errors and UB
-fsanitize=thread              # TSan: data-race detection

Combine -O2 -flto=thin -fprofile-use for the best generic production binary without touching source code. Add -march=native only when the binary will run exclusively on the build host (containers built and run on identical hardware).

See Also

  • Move Semantics — rvalue references, forwarding references, and the Rule of Five
  • RAII — deterministic resource management that eliminates allocation overhead in hot paths
  • Concepts — C++20 constraints that improve inlining opportunities by reducing template instantiation depth
  • Coroutines — C++20 cooperative multitasking without the thread-switch overhead of OS threads