Profiling C++ — Tracy, perf, Valgrind — bigcpp

Choosing a profiler

Tool	Platform	Overhead	Best for
Tracy	Win/Mac/Linux	Very low	Games, real-time, frame-level
perf	Linux	Very low	Production systems, kernel
Valgrind/Callgrind	Linux/Mac	20–100×	Exact instruction counts
gprof	Any (GCC/Clang)	Low	Quick baseline profiling
Instruments	macOS	Low	Apple platform profiling
Intel VTune	Win/Linux	Low	Deep CPU microarch analysis
Heaptrack	Linux	Low	Memory allocation profiling

Tracy — frame-level profiler

Tracy is an instrumented profiler designed for games and real-time apps. It streams timing data to a companion GUI in real time.

Setup (CMake)

cmake

include(FetchContent)
FetchContent_Declare(tracy
    GIT_REPOSITORY https://github.com/wolfpld/tracy.git
    GIT_TAG v0.10
    GIT_SHALLOW TRUE
)
FetchContent_MakeAvailable(tracy)

target_link_libraries(myapp PRIVATE TracyClient)
target_compile_definitions(myapp PRIVATE TRACY_ENABLE)

Instrumentation

cpp

Godbolt

#include <tracy/Tracy.hpp>

void update_physics(float dt) {
    ZoneScoped;  // automatic zone — function name + file/line
    // ... physics work
}

void render_frame() {
    ZoneScopedN("RenderFrame");  // custom zone name

    {
        ZoneScopedN("DrawMeshes");
        draw_all_meshes();
    }
    {
        ZoneScopedN("PostProcess");
        apply_bloom();
        apply_tonemapping();
    }

    FrameMark;  // marks end of frame — Tracy shows FPS graph
}

void on_asset_load(std::string_view name) {
    ZoneScopedN("LoadAsset");
    ZoneText(name.data(), name.size());  // attach string to zone
    // ...
}

// Locks (shows contention in Tracy's lock view)
TracyLockable(std::mutex, mtx);
std::lock_guard lock(mtx);

// Memory tracking
void* ptr = malloc(size);
TracyAlloc(ptr, size);
// ...
TracyFree(ptr);
free(ptr);

Build release with Tracy

bash

cmake -DCMAKE_BUILD_TYPE=Release \
      -DTRACY_ENABLE=ON \
      -B build && cmake --build build

# Run the Tracy server GUI in another terminal
./tracy-profiler
# Then run your app — connects automatically on localhost:8086

Linux perf — statistical profiling

perf samples the CPU program counter at high frequency — no instrumentation needed.

CPU profiling

bash

# Record: sample at 99 Hz for 30 seconds
perf record -g -F 99 -p $(pgrep myapp) -- sleep 30

# Or record entire run:
perf record -g -F 99 ./myapp

# Report: top functions
perf report

# Flamegraph (install FlameGraph scripts from Brendan Gregg)
perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > flamegraph.svg

Hardware performance counters

bash

# Cache misses
perf stat -e cache-misses,cache-references,instructions,cycles ./myapp

# Branch mispredictions
perf stat -e branch-misses,branches ./myapp

# LLC misses (last-level cache — main cause of memory stalls)
perf stat -e LLC-loads,LLC-load-misses,LLC-stores ./myapp

# Memory bandwidth
perf stat -e mem-loads,mem-stores ./myapp

perf annotate — see which instructions are hot

bash

perf record -g ./myapp
perf annotate my_hot_function
# Shows per-instruction cycle counts in the source

Valgrind Callgrind — instruction-level profiling

Callgrind runs the program under a virtual CPU — no sampling, exact counts. 20–100× slower.

bash

# Build with debug info (but optimize for real performance)
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -B build && cmake --build build

# Profile
valgrind --tool=callgrind --callgrind-out-file=callgrind.out ./myapp

# View in KCachegrind (GUI) or:
callgrind_annotate callgrind.out | head -100

Callgrind with cache simulation

bash

# Count cache misses per function (L1, L2, LLC)
valgrind --tool=callgrind \
         --simulate-cache=yes \
         --callgrind-out-file=callgrind.out \
         ./myapp

callgrind_annotate --auto=yes callgrind.out

Cachegrind (standalone cache analysis)

bash

valgrind --tool=cachegrind ./myapp
cg_annotate cachegrind.out.PID
# Reports: I1mr, LLi, D1mr, LLd (instruction/data L1 and LL miss rates)

gprof — quick baseline (compile-time)

bash

# Compile with profiling flags
g++ -O2 -pg -o myapp main.cpp

# Run (generates gmon.out)
./myapp

# Generate report
gprof myapp gmon.out | head -50

Sample output:

cpp

Godbolt

Flat profile:
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 45.20      1.23     1.23    10000     0.12     0.18  compute_fft
 20.30      1.78     0.55   500000     0.00     0.00  update_particles

Limitation: can't profile shared libraries; inlining distorts results; use perf or Tracy for production.

Heaptrack — allocation profiling

bash

# Find what's allocating memory and causing fragmentation
heaptrack ./myapp
heaptrack_gui heaptrack.myapp.XXXXX.gz

# Or in terminal:
heaptrack_print heaptrack.myapp.XXXXX.gz | head -30

Shows: peak memory, allocation call stacks, allocation hotspots.

Reading a flamegraph

cpp

Godbolt

main ─────────────────────────────────────────────────── 100%
  update ──────────────────────────── 70%
    physics ─────────── 45%          │
      broad_phase ── 30%             │
      narrow_phase ─ 15%             │
    ai ──────────── 25%              │
  render ─────────────── 30%
    shadow_pass ── 15%
    geometry ───── 10%
    post ──────────  5%

Width = time spent (including children)
Flat top = self time — where CPU actually executed
A wide, flat top bar is a hotspot to optimize
A tall narrow tower = deep call chain, look at the top

Profiling checklist

Build with optimizations (-O2/-O3) and debug info (-g or RelWithDebInfo) — profiling a debug build is misleading
Profile a realistic workload — synthetic microbenchmarks miss real-world bottlenecks
Identify the bottleneck type first — CPU bound? Memory bound? I/O bound?
Start at the top — optimize the largest hotspot first (Amdahl's law)
Measure after each change — never assume an optimization helped; verify
Check cache misses — often the real cause of "slow CPU" code
Look for unexpected allocations — new/delete in hot paths kills cache

Google Benchmark — Microbenchmarking

Flamegraphs tell you where time goes; Google Benchmark tells you exactly how fast a function runs. Use it to measure isolated routines before and after optimization, with enough statistical rigor to trust the result.

CMake integration

cmake

include(FetchContent)
FetchContent_Declare(googlebenchmark
    GIT_REPOSITORY https://github.com/google/benchmark.git
    GIT_TAG        v1.8.3
    GIT_SHALLOW    TRUE
)
set(BENCHMARK_ENABLE_TESTING OFF)
FetchContent_MakeAvailable(googlebenchmark)

target_link_libraries(mybench PRIVATE benchmark::benchmark)

Writing benchmarks

cpp

Godbolt

#include <benchmark/benchmark.h>
#include <vector>
#include <numeric>

// Basic benchmark
static void BM_VectorSum(benchmark::State& state) {
    std::vector<int> v(state.range(0), 1);
    for (auto _ : state) {
        // benchmark::DoNotOptimize prevents the compiler from eliminating
        // the result as dead code — essential for benchmarking pure functions
        benchmark::DoNotOptimize(std::accumulate(v.begin(), v.end(), 0));
    }
    state.SetItemsProcessed(state.iterations() * state.range(0));
}
// Log-scale range: 8, 64, 512, 4096, 8192 elements
BENCHMARK(BM_VectorSum)->Range(8, 8 << 10);

// Benchmark with setup state (not timed)
static void BM_SortVec(benchmark::State& state) {
    std::vector<int> src(state.range(0));
    std::iota(src.rbegin(), src.rend(), 0);  // worst-case descending
    for (auto _ : state) {
        auto v = src;  // copy is inside loop — included in time
        std::sort(v.begin(), v.end());
        benchmark::ClobberMemory();  // force write-back; pair with DoNotOptimize
    }
    state.SetBytesProcessed(
        static_cast<int64_t>(state.iterations()) * state.range(0) * sizeof(int));
}
BENCHMARK(BM_SortVec)->RangeMultiplier(4)->Range(64, 64 << 10);

BENCHMARK_MAIN();

state.SetBytesProcessed and state.SetItemsProcessed make Google Benchmark report throughput (GB/s or items/s) automatically — much more useful than raw nanoseconds when comparing implementations.

Running benchmarks

bash

# Build
cmake -DCMAKE_BUILD_TYPE=Release -B build && cmake --build build

# Run all
./build/mybench

# Filter to a specific benchmark
./build/mybench --benchmark_filter=BM_VectorSum

# Repeat 10 times and report mean/stddev
./build/mybench --benchmark_repetitions=10

# Save results for tracking over time
./build/mybench \
    --benchmark_out=results.json \
    --benchmark_out_format=json

# Compare two JSON files (requires benchmark_tools)
compare.py benchmarks results_before.json results_after.json

Always build benchmarks with -O2/-O3 and LTO disabled; LTO can inline across benchmark boundaries and distort results. Run on an isolated core (taskset -c 3 ./mybench) and disable CPU frequency scaling (cpupower frequency-set -g performance) for reproducibility.

Intel VTune Profiler

VTune sees inside the CPU in ways that perf cannot — DRAM bandwidth, NUMA effects, front-end vs back-end bound stalls, and the exact reason for a branch misprediction. It is free for non-commercial use as part of Intel oneAPI.

Hotspot collection

bash

# Collect hotspots (equivalent to perf record, but with richer data)
vtune -collect hotspots -r ./vtune_result -- ./myapp

# Open the GUI
vtune-gui ./vtune_result

# Or generate a text summary
vtune -report summary -r ./vtune_result

The GUI provides four complementary views: Flame Graph (same visual idiom as perf's flamegraph SVG), Bottom-Up (flat self-time ranking), Call Stack (aggregated call trees), and Source (annotated source lines with stall counts).

Microarchitecture exploration

bash

# Breaks down stall budget: front-end bound, back-end bound,
# retiring (good), bad speculation (branch mispredictions)
vtune -collect uarch-exploration -r ./vtune_uarch -- ./myapp
vtune -report summary -r ./vtune_uarch

The Top-Down Microarchitecture Analysis (TMA) output classifies every CPU cycle into one of four buckets. If your app is back-end bound / memory bound, optimize data layout (SoA, cache alignment). If it is front-end bound / fetch latency, look at iTLB pressure or branch target buffer (BTB) thrashing from large binaries.

Memory access profiling

bash

# Attribute cache misses to specific load/store instructions
vtune -collect memory-access -r ./vtune_mem -- ./myapp
vtune -report top-down -r ./vtune_mem

VTune's memory access report shows L1/L2/LLC hit rates and DRAM bandwidth per function — far more precise than perf stat -e cache-misses, which gives only global totals.

Sampling vs Instrumentation — When to Use Each

Every profiler makes the same trade-off: low overhead with approximate data (sampling) or exact data with significant overhead (instrumentation). Understanding this distinction prevents misinterpreting results.

Profiler	Type	Overhead	Data quality	Best for
perf	Sampling	< 1%	Approximate (probabilistic)	Production systems, broad overview
VTune (hotspots)	Sampling	< 5%	Approximate + HW counters	CPU microarch analysis
Instruments (macOS)	Sampling	< 2%	Approximate	Apple platform profiling
Tracy (statistical)	Sampling	Very low	Approximate	Real-time / games overview
gprof	Instrumentation	Low–medium	Exact call counts	Quick compile-time baseline
Callgrind	Instrumentation	20–100×	Exact instruction counts	Algorithmic analysis, CI regression
Tracy (`ZoneScoped`)	Instrumentation	Low (ring buffer)	Exact zone timing	Frame-level game profiling

The practical workflow: start with a sampling profiler to identify the hot 5% of the codebase. Then switch to Callgrind or Tracy zones to understand that 5% precisely. Running Callgrind on the whole program first is wasteful; running perf on a well-instrumented game loop misses the detail you need.

Why 99 Hz and not 100 Hz? Sampling at a round frequency (100 Hz = every 10 ms) risks synchronizing with OS scheduler ticks (typically 10 ms on Linux with CONFIG_HZ=100) or your own timer callbacks. A prime-adjacent rate like 99 Hz ensures samples land at pseudo-random phases of periodic work, giving an unbiased estimate.

NUMA and Memory Access Patterns

On multi-socket servers, remote NUMA memory accesses can be 2–3× slower than local accesses — and they are invisible unless you explicitly look for them. A program that runs fine on a laptop may stall badly on a 2P server.

Inspect NUMA topology

bash

# Show NUMA nodes, CPUs per node, memory per node
numactl --hardware

# Pin process to node 0's CPUs and memory (eliminates remote accesses for testing)
numactl --cpunodebind=0 --membind=0 ./myapp

# Compare: run on node 0 CPUs but allocate from node 1 (artificial worst case)
numactl --cpunodebind=0 --membind=1 ./myapp

Detect remote NUMA traffic with perf

bash

# Count outstanding remote NUMA data reads (Intel-specific PMU event)
perf stat -e OFFCORE_REQUESTS_OUTSTANDING.ALL_DATA_RD ./myapp

# Detect false sharing — two threads writing to different vars on the same cache line
perf c2c record -g -- ./myapp
perf c2c report

False sharing is a common NUMA-adjacent bug: two threads each own a variable, but the variables share a 64-byte cache line, causing the cache line to ping-pong between cores. Fix it with alignment:

cpp

Godbolt

// Bad: x and y likely share a cache line
struct Counters {
    std::atomic<int> x;
    std::atomic<int> y;
};

// Good: each variable occupies its own cache line
struct alignas(64) AlignedCounter {
    std::atomic<int> value;
    char pad[60];  // or: std::byte pad[64 - sizeof(std::atomic<int>)];
};

AlignedCounter counters[NUM_THREADS];

alignas(64) guarantees 64-byte alignment matching a typical x86 cache line. For data shared across NUMA nodes, prefer first-touch allocation policies: allocate from within the thread that will use the data, so the OS places pages on the local NUMA node.

Profiling in Production (Continuous Profiling)

Reproducing a performance problem in development is often impossible — the production workload, memory layout, and concurrency pattern are too different. Continuous profiling keeps a low-overhead sampling profiler running permanently and captures profiles on demand.

Linux perf in production

bash

# Record a 10-second profile of a running process every 5 minutes (cron job)
perf record -g -F 49 -p $(pgrep myapp) -- sleep 10 -o /var/profiles/$(date +%s).data

# On capture, convert immediately to avoid large binary files
perf script -i /var/profiles/latest.data | \
    stackcollapse-perf.pl | \
    flamegraph.pl > /var/www/profiles/latest.svg

Use 49 Hz in production (even lower than the development 99 Hz) to keep CPU overhead under 0.5%. Store profiles in a ring buffer directory and rotate with logrotate.

eBPF-based continuous profiling

eBPF profilers run entirely in the kernel with near-zero overhead and require no process restart:

bash

# Parca agent (open-source, Kubernetes-native)
# Deploys as a DaemonSet; profiles all pods automatically
kubectl apply -f https://github.com/parca-dev/parca-agent/releases/latest/download/kubernetes-manifest.yaml

# Pixie (CNCF project) — also captures network, DB queries, CPU profiles
px deploy
px run px/cpu_flamegraph

For non-Kubernetes deployments, bpftrace can capture stack traces from any process without instrumentation:

bash

# Sample all threads of PID 1234 at 99 Hz, write folded stacks
bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack] = count(); }' \
    --format=json -o stacks.json

When NOT to use continuous profiling

Embedded / bare-metal systems: no OS scheduler to hook; sampling profilers require timer interrupts or debug hardware (ETM/ETB on ARM Cortex-M).
Hard real-time: even a < 1% overhead is intolerable when deadlines are in microseconds; use hardware performance counters read at task boundaries instead (PMU_EVENTSELx on x86, PMCCNTR_EL0 on ARMv8).
Security-sensitive environments: /proc/PID/mem access required by some profilers may be blocked by SELinux/AppArmor policies.

Ring-buffer capture on signal

A common production pattern: the profiler always writes to a fixed-size ring buffer in shared memory. On SIGUSR1, the process snapshots the buffer to disk without stopping:

cpp

Godbolt

// Rough sketch — production implementations use lock-free ring buffers
static RingBuffer<ProfileEvent, 65536> g_profile_buf;

void on_sigusr1(int) {
    // signal-safe: write to pre-opened fd, no malloc
    g_profile_buf.flush_to_fd(g_profile_fd);
}
// ...
std::signal(SIGUSR1, on_sigusr1);

This is the approach used by Tracy's built-in on-demand capture and by Google's internal profiling infrastructure.