Linux perf Profiler

TL;DR

perf stat for overall hardware counter summary. perf record + perf report for function-level hotspots. Flame graphs (via Brendan Gregg's scripts) give the most actionable view.

bash

# Quick summary
perf stat ./myapp

# Record and report
perf record -g ./myapp
perf report

# Flame graph
perf record -F 99 -g ./myapp
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

Installation

bash

# Ubuntu/Debian
sudo apt install linux-perf linux-tools-common linux-tools-$(uname -r)

# Fedora/RHEL
sudo dnf install perf

# Allow non-root profiling (set to -1 for no restriction, 0 for root only)
echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid

perf stat — Hardware Counters

bash

# Basic stats: cycles, instructions, cache misses
perf stat ./myapp

# Output example:
#  3,241,123,456  cycles
#  8,012,345,678  instructions              #  2.47  insns per cycle
#     12,345,678  cache-references
#        123,456  cache-misses              #    1.00% of all cache refs
#      1,234,567  branch-misses

# Specific events
perf stat -e cycles,instructions,L1-dcache-misses,LLC-misses ./myapp

# Compare multiple runs
perf stat -r 5 ./myapp  # repeat 5 times, show std deviation

# All events
perf list                # list all available events
perf stat -e 'cache-misses,cache-references,branch-misses,page-faults' ./myapp

perf record / report — Sampling

bash

# Record with call graph (frame pointer)
perf record -g ./myapp

# Record with DWARF call graph (more accurate, more data)
perf record --call-graph dwarf ./myapp

# Record at specific frequency
perf record -F 1000 -g ./myapp  # 1000 samples/sec

# Record a running process
perf record -F 99 -g -p <PID> -- sleep 30

# Record all CPUs
perf record -F 99 -g -a -- sleep 10

# Interactive report
perf report

# Report to stdout
perf report --stdio

# Show top functions
perf report --sort=dso,sym --stdio | head -50

cpp

Godbolt

# Interactive TUI keys:
Enter     — expand/collapse call tree
d         — annotate with source/assembly
?         — help
q         — quit
A         — toggle annotation
s         — sort order
z/Z       — zoom in/out

perf annotate — Source + Assembly

bash

# Annotate source with hot instructions
perf annotate --stdio -s my_function

# Or in report TUI: press 'a' on a function

cpp

Godbolt

# Output shows percentage of time per instruction:
     0.52 :  400ca0:  mov    (%rdi),%eax
    12.38 :  400ca3:  add    %eax,%esi    ← hot instruction
     0.11 :  400ca5:  mov    %esi,(%rdi)

perf diff — Compare Two Profiles

bash

perf record -o before.data ./myapp_before
perf record -o after.data ./myapp_after

perf diff before.data after.data
# Shows % change per symbol

Flame Graphs

bash

# Install Brendan Gregg's flamegraph scripts
git clone https://github.com/brendangregg/FlameGraph.git
export PATH=$PATH:./FlameGraph

# Record
perf record -F 99 -g ./myapp -- args

# Generate flame graph
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg
firefox flame.svg

# Off-CPU flame graph (time waiting, not running)
perf record -e sched:sched_switch -ag ./myapp
perf script | stackcollapse-perf.pl | flamegraph.pl --color=io > offcpu.svg

Specific Use Cases

Cache Miss Analysis

bash

perf stat -e \
  L1-dcache-loads,L1-dcache-load-misses,\
  L1-icache-loads,L1-icache-load-misses,\
  LLC-loads,LLC-load-misses \
  ./myapp

Branch Misprediction

bash

perf stat -e branches,branch-misses ./myapp
perf record -e branch-misses -g ./myapp
perf report

Memory Bandwidth

bash

perf stat -e \
  uncore_imc/data_reads/,\
  uncore_imc/data_writes/ \
  ./myapp

Syscall Tracing

bash

# Count syscalls
perf stat -e 'syscalls:sys_enter_*' ./myapp

# Trace specific syscall
perf trace -e mmap ./myapp

# All system calls with latency
perf trace --summary ./myapp

Build Flags for Profiling

bash

# Keep symbols, enable call graphs
g++ -O2 -g -fno-omit-frame-pointer -o myapp myapp.cpp

# For accurate DWARF call graphs
g++ -O2 -g -gdwarf-4 -fno-optimize-sibling-calls -o myapp myapp.cpp

cmake

# CMake
add_compile_options(-fno-omit-frame-pointer)
set(CMAKE_BUILD_TYPE RelWithDebInfo)  # O2 + debug info

perf vs Other Profilers

Tool	Mechanism	Overhead	Granularity
perf	Hardware PMU sampling	Very low	Function + line
Valgrind/callgrind	Instrumentation	~10-50x	Every instruction
gprof	Instrumented + sampling	Low	Function
Heaptrack	Heap allocation tracking	Low	Allocation site
Intel VTune	Hardware PMU	Very low	Microarchitecture
Tracy	Manual instrumentation	Very low	Custom zones

Common Workflows

bash

# 1. Find CPU hotspot
perf record -g ./myapp && perf report

# 2. Check cache efficiency
perf stat -e cache-misses,cache-references ./myapp

# 3. Understand where time goes (flame graph)
perf record -F 99 -g ./myapp
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

# 4. Compare before/after optimization
perf record -o v1.data ./myapp_v1
perf record -o v2.data ./myapp_v2
perf diff v1.data v2.data

# 5. Identify lock contention
perf lock record ./myapp
perf lock report

perf top gives you a real-time flame-sorted view of the running system without writing any data file. It is the fastest way to answer "what is burning CPU right now?" — useful when you cannot restart the workload or when you just need a quick sanity check before committing to a full perf record session.

bash

# System-wide live view (requires perf_event_paranoid <= 1)
perf top

# Attach to a specific process
perf top -p <PID>

# Watch a single hardware event instead of cycles
perf top -e cache-misses

# Higher sampling rate for short-lived hot paths
perf top -F 4000

Key TUI bindings while perf top is running:

cpp

Godbolt

s         — cycle sort order (overhead / shared object / symbol)
+         — expand a call-chain entry
z         — toggle zero-count entries
a         — annotate the selected symbol with assembly
q         — quit

Use perf top for quick live inspection; switch to perf record when you need reproducible data, flame graphs, or perf diff comparisons.

perf probe — Dynamic Tracepoints

Add a tracepoint to any function or source line in a running binary — no recompilation required. perf probe writes a kernel uprobe on the target address, so you pay zero overhead when the probe is inactive and only instrument exactly the code path you care about.

bash

# Add a tracepoint on function entry
perf probe -x ./myapp function_name

# Tracepoint at line 5 of foo(), capturing local variables x and y
perf probe -x ./myapp 'foo:5 x y'

# Record only hits on that tracepoint (with call graph)
perf record -e probe:function_name -g ./myapp

# Dump each hit with captured variable values
perf script

# List active probes
perf probe -l

# Remove all probes when done
perf probe -d '*'

The binary must have DWARF debug info (-g) for variable capture to work. For production binaries stripped of debug symbols, tracepoints can still fire on function entry — you just lose variable visibility.

Intel PT — Instruction-Level Tracing

Intel PT records the complete control flow of a program in hardware with near-zero overhead. Unlike sampling-based profilers, it captures every branch taken, so you get exact call sequences rather than statistical approximations — invaluable for intermittent bugs and precise hotpath analysis.

Check availability first:

bash

perf list | grep intel_pt
# Should show: intel_pt// [Kernel PMU event]

bash

# Full instruction trace (data can grow large — use with short workloads)
perf record -e intel_pt// ./myapp

# Decode with time-based sampling: one snapshot every 100 ns
perf script --itrace=i100ns

# Branches and errors only (compact output)
perf script --itrace=be

# Calls and returns (reconstructs the call graph exactly)
perf script --itrace=cr

# Snapshot mode: only keep the last N MiB of trace (caps file size)
perf record -e intel_pt// --snapshot ./myapp

Intel PT output can be enormous for long-running workloads. Use --snapshot or --itrace sampling intervals to keep the decoded output manageable, and prefer perf script --itrace=cr when you only need the call graph.

perf lock — Lock Contention Analysis

Pinpoint exactly which mutexes are serializing your threads without the ~5–20x overhead of ThreadSanitizer. perf lock uses kernel tracepoints on pthread_mutex_* and futex calls, so it captures real contention under production load.

bash

# Record lock events
perf lock record ./myapp

# Show per-lock summary
perf lock report

Example output columns:

cpp

Godbolt

Name              acquired  contended  total-wait   avg-wait   max-wait
work_queue_mutex     98432       4821    12.341 ms   2.559 us  18.234 us
stats_mutex          12100         23     0.041 ms   1.783 us   3.102 us

Key columns to focus on:

contended — how many acquisitions had to wait (high number = hot lock)
total-wait — cumulative time threads were blocked on this lock
avg-wait — average wait per contention event (high value = long critical section)

Locks with high contended counts but low avg-wait suggest high frequency of short critical sections — candidates for a read-write lock or lock-free structure. High avg-wait points to work that should be moved out of the critical section.

perf mem — Memory Access Profiling

perf mem uses PEBS (Precise Event-Based Sampling) to attribute each sampled memory load or store to its exact source in the cache hierarchy. This lets you find data structures that are cache-cold or that create excessive DRAM traffic, without guessing from cache-miss counters alone.

PEBS requires a modern Intel or AMD CPU (Zen 2+). Check support:

bash

perf list | grep -i pebs
# Or simply try the record step; perf will error if unsupported

bash

# Record memory accesses
perf mem record ./myapp

# Report with latency and cache-level breakdown
perf mem report

The report groups samples by symbol and annotates each with the memory level that satisfied the access:

cpp

Godbolt

Overhead  Symbol               Local weight  Mem level
  34.2%   process_buffer              142    L3 miss (RAM)
  21.7%   hash_lookup                  38    L2 hit
   9.1%   update_counters               4    L1 hit

High "L3 miss (RAM)" overhead on a hot function is a direct signal to improve data locality — consider structure-of-arrays layout, prefetching, or reducing working-set size. Pair perf mem findings with perf annotate to see which exact load/store instructions are responsible.

Linux perf Profiler

TL;DR

Installation

perf stat — Hardware Counters

perf record / report — Sampling

perf report Navigation

perf annotate — Source + Assembly

perf diff — Compare Two Profiles

Flame Graphs

Specific Use Cases

Cache Miss Analysis

Branch Misprediction

Memory Bandwidth

Syscall Tracing

Build Flags for Profiling

perf vs Other Profilers

Common Workflows

perf top — Live Profiling

perf probe — Dynamic Tracepoints

Intel PT — Instruction-Level Tracing

perf lock — Lock Contention Analysis

perf mem — Memory Access Profiling

See Also