rr — Time-Travel Debugger — bigcpp

TL;DR

rr records a program's entire execution and replays it deterministically — making intermittent bugs trivially reproducible. Replay launches a GDB session with additional reverse-execution commands.

Reverse debugging: reverse-continue, reverse-next, reverse-finish
Linux/x86-64 only (requires hardware performance counters)
Record once, debug forever — heisenbug-proof

bash

rr record ./myapp arg1 arg2   # record execution
rr replay                     # replay in GDB with reverse commands

Installation

bash

# Ubuntu / Debian
sudo apt install rr

# Fedora / RHEL
sudo dnf install rr

# From GitHub releases (latest stable)
wget https://github.com/rr-debugger/rr/releases/download/5.7.0/rr-6.x-linux64.tar.gz
tar xf rr-6.x-linux64.tar.gz
sudo cp rr-6.x/bin/rr /usr/local/bin/

# Verify installation
rr --version

Prerequisite: performance counter access

rr requires access to CPU performance counters. Without this, rr record will fail.

bash

# Temporary (reverts on reboot)
echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid

# Permanent via sysctl
echo 'kernel.perf_event_paranoid=1' | sudo tee /etc/sysctl.d/51-perf-event.conf
sudo sysctl --system

Values: 1 allows user-space access, 2 (default on many distros) blocks it.

Recording

bash

# Basic recording
rr record ./myapp

# With program arguments
rr record ./myapp arg1 arg2

# Separate rr flags from program args with --
rr record -n -- ./myapp --verbose --output=file.log

# -n: disable chaos / ASLR randomization (more reproducible, fewer recordings needed)
rr record -n ./myapp

# Record with chaos mode (randomized scheduling — see below)
rr record --chaos ./myapp

# Follow forks — record child processes too
rr record -f ./myapp

# Set a specific recording directory
rr record -d /tmp/my-recordings ./myapp

Recordings are stored in ~/.local/share/rr/ by default, named <binary>-0, <binary>-1, etc.

Managing recordings

bash

# List all recordings
rr ls

# Show details of a specific recording
rr ls myapp-0

# Pack a recording for transfer to another machine
rr pack latest
rr pack ~/.local/share/rr/myapp-0

# After packing, copy the directory and replay on another Linux x86-64 machine
scp -r ~/.local/share/rr/myapp-0 user@remote:~/

Replaying

bash

# Replay the most recent recording
rr replay

# Replay a specific recording by path
rr replay ~/.local/share/rr/myapp-0

# Replay with extra GDB flags
rr replay -- -ex "break main" -ex "continue"

# Expose a GDB server on a port (for IDE integration)
rr replay -s 1234

# Connect from another GDB instance
gdb ./myapp
(gdb) target remote :1234

Replay launches a GDB session. All standard GDB commands work normally. rr adds reverse-execution commands on top.

Build your binary with debug info for readable backtraces:

bash

g++ -g -O0 -o myapp main.cpp
cmake -DCMAKE_BUILD_TYPE=Debug ..

Reverse Debugging Commands

These are available inside rr replay (not in plain GDB).

cpp

Godbolt

(rr) continue               # run forward to next breakpoint / end of program
(rr) reverse-continue       # rc  — run BACKWARD to previous breakpoint / start
(rr) next                   # step forward, skip over function calls
(rr) reverse-next           # rn  — step BACKWARD, skip over function calls
(rr) step                   # step forward into function calls
(rr) reverse-step           # rs  — step BACKWARD into function calls
(rr) finish                 # run forward until current function returns
(rr) reverse-finish         # run BACKWARD to the call site of current function

Short aliases (same as in GDB, rr honours them):

cpp

Godbolt

(rr) rc     # reverse-continue
(rr) rn     # reverse-next
(rr) rs     # reverse-step

Standard GDB commands that still work as expected in replay:

cpp

Godbolt

(rr) break main             # set breakpoint
(rr) break file.cpp:42
(rr) watch expr             # watchpoint (hardware, very fast in rr)
(rr) info locals            # print local variables
(rr) bt                     # backtrace
(rr) print x               # print variable
(rr) frame N               # switch stack frame
(rr) thread N              # switch thread

Practical Workflow: Catching Intermittent Bugs

Run the program under rr until it crashes or misbehaves.
rr replay — you now have a deterministic reproduction, every time.
Set a breakpoint at the symptom (crash site, assertion, wrong output).
reverse-continue to travel backward in time to the cause.
Repeat until the root cause is found.

bash

# Step 1 — record until the intermittent crash occurs
rr record ./myapp

# Step 2 — reproduce it deterministically
rr replay

# Step 3 — inside GDB, break at the symptom
(rr) break __assert_fail      # assertion failure
(rr) continue                 # run forward to the assertion

# Step 4 — travel back to find why
(rr) reverse-continue         # go back to the previous interesting event
(rr) reverse-continue         # keep going back
(rr) print ptr                # inspect state at each stop

Watchpoints + Reverse = Finding Root Cause

Hardware watchpoints in rr run at near-native speed. Combine them with reverse execution to pinpoint exactly when a memory location was corrupted.

cpp

Godbolt

# Break when a variable is written
(rr) watch -l my_var

# Or watch a raw memory address
(rr) watch -l *(int*)0x7fff1234abcd

# Run forward to first write
(rr) continue

# Now run BACKWARD to find the previous write (the bad one)
(rr) reverse-continue

# Inspect the backtrace at the point of corruption
(rr) bt
(rr) info locals
(rr) print my_var

Watching through a pointer:

cpp

Godbolt

(rr) watch -l *ptr            # fires whenever *ptr changes
(rr) reverse-continue         # find the write that caused the problem

watch -l (location watchpoint) follows the memory address rather than the logical variable, which is more reliable across function calls.

Checkpoints

Checkpoints let you save a position during replay and jump back to it, so you can explore multiple branches of execution without re-recording.

cpp

Godbolt

(rr) checkpoint               # save current replay position
(rr) info checkpoints         # list all saved checkpoints
(rr) restart 1                # jump to checkpoint 1
(rr) restart 0                # jump to checkpoint 0 (start of recording)

Typical use:

cpp

Godbolt

(rr) break Widget::render
(rr) continue                  # reach the function
(rr) checkpoint                # save position here

# Explore one hypothesis
(rr) step
(rr) print this->width_

# Jump back and try another
(rr) restart 1
(rr) print this->data_

Multi-Process / Multi-Thread

rr records all threads in a process. Replay is deterministic regardless of what the scheduler actually did — every thread switch is replayed in the exact same order.

bash

# Record child processes created via fork/exec
rr record -f ./myapp

# Record a program that spawns many threads
rr record ./multithreaded_app

Inside replay, switch between threads normally:

cpp

Godbolt

(rr) info threads             # list all threads
(rr) thread 2                 # switch to thread 2
(rr) bt                       # backtrace of thread 2
(rr) reverse-continue         # reverse-execute in current thread's timeline

All threads share the same recording timeline. Reverse commands respect the interleaved order they were recorded in.

Chaos Mode — Exposing Race Conditions

Chaos mode randomizes thread scheduling to maximize the probability of hitting rare interleavings. Each run is still deterministically replayable once recorded.

bash

# Record with randomized scheduling
rr record --chaos ./myapp

# Record multiple times to hit different interleavings
for i in $(seq 1 20); do
  rr record --chaos ./myapp && echo "Run $i: clean" || echo "Run $i: FAILED"
done

# Replay whichever run crashed
rr ls                          # find the failing recording
rr replay ~/.local/share/rr/myapp-5

Combine with AddressSanitizer for maximum coverage:

bash

g++ -g -O1 -fsanitize=address,undefined -o myapp main.cpp
rr record --chaos ./myapp

Integration with IDEs

VS Code (Native Debug extension)

Install the "Native Debug" extension, then configure a launch entry:

json

{
  "type": "gdb",
  "request": "attach",
  "name": "rr replay",
  "executable": "${workspaceFolder}/build/myapp",
  "target": ":1234",
  "remote": true,
  "cwd": "${workspaceFolder}",
  "valuesFormatting": "parseText"
}

Start the server first:

bash

rr replay -s 1234

Then launch the "rr replay" debug configuration in VS Code.

CLion

CLion's GDB integration works with rr's GDB server:

Run rr replay -s 1234 in a terminal.
In CLion: Run → Attach to Process is not needed; instead use Run → Edit Configurations → GDB Remote Debug.
Set target: localhost:1234, symbol file: path to your binary.
Connect. All CLion stepping and variable inspection works; use the GDB console for reverse commands.

Command-line GDB script

bash

# Automate common startup steps
rr replay -- -ex "break __sanitizer::Die" \
             -ex "break abort" \
             -ex "continue"

Running rr in Docker

Docker's default seccomp profile and dropped capabilities block performance counters. Add the required flags:

bash

docker run \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  my-image \
  rr record ./myapp

Or in a docker-compose.yml:

yaml

services:
  debug:
    image: my-image
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined

Running rr in VMs / KVM

Most hypervisors expose virtual performance counters, but KVM requires explicit enablement:

bash

# On the KVM host — enable PMU passthrough for Intel
sudo modprobe kvm_intel enable_pmu=1

# Or persistently
echo 'options kvm_intel enable_pmu=1' | sudo tee /etc/modprobe.d/kvm-pmu.conf

For AMD:

bash

sudo modprobe kvm_amd avic=1

VMware and VirtualBox: enable "Hardware Performance Counters" / "VT-x/AMD-V" in VM settings. Not all hypervisor versions support this.

Limitations

Constraint	Detail
OS	Linux only (kernel 3.11+)
Architecture	x86-64 primary; x86-32 partial; no ARM, no macOS
Hardware counters	Required; unavailable in most VMs by default
Overhead	2–10× slowdown during recording (acceptable for bug hunting)
Signals	Most signals are recorded correctly; `SIGKILL` terminates recording
I/O	File I/O, network I/O recorded — replay sees the same bytes
GPU	GPU execution not recorded
Self-modifying code	Not supported
32-bit PIE	Partially supported; may require `-n` flag

Comparison: rr vs Traditional Debugging

Problem	Traditional GDB	rr
Intermittent crash	Re-run until it happens again	Record once, replay forever
Heisenbug (changes when observed)	Often impossible to catch	Deterministic — heisenbug-proof
Finding when a variable was corrupted	Manual print tracing or re-runs	`watch -l` + `reverse-continue`
Multi-thread race condition	TSan heuristics, hard to reproduce	Record with `--chaos`, replay exactly
Debugging in production	Attach GDB (intrusive, risky)	Record production run, debug offline
"How did we get here?"	Only forward stepping	`reverse-continue` / `reverse-finish`

Quick Reference Card

bash

# Record
rr record ./app                          # basic
rr record --chaos ./app                  # expose races
rr record -f ./app                       # follow forks
rr ls                                    # list recordings
rr pack latest                           # pack for transfer

# Replay
rr replay                                # most recent
rr replay ~/.local/share/rr/app-0       # specific recording
rr replay -s 1234                        # GDB server on port 1234

# Inside rr replay (GDB session)
continue / c                             # forward
reverse-continue / rc                    # backward
next / n                                 # forward step-over
reverse-next / rn                        # backward step-over
step / s                                 # forward step-into
reverse-step / rs                        # backward step-into
finish                                   # forward to return
reverse-finish                           # backward to call site
watch -l var                             # memory watchpoint
checkpoint                               # save position
info checkpoints                         # list checkpoints
restart N                                # jump to checkpoint N
info threads                             # list threads
thread N                                 # switch thread