Small Buffer Optimization

Overview

Every call to operator new carries cost: the allocator must locate a free block, update bookkeeping, and return a pointer that the CPU's cache likely does not hold. For containers whose typical payload is small — a short string, a lightweight callable, a pointer-sized variant — that cost dominates runtime.

Small Buffer Optimization sidesteps allocation on the common path by embedding a raw byte array inside the containing object. The type maintains a discriminant — often a single pointer that is null on the inline path — to know whether live data resides in the buffer or on the heap. The public interface is identical in both cases; only the internal storage path differs.

The C++ standard library uses this technique pervasively. Every conforming std::string implementation employs Short String Optimization (SSO), a direct specialization of SBO that typically stores up to 15 or 22 bytes without touching the heap. Since C++11, std::function maintains an internal buffer for small callable objects; moving a lambda that captures a single pointer never allocates. Since C++17, std::any is explicitly encouraged by the standard to avoid dynamic allocation for small, trivially-destructible types — std::is_nothrow_move_constructible_v<T> being true is the usual gate.

The tradeoff is object size. The inline buffer inflates sizeof of the owning class regardless of whether the buffer is ever used. For objects stored densely in containers, oversizing the buffer hurts cache utilization. The correct buffer size is workload-specific and should be driven by profiling.

Syntax

The structural skeleton requires four components: an aligned raw buffer, an optional heap pointer that doubles as an inline/heap discriminant, a vtable or discriminant tag, and correct handling of copy, move, and destruction on both paths.

#include <cstddef>
#include <memory>
#include <new>
#include <type_traits>
#include <utility>

template <std::size_t InlineCapacity = 32>
class SBOStorage {
public:
    SBOStorage() noexcept = default;

    template <typename T>
    explicit SBOStorage(T&& value) {
        using U = std::decay_t<T>;
        static_assert(std::is_copy_constructible_v<U>); // C++17 trait
        if constexpr (fits_inline<U>()) {               // C++17 if constexpr
            ::new (buf_) U(std::forward<T>(value));
        } else {
            heap_ = ::new U(std::forward<T>(value));
        }
    }

    ~SBOStorage() {
        ::operator delete(heap_); // no-op when null
    }

private:
    template <typename T>
    static constexpr bool fits_inline() noexcept {
        return sizeof(T) <= InlineCapacity &&
               alignof(T) <= alignof(std::max_align_t);
    }

    // alignas ensures safe placement new for any scalar type
    alignas(std::max_align_t) char buf_[InlineCapacity]{};
    void* heap_ = nullptr; // null ↔ inline path active
};

Using alignas(std::max_align_t) on the buffer satisfies alignment requirements for every fundamental type. For types with extended alignment (e.g., __m256), you need an explicit alignas(alignof(T)) check and either reject them or allocate on the heap. std::aligned_storage was deprecated in C++23; prefer the alignas char array directly.

Examples

Type-erased value container

Below is a production-viable pattern: a non-owning vtable handles type-specific operations, while storage dispatch is decided at construction time via if constexpr.

#include <cassert>
#include <cstddef>
#include <new>
#include <type_traits>
#include <utility>

template <std::size_t N = 48>
class AnySmall {
    struct Ops {
        void (*destroy)(void*) noexcept;
        void (*copy_to)(void* dst, const void* src);
        void (*move_to)(void* dst, void* src) noexcept;
    };

    template <typename T>
    static const Ops* ops_for() noexcept {
        static constexpr Ops kOps{
            [](void* p) noexcept { static_cast<T*>(p)->~T(); },
            [](void* dst, const void* src) {
                ::new (dst) T(*static_cast<const T*>(src));
            },
            [](void* dst, void* src) noexcept {
                ::new (dst) T(std::move(*static_cast<T*>(src)));
                static_cast<T*>(src)->~T();
            },
        };
        return &kOps;
    }

    template <typename T>
    static constexpr bool inline_ok() noexcept {
        return sizeof(T) <= N && alignof(T) <= alignof(std::max_align_t) &&
               std::is_nothrow_move_constructible_v<T>; // C++11 trait
    }

    alignas(std::max_align_t) char buf_[N]{};
    void*      heap_  = nullptr;
    const Ops* ops_   = nullptr;

    void* live() noexcept { return heap_ ? heap_ : static_cast<void*>(buf_); }
    const void* live() const noexcept {
        return heap_ ? heap_ : static_cast<const void*>(buf_);
    }

public:
    AnySmall() = default;

    template <typename T, typename = std::enable_if_t< // C++11 SFINAE
                  !std::is_same_v<std::decay_t<T>, AnySmall>>>
    explicit AnySmall(T&& v) {
        using U = std::decay_t<T>;
        ops_ = ops_for<U>();
        if constexpr (inline_ok<U>()) {
            ::new (buf_) U(std::forward<T>(v));
        } else {
            heap_ = ::new U(std::forward<T>(v));
        }
    }

    AnySmall(AnySmall&& o) noexcept : ops_(o.ops_) {
        if (!ops_) return;
        if (o.heap_) {
            heap_   = o.heap_;
            o.heap_ = nullptr;
        } else {
            ops_->move_to(buf_, o.buf_); // O(N), not O(1)
        }
        o.ops_ = nullptr;
    }

    AnySmall(const AnySmall& o) : ops_(o.ops_) {
        if (!ops_) return;
        if (o.heap_) {
            // allocate raw storage then copy-construct in place
            heap_ = ::operator new(/* sizeof(T) unknown here — use vtable size field */0);
            ops_->copy_to(heap_, o.heap_);
        } else {
            ops_->copy_to(buf_, o.buf_);
        }
    }

    ~AnySmall() {
        if (ops_) ops_->destroy(live());
        ::operator delete(heap_);
    }
};

In practice, the vtable is extended with a size and align field so the copy constructor can call ::operator new(size) with the correct size. Libraries like llvm::Any and folly::Function follow exactly this pattern.

Observing SSO in std::string

#include <iostream>
#include <string>

int main() {
    std::string small = "hello";              // no heap allocation on libstdc++/libc++
    std::string large(64, 'x');              // heap allocated

    // Capacity at which heap kicks in is implementation-defined.
    // On libstdc++ (GCC) the threshold is 15 bytes for char.
    // On libc++ (Clang) it is typically 22 bytes.
    std::cout << small.capacity() << '\n';   // often 15 or 22
    std::cout << large.capacity() << '\n';   // >= 64
}

std::function and small callables (C++11)

#include <functional>

void register_handler(std::function<void(int)> f); // C++11

int main() {
    int offset = 10;
    // Single captured int: fits inline, no allocation.
    register_handler([offset](int x) { return x + offset; });

    // Capturing a large struct forces heap fallback.
    struct Big { char data[256]; };
    Big b{};
    register_handler([b](int x) { (void)b; return x; }); // heap allocated
}

Best Practices

Size the buffer from data, not intuition. Profile the distribution of payload sizes before choosing N. A buffer of 48–64 bytes covers the common case in most event systems and callback registries without excessive stack waste.

Gate inline storage on is_nothrow_move_constructible. If moving the inline object can throw, a move of the container becomes potentially throwing, which breaks many standard-library guarantees. std::any uses this gate explicitly.

Extend the vtable with size and alignment. A vtable entry for alloc_size and alloc_align lets the copy path allocate correctly without knowing the concrete type. This avoids storing a separate type-erased sizeof.

Mark move operations noexcept when the inline path is always noexcept. Containers such as std::vector choose between move and copy during reallocation based on noexcept on the move constructor. Failing to propagate noexcept correctly degrades push_back from O(1) amortized to O(n).

Common Pitfalls

Assuming moves are O(1). When the inline buffer is active, a move must memcpy or move-construct the buffer contents. There is no pointer to swap. Code that benchmarks moves expecting constant time will find linear-in-buffer-size performance on the inline path.

Alignment UB. Storing a double (8-byte aligned) into a buffer declared as char buf_[N] without alignas is undefined behavior. Always use alignas(std::max_align_t) for a general-purpose buffer, or alignas(alignof(T)) when the stored type is known at compile time.

Double-free on self-move. Naive operator=(AnySmall&& o) that frees heap_ then copies o.heap_ is safe only if this != &o. Self-move must be explicitly guarded or handled by the standard swap idiom.

Forgetting to nullify after move. After transferring heap_ to the new owner, the source must set heap_ = nullptr. Otherwise its destructor will call ::operator delete on memory it no longer owns.

Buffer too large in hot paths. A 256-byte SBO buffer on a type that lives in a std::vector<T> means 256 bytes per element even when all values are small. The resulting cache pressure can erase the allocation savings entirely. Prefer the smallest buffer that covers the p95 payload.

std::inplace_vector (C++26)

C++26 introduces std::inplace_vector<T, N> (paper P0843), a vector-like container whose entire storage lives inside the object itself — no heap allocation is ever performed, not even as a fallback. This is distinct from SBO containers like boost::container::small_vector, which spill to the heap when the inline capacity is exceeded: inplace_vector hard-caps its size at N and throws std::bad_alloc (or calls std::terminate for the _unchecked variants) if you attempt to push past capacity.

#include <inplace_vector> // C++26

// A fixed-capacity ring of recent events; lives entirely on the stack.
std::inplace_vector<int, 8> recent; // C++26

recent.push_back(1);
recent.push_back(2);
recent.emplace_back(3);

// API is a strict subset of std::vector.
for (int v : recent) { /* ... */ }

static_assert(recent.max_size() == 8); // compile-time constant

// Attempting to exceed capacity throws std::bad_alloc.
// recent.resize(9); // throws

// _unchecked variants are UB / terminate on overflow — use in
// performance-critical paths after a bounds check.
if (recent.size() < recent.capacity())
    recent.push_back_unchecked(4); // C++26, no exception overhead

Because max_size() == N is a compile-time constant, the compiler can prove that no allocation path exists and generate tighter code than for a heap-capable container. The primary use cases are stack-allocated command queues, fixed-depth undo buffers, and embedded-systems code where operator new is forbidden. When the maximum size truly is fixed by design, inplace_vector expresses that constraint in the type system rather than relying on a runtime check or documentation.

Prefer std::inplace_vector over a raw array when you need the vector convenience API (iterators, size(), emplace_back) but the maximum count is known at compile time. It is strictly safer than a C array and avoids the heap-allocation bookkeeping of small_vector.

PMR Allocator as an SBO Alternative

The Polymorphic Memory Resource (PMR) library introduced in C++17 provides a flexible, policy-based path to inline storage. Rather than baking the buffer size into the container's type, you supply a stack buffer at the call site and point a std::pmr::monotonic_buffer_resource at it. Any PMR-aware container — std::pmr::vector, std::pmr::string, std::pmr::unordered_map — then draws from that buffer before touching the heap.

#include <array>
#include <memory_resource> // C++17
#include <vector>

void process_batch(const int* data, std::size_t n) {
    // Stack buffer: enough for up to 64 ints without heap.
    alignas(alignof(int)) std::array<std::byte, 64 * sizeof(int)> stack_buf; // C++17

    std::pmr::monotonic_buffer_resource rsrc{
        stack_buf.data(), stack_buf.size(),
        std::pmr::get_default_resource() // upstream fallback
    };

    // Zero heap allocations as long as n <= 64.
    std::pmr::vector<int> v{&rsrc}; // C++17
    v.reserve(n);
    for (std::size_t i = 0; i < n; ++i)
        v.push_back(data[i]);

    // ... use v normally; rsrc and stack_buf go out of scope together.
}

When n <= 64 the vector draws its backing storage entirely from stack_buf. When n > 64, monotonic_buffer_resource exhausts its inline memory and delegates to the upstream allocator (std::pmr::get_default_resource(), which calls operator new). The fallback is transparent: the container does not know or care.

The critical difference from a baked-in SBO type is that the buffer size is a runtime policy decided at the call site, not a template parameter baked into the type. This makes PMR well-suited to workloads where the typical size is known from profiling but the worst case must still be handled gracefully. The cost is that each PMR container carries a pointer to its resource, and the resource itself must outlive the container — lifetime management is the programmer's responsibility.

Prefer the PMR pattern when you need SBO-like behaviour on standard-library containers without creating a custom type. Use std::pmr::monotonic_buffer_resource for short-lived scratch allocations; use std::pmr::unsynchronized_pool_resource when objects are individually freed and reused.

folly::Function vs std::function

std::function (C++11) is copyable, which forces it to copy the stored callable on every copy of the wrapper. For large capturing lambdas this is expensive, and for move-only callables (those capturing a std::unique_ptr, for example) it is impossible. Facebook's Folly library addresses both problems with folly::Function<Sig>.

folly::Function uses a configurable inline buffer — defaulting to six pointer-sizes (48 bytes on 64-bit platforms) — and is deliberately move-only: there is no copy constructor. When the stored callable fits within the inline buffer, constructing or moving folly::Function never allocates. When it does not fit, a single heap allocation is made at construction time and subsequent moves are pointer-swaps.

// Requires Folly; illustrative — not part of the standard library.
#include <folly/Function.h>

// Move-only: stores a unique_ptr capture without heap allocation
// (assuming the lambda fits in the 48-byte inline buffer).
auto make_handler(std::unique_ptr<Widget> w) {
    return folly::Function<void(int)>{   // move-only, SBO inline
        [w = std::move(w)](int x) mutable {
            w->handle(x);
        }
    };
}

// std::function cannot store this lambda at all:
// std::function<void(int)> f = make_handler(...); // compile error

The design lesson is general: any production SBO-based callable wrapper should default to move-only semantics. Copyability requires either that the stored object be copyable (ruling out unique_ptr captures) or that copying performs a heap allocation (defeating the SBO purpose on the copy path). Most real codebases that outgrow std::function either adopt Folly, LLVM's llvm::unique_function, or write a similar wrapper in-house with an explicit inline size template parameter.

Common Pitfall: relying on std::function copy semantics to share a callback across threads. Copies of std::function are independent objects; they do not share state. If sharing is the intent, wrap the callable in a std::shared_ptr explicitly. Moving a folly::Function is the correct idiom when ownership transfer is all that is needed.

Sizing Heuristics and Benchmarking

Choosing the right inline buffer size involves a tradeoff curve with two failure modes: a buffer that is too small causes frequent heap fallbacks that eliminate the latency benefit; a buffer that is too large inflates every object and evicts useful data from cache lines. The sweet spot is workload-specific, but empirical guidance narrows the search:

• Callback registries and event handlers: 1–4 pointer-sizes (8–32 bytes on 64-bit). Most lambdas capture a this pointer and at most one or two small values.
• Type-erased value containers (any-style): 2–8 pointer-sizes (16–64 bytes). Values like small strings, pairs, and lightweight structs tend to fall in this range.
• Coroutine frame or task objects: measure concretely — frame sizes vary widely with captured state.

A Google Benchmark harness can expose the allocation rate directly by counting operator new calls:

// Requires Google Benchmark and a counting allocator shim.  // C++17
#include <benchmark/benchmark.h>
#include <atomic>
#include <new>

static std::atomic<std::size_t> g_alloc_count{0};

void* operator new(std::size_t n) {
    ++g_alloc_count;
    return std::malloc(n); // real programs should use the original operator new
}

template <std::size_t N>
static void BM_AnySmall(benchmark::State& state) {
    for (auto _ : state) {
        g_alloc_count.store(0, std::memory_order_relaxed);
        AnySmall<N> a{MyPayload{}}; // MyPayload is the type under test
        benchmark::DoNotOptimize(a);
        state.counters["allocs"] = static_cast<double>(
            g_alloc_count.load(std::memory_order_relaxed));
    }
}

BENCHMARK_TEMPLATE(BM_AnySmall,  8);
BENCHMARK_TEMPLATE(BM_AnySmall, 16);
BENCHMARK_TEMPLATE(BM_AnySmall, 32);
BENCHMARK_TEMPLATE(BM_AnySmall, 64);
BENCHMARK_MAIN();

Running this benchmark for the actual payload types in your codebase shows the exact buffer size at which the allocation count drops to zero. Pick the smallest N that achieves zero allocations for the p95 payload.

Use a static_assert at the definition site to catch regressions when a payload type grows:

// Ensures MyPayload still fits the chosen buffer; fails at compile time
// if a future change to MyPayload exceeds the limit.           // C++11
static_assert(sizeof(MyPayload) <= 48,
    "MyPayload grew past the SBO inline limit; either shrink it "
    "or increase AnySmall<N>");
static_assert(alignof(MyPayload) <= alignof(std::max_align_t),
    "MyPayload has extended alignment; handle separately");

Best Practice: add sizeof and alignof assertions alongside every SBO buffer size constant. They compile away to nothing in production and prevent silent performance regressions when types change.

Extended Alignment Types

The default alignas(std::max_align_t) buffer covers all fundamental types, but SIMD intrinsic types require stricter alignment than std::max_align_t provides on many platforms. __m128 requires 16-byte alignment and __m256 requires 32-byte alignment; on platforms where std::max_align_t is 8 or 16 bytes, placement-new of a SIMD type into an insufficiently aligned buffer is undefined behavior.

std::aligned_storage<Size, Align> was the historical solution, but it was deprecated in C++23 because it is error-prone and its functionality is fully covered by an alignas array:

// C++23: std::aligned_storage is deprecated. Use this instead:
template <std::size_t Size, std::size_t Align>
struct AlignedBuffer {
    alignas(Align) std::byte data[Size]; // C++17 std::byte
};

For an SBO buffer that must accommodate SIMD types alongside normal types, compute the required alignment at compile time:

#include <algorithm>   // std::max
#include <cstddef>
#include <type_traits>

// Maximum alignment across a list of candidate types.          // C++17
template <typename... Ts>
inline constexpr std::size_t max_align_of =
    std::max({alignof(Ts)...});

// Buffer sized and aligned for the worst-case candidate.
template <std::size_t N,
          std::size_t ExtraAlign = alignof(std::max_align_t)>
struct SIMDCapableBuf {
    static constexpr std::size_t kAlign =
        std::max(alignof(std::max_align_t), ExtraAlign);
    alignas(kAlign) std::byte data[N]; // C++17
};

// Usage: inline buffer that can hold either a double or __m256.
// (32-byte alignment satisfies both.)
// SIMDCapableBuf<64, 32> buf;

When the maximum alignment of any storable type is not known until the type is actually stored, the fits_inline gate in AnySmall must reject types whose alignof(T) exceeds the buffer's compile-time alignment:

template <typename T>
static constexpr bool inline_ok() noexcept {
    return sizeof(T) <= N &&
           alignof(T) <= kAlign &&           // reject over-aligned types
           std::is_nothrow_move_constructible_v<T>; // C++11
}

Common Pitfall: omitting the alignof check in inline_ok and placing a SIMD type into a max_align_t-aligned buffer. The misalignment is not caught at compile time and produces undefined behavior at the reinterpret_cast or pointer dereference inside the placement new. Always test with -fsanitize=undefined during development to catch alignment violations early.

Copy Constructor Sizing Problem

The AnySmall example above contains a deliberate placeholder in the copy constructor:

heap_ = ::operator new(/* sizeof(T) unknown here — use vtable size field */0);

This is the classic mistake in hand-rolled SBO containers. At the point of the copy constructor, the concrete type T has been erased; only the Ops* pointer remains. ::operator new(0) allocates an implementation-defined (possibly zero-size) block and then copy_to writes sizeof(T) bytes into it — a buffer overflow.

The correct fix is to store the object's size (and alignment, for ::operator new(size, align) in C++17) in the Ops vtable:

struct Ops {
    void        (*destroy)(void*) noexcept;
    void        (*copy_to)(void* dst, const void* src);
    void        (*move_to)(void* dst, void* src) noexcept;
    std::size_t   size;   // sizeof(T) — added field        // C++11
    std::size_t   align;  // alignof(T) — for aligned new   // C++17
};

template <typename T>
static const Ops* ops_for() noexcept {
    static constexpr Ops kOps{
        [](void* p) noexcept { static_cast<T*>(p)->~T(); },
        [](void* dst, const void* src) {
            ::new (dst) T(*static_cast<const T*>(src));
        },
        [](void* dst, void* src) noexcept {
            ::new (dst) T(std::move(*static_cast<T*>(src)));
            static_cast<T*>(src)->~T();
        },
        sizeof(T),   // size field
        alignof(T),  // align field
    };
    return &kOps;
}

With size and align in the vtable, the copy constructor becomes correct and alignment-safe:

AnySmall(const AnySmall& o) : ops_(o.ops_) {
    if (!ops_) return;
    if (o.heap_) {
        // Allocate correctly-sized, correctly-aligned raw storage. // C++17
        heap_ = ::operator new(ops_->size,
                               static_cast<std::align_val_t>(ops_->align));
        ops_->copy_to(heap_, o.heap_);
    } else {
        ops_->copy_to(buf_, o.buf_);
    }
}

~AnySmall() {
    if (ops_) ops_->destroy(live());
    if (heap_)
        ::operator delete(heap_,
                          static_cast<std::align_val_t>(ops_->align)); // C++17
}

The aligned operator new / operator delete overloads were added in C++17. On C++11/14, fall back to std::aligned_alloc or posix_memalign for over-aligned types, or simply reject them at the inline_ok gate so they never reach the heap path.

Best Practice: always include size and align fields in the Ops vtable of any type-erasing SBO container. Without them, the copy path is either broken (wrong allocation size) or undefined behavior (misaligned memory). This is the single most common correctness bug in hand-rolled SBO implementations.

See Also

• std::any (C++17) — standard type-erased container encouraged to use SBO internally
• std::function (C++11) — type-erases callables with an internal fixed buffer
• std::string — Short String Optimization is the canonical SBO specialization for characters
• std::inplace_vector (C++26) — fixed-capacity vector with guaranteed no heap allocation
• boost::container::small_vector / llvm::SmallVector — SBO applied to dynamic arrays: inline storage for up to N elements before spilling to the heap
• std::pmr::monotonic_buffer_resource (C++17) — stack-buffer-backed PMR resource for allocation-free standard containers
• folly::Function — move-only, SBO-backed callable with configurable inline size