How to run the benchmark suite

Tags:

Eigen’s benchmarks live under benchmarks/ in the source tree as a standalone CMake project built on Google Benchmark. The BTL harness and EIGEN_BUILD_BTL option that this page used to describe were removed in February 2026.

Eigen’s own benchmarks

One executable per family, bench_<topic>, in a module directory (benchmarks/Core/bench_gemm, benchmarks/QR/bench_qr, …). They measure Eigen against itself, which is what a performance change in a merge request is judged by.

cmake -G Ninja -S benchmarks -B build-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-bench --target bench_gemm
./build-bench/Core/bench_gemm --benchmark_repetitions=10 --benchmark_report_aggregates_only=true

-DCMAKE_CXX_FLAGS="-mavx2 -mfma" (or -mavx512dq -mfma, -march=native) selects the instruction set; the default is the compiler’s baseline. Google Benchmark’s --benchmark_filter=REGEX narrows a run and --benchmark_out=FILE --benchmark_out_format=json keeps the raw numbers.

Comparing against a BLAS/LAPACK library

The cross-library comparison behind the benchmark pages is benchmarks/comparison/ (in review as merge request !2903). It builds one binary per reference library, each carrying an Eigen arm and a library arm of the same operation on the same operands, validates the library’s result against Eigen’s, and records where and how the run was made:

cd benchmarks/comparison
python3 run.py --machine <profile> --arms openblas --ops GEMM,GEMV,POTRF --scalars f32,f64
python3 reduce.py --glob 'results/<profile>/*/*.json' --baseline openblas --out merged.json

A machine profile is a small TOML file in benchmarks/comparison/machines/ naming the host, its instruction-set targets and how each library is found; copy the closest one and adjust it before running on a new machine. run.py refuses to measure from a modified worktree or on a loaded machine unless told otherwise, and every result file states which caveats apply.

Measuring well

  • Stop everything else on the machine; a load average above 1 already moves single-threaded numbers.
  • Disable turbo boost or, on Linux, set the performance governor where the platform allows it, and say so when it does not (virtual machines and WSL2 usually do not).
  • Take several repetitions and compare medians; never the best run.
  • Pin every library to one thread (OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, BLIS_NUM_THREADS, OMP_NUM_THREADS) unless thread scaling is what is being measured; an unset variable silently races a multithreaded library against single-threaded Eigen.
  • Record the CPU model, compiler and flags, Eigen commit and library versions next to every number. A rate without them cannot be compared to anything later.