Running VictoriaMetrics on ARM-based processors

ARM processors become more popular and more cost-effective according to many benchmarks. One of them was made by Percona for MySQL.

Some of our users reported issues with VictoriaMetrics at AWS Graviton instances. The main concerns were higher CPU and disk IO usage compared to x86 instances of the same size and for the same workload.

By that time, we verified that VictoriaMetrics works fine for raspberry and IoT devices, but didn’t do any optimizations for ARM builds.

The main difference between x86 and ARM builds is the library we use for data encoding. x86-build uses gozstd library, a wrapper over Facebook’s zstd written in C.

Cross-compiled ARM64 build uses compress library by Klaus Post written in native Go.

So in many aspects, the performance difference between x86-build and cross-compiled ARM64 build heavily depends on the performance of these libraries.

That’s why, in order to improve performance of the ARM builds we added CGO build with some tweaks.

For my test I created 3 instances at AWS:

1. m5.4xlarge - intel x86 based instances with 16 CPU and 64 RAM $0.768 hour to run x86 build of VictoriaMetrics.
2. m6g.4xlarge - graviton2 based instance with 16 CPU and 64 RAM $0.61 hour to run cross-compiled ARM64 build without CGO.
3. m6g.4xlarge - graviton2 based instance with 16 CPU and 64 RAM $0.616 hour to run CGO build for ARM64.

And installed vmsingle v1.72.0 on each of the instances.

For workload generation, I’ve used our benchmark suite and set up a separate vmsingle node for metrics collection.

Initial CPU profiling proves this theory and shows performance improvements with gozstd lib:

CPU usage by VictoriaMetrics before optimizations

CPU usage by VictoriaMetrics after optimizations

Optimized ARM and x86 versions show almost the same result for disk IO usage:

Disk writes/reads during the benchmark

Query performance for x86 version outperforms optimized ARM by ~10% and unoptimized ARM by ~25%:

Query latency during the benchmark

One of major challenges was to add CGO build into our cross-compilation pipeline. We are using musl based builds and the default musl compiler isn’t aware of how to build code for ARM. Instead, special aarch64-musl-gcc compiler must be used:

CC=/path_to_folder/bin/aarch64-linux-musl-gcc \
GOOS=linux GOARCH=arm64 CGO_ENABLED=1 go build main.go

Important note, your C-lang dependencies must be built with the same compiler. In my case, I had to rebuild gozstd lib.

• VictoriaMetrics for ARM has better cost-performance compared with x86 machines. ARM instances are ~ 20% cheaper than x86 with the same performance.
• Read queries latency at x86 system is better - x86 instance has ~10% lower query duration.
• VictoriaMetrics has production-ready builds for ARM, prebuilt binaries, docker images since v1.73.0 release.