How to Decommission a vmstorage Node from a VictoriaMetrics Cluster

How to Decommission a vmstorage Node from a VictoriaMetrics Cluster

Share: Share on LinkedIn Share on X (Twitter)

Problem

#

We need to remove a vmstorage node from VictoriaMetrics cluster gracefully. Every vmstorage node contains its own portion of data and removing the vmstorage node from the cluster creates gaps in the graph (because replication is out of scope).

Setup example

#

We have a VictoriaMetrics cluster with 2 vminsert, 2 vmselect and 3 vmstorage nodes. We want to gracefully remove vmstorage A from the cluster.

Solution One

#

  1. Remove vmstorage A from the vminsert list
  2. Wait for the retention period
  3. Remove vmstorage A from the cluster

Note: please expect higher resource usage on the existing vmstorage nodes (vmstorage B and vmstorage C), as they now need to handle all the incoming data.

Pros: Simple implementation

Cons: You may need to wait for a long period of time

Solution Two

#

  1. Remove vmstorage A from the vminsert list (same as in Solution One).
  2. Set up a dedicated vmselect node that knows only about the vmstorage node that we want to remove (vmstorage A). We need this vmselect node for migration data from vmstorage A to other vmstorage nodes in the cluster.
  3. Using vmctl native import/export reads data from vmselect for vmstorage A and writes data back to vminsert nodes. 4. This process creates duplicates.
  4. Turn on deduplication on vmselect nodes.
  5. Remove vmstorage A from the cluster.

Note: Please expect higher resource usage on the existing nodes (vmstorage B and vmstorage C), as they now need to handle all the incoming data.

Pros: Faster way to decommission a vmstorage node.

Cons: The process is more complex compared to solution One. The vmctl import/export process may require tuning if you migrate hundreds GB of data (or more).

Hint : downsampling reduces the amount of data in a cluster; after downsampling, the vmctl migration requires less data to transfer and less time.

We trust that this is helpful!

Please let us know how you get on or if you have any questions by submitting a comment below.

Leave a comment below or Contact Us if you have any questions!
comments powered by Disqus

You might also like:

Upcoming Conferences & Meetups: Where to Meet Our Team

We love connecting with our community in person, and the next few months are packed with opportunities to do just that. Our team will be attending (and in some cases, speaking at) several conferences and meetups. If you’re planning to be there, we’d love to meet you—here’s where you can find us.

VictoriaMetrics Long-Term Support (LTS): H2 2025 Update

As we’re half-way through the year, we’d like to take this opportunity to provide an update on the most recent changes in our Long-Term Support (LTS) releases.

Creating a Sustainable Open Source Business Model - Introduction

Open source defies everything you’ve ever heard or learned about business before. This blog post is an introduction to how we’re creating a sustainable business model rooted in open source.

Full-Stack Observability with VictoriaMetrics in the OTel Demo

The OpenTelemetry Astronomy Shop demo has long served as a reference environment for exploring observability in distributed systems, but until now it shipped with only a Prometheus datasource. VictoriaMetrics forked the demo and extended it with VictoriaMetrics, VictoriaLogs, and VictoriaTraces, providing insights into VictoriaMetrics’ observability stack where metrics, logs, and traces flow into a unified backend.