How to Decommission a vmstorage Node from a VictoriaMetrics Cluster

How to Decommission a vmstorage Node from a VictoriaMetrics Cluster

Share: Share on LinkedIn Share on X (Twitter)

Problem

#

We need to remove a vmstorage node from VictoriaMetrics cluster gracefully. Every vmstorage node contains its own portion of data and removing the vmstorage node from the cluster creates gaps in the graph (because replication is out of scope).

Setup example

#

We have a VictoriaMetrics cluster with 2 vminsert, 2 vmselect and 3 vmstorage nodes. We want to gracefully remove vmstorage A from the cluster.

Solution One

#

  1. Remove vmstorage A from the vminsert list
  2. Wait for the retention period
  3. Remove vmstorage A from the cluster

Note: please expect higher resource usage on the existing vmstorage nodes (vmstorage B and vmstorage C), as they now need to handle all the incoming data.

Pros: Simple implementation

Cons: You may need to wait for a long period of time

Solution Two

#

  1. Remove vmstorage A from the vminsert list (same as in Solution One).
  2. Set up a dedicated vmselect node that knows only about the vmstorage node that we want to remove (vmstorage A). We need this vmselect node for migration data from vmstorage A to other vmstorage nodes in the cluster.
  3. Using vmctl native import/export reads data from vmselect for vmstorage A and writes data back to vminsert nodes. 4. This process creates duplicates.
  4. Turn on deduplication on vmselect nodes.
  5. Remove vmstorage A from the cluster.

Note: Please expect higher resource usage on the existing nodes (vmstorage B and vmstorage C), as they now need to handle all the incoming data.

Pros: Faster way to decommission a vmstorage node.

Cons: The process is more complex compared to solution One. The vmctl import/export process may require tuning if you migrate hundreds GB of data (or more).

Hint : downsampling reduces the amount of data in a cluster; after downsampling, the vmctl migration requires less data to transfer and less time.

We trust that this is helpful!

Please let us know how you get on or if you have any questions by submitting a comment below.

Leave a comment below or Contact Us if you have any questions!
comments powered by Disqus

You might also like:

How DreamHost Slashed Memory Usage by 80% and Scaled to 76 Million Time Series

VictoriaMetrics delivers a complete open-source observability stack, combining a highly scalable time series database with a powerful log management system. This streamlined, single-binary solution simplifies deployment while providing fast, cost-effective monitoring and logging at any scale.

Upcoming Conferences & Meetups: Where to Meet Our Team

We love connecting with our community in person, and the next few months are packed with opportunities to do just that. Our team will be attending (and in some cases, speaking at) several conferences and meetups. If you’re planning to be there, we’d love to meet you—here’s where you can find us.

VictoriaMetrics Long-Term Support (LTS): H2 2025 Update

As we’re half-way through the year, we’d like to take this opportunity to provide an update on the most recent changes in our Long-Term Support (LTS) releases.

Creating a Sustainable Open Source Business Model - Introduction

Open source defies everything you’ve ever heard or learned about business before. This blog post is an introduction to how we’re creating a sustainable business model rooted in open source.