We need to remove a vmstorage node from VictoriaMetrics cluster gracefully. Every vmstorage node contains its own portion of data and removing the vmstorage node from the cluster creates gaps in the graph (because replication is out of scope).
We have a VictoriaMetrics cluster with 2 vminsert, 2 vmselect and 3 vmstorage nodes. We want to gracefully remove vmstorage A
from the cluster.
vmstorage A
from the vminsert listvmstorage A
from the clusterNote: please expect higher resource usage on the existing vmstorage nodes (vmstorage B
and vmstorage C
), as they now need to handle all the incoming data.
Pros: Simple implementation
Cons: You may need to wait for a long period of time
vmstorage A
from the vminsert list (same as in Solution One).vmstorage A
and writes data back to vminsert nodes. 4. This process creates duplicates.vmstorage A
from the cluster.Note: Please expect higher resource usage on the existing nodes (vmstorage B
and vmstorage C
), as they now need to handle all the incoming data.
Pros: Faster way to decommission a vmstorage node.
Cons: The process is more complex compared to solution One. The vmctl import/export process may require tuning if you migrate hundreds GB of data (or more).
Hint : downsampling reduces the amount of data in a cluster; after downsampling, the vmctl migration requires less data to transfer and less time.
We trust that this is helpful!
Please let us know how you get on or if you have any questions by submitting a comment below.