- Blog /
- vmagent High-Availability Examples
vmagent supports both the pushing and pulling (scrape) of metrics and here are examples of high availability setups for both cases.
vmagent push model high availability
Multiple sources can push data simultaneously to two groups of identically configured vmagents via the following supported push protocols:
The groups must be located in distinct availability zones for high availability puproses. E.g. when a group in a single AZ becomes unavailable, the collected data continues flowing to the group in another AZ.
The load-balancer in front of vmagent groups serves the following purposes:
vmagent nodes push the incoming data to VictoriaMetrics (either single-node or cluster version may be used for storing the data).
The deduplication must be set up at VictoriaMetrics in order to remove duplicate samples received via two vmagent groups. See how to deduplicate data in single-node VictoriaMetrics and how to deduplicate data in cluster version of VictoriaMetrics.
vmagent pull model high availability for small-to-medium loads
Two identically configured vmagent instances scrape the same targets and send the collected metrics to VictoriaMetrics (either single-node or cluster version may be used for storing the data). If a single vmagent instance is temporarily unavailable, then the remaining vmagent continues scraping the metrics. This prevents from gaps in the collected data at VictoriaMetrics.
The deduplication must be set up at VictoriaMetrics in order to remove duplicate samples received via two vmagent instances. See how to deduplicate data in single-node VictoriaMetrics and how to deduplicate data in cluster version of VictoriaMetrics.
vmagent pull model high availability for high loads
If a single vmagent has no enough resources for scraping the needed targets, then it is possible to spread scrape targets among a group of vmagent instances. This can be achieved with the following command-line flags:
-promscrape.cluster.membersCount
- the number of members in the group of vmagent instances-promscrape.cluster.memberNum
- unique index of vmagent instance in the group-promscrape.cluster.replicationFactor
- the number of vmagent instances in the cluster, which scrape every configured targetEvery vmagent instance in the group must have identical -promscrape.config
files and identical command-line flags except of -promscrape.cluster.memberNum
flag.
Every vmagent instance must have unique value for the -promscrape.cluster.memberNum
flag in the range [0 .. promscrape.cluster.membersCount-1]
.
In this setup every vmagent instance scrapes promscrape.cluster.replicationFactor / promscrape.cluster.membersCount
part of all the scrape targets
defined at -promscrape.config
(including dynamically discovered scrape targets).
If -promscrape.cluster.replicationFactor
is set to a value greater than 1, then deduplication must be set up at VictoriaMetrics in order to remove duplicate samples received from vmagent replicas. See how to deduplicate data in single-node VictoriaMetrics and how to deduplicate data in cluster version of VictoriaMetrics.
Give it a try and please do share your feedback or ask us any questions via the comments section below. We’d love to know your experience!
We love connecting with our community in person, and the next few months are packed with opportunities to do just that. Our team will be attending (and in some cases, speaking at) several conferences and meetups. If you’re planning to be there, we’d love to meet you—here’s where you can find us.
As we’re half-way through the year, we’d like to take this opportunity to provide an update on the most recent changes in our Long-Term Support (LTS) releases.
Open source defies everything you’ve ever heard or learned about business before. This blog post is an introduction to how we’re creating a sustainable business model rooted in open source.
The OpenTelemetry Astronomy Shop demo has long served as a reference environment for exploring observability in distributed systems, but until now it shipped with only a Prometheus datasource. VictoriaMetrics forked the demo and extended it with VictoriaMetrics, VictoriaLogs, and VictoriaTraces, providing insights into VictoriaMetrics’ observability stack where metrics, logs, and traces flow into a unified backend.