Monitoring Proxmox VE via Managed VictoriaMetrics and vmagent

In this blog post we’re going to walk you through how to monitor Proxmox VE via Managed VictoriaMetrics and its vmagent, including a step by step guide on how to set up, configure and visualise this environment.

Proxmox VE is a complete, open-source server management platform for enterprise virtualization. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage and networking functionality, on a single platform. With the integrated web-based user interface users can manage VMs and containers, high availability for clusters, or the integrated disaster recovery tools with ease.

Managed VictoriaMetrics allows users running VictoriaMetrics at AWS without the need to perform typical DevOps tasks such as proper configuration, monitoring, logs collection, access protection, software updates, backups, etc.

The guide covers:

• How to setup an instance on Managed VictoriaMetrics
• How to setup vmagent on Proxmox VE Hypervisor server
• How to configure Proxmox’ built-in monitoring to work with vmagent
• How to visualize stored data

Preconditions:

• Proxmox VE ver 7.0+
• Managed VictoriaMetrics account registered via AWS Marketplace
• Terminal with curl installed
• Installed Grafana
• Grafana dashboard for Proxmox VE

1. Setup Managed VictoriaMetrics instance

If you don’t have the Managed VictoriaMetrics account, please follow the guide.

To read more about Managed VictoriaMetrics see the announcement blogpost

Open https://dbaas.victoriametrics.com/instances and create a new VictoriaMetrics instance.

Configure the instance with parameters that best suit your case:

Once the instance is created and provisioned, you will get an email notification. Click on the created instance to see configuration details:

Go to the “Access” tab, press on the Example button and choose the “Writing” section to see the authorization token for access to this instance:

Copy the parameters from vmagent’s config field - you’ll need them later for vmagent configuration.

2. Install vmagent

The following script will install vmagent on Proxmox VE server. Please note, that remoteWrite.bearerToken and remoteWrite.url fields need to be updated with corresponding values from the previous step:

Script should be running by root user or by user with sudo rights

#!/bin/bash
set -e
apt update && apt install -y curl wget net-tools jq

mkdir -p /etc/victoriametrics/vmagent
mkdir -p /var/lib/vmagent-remotewrite-data

if ! getent passwd victoriametrics >/dev/null 2>&1; then
  adduser --system --home /var/lib/vmagent-remotewrite-data --group victoriametrics
fi

chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data

VM_VERSION=`curl -sg "https://api.github.com/repos/VictoriaMetrics/VictoriaMetrics/tags" | jq -r '.[0].name'`

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/vmutils-amd64-${VM_VERSION}.tar.gz -O /tmp/vmutils.tar.gz

cd /tmp && tar -xzvf /tmp/vmutils.tar.gz vmagent-prod
mv /tmp/vmagent-prod /usr/bin
chmod +x /usr/bin/vmagent-prod
chown root:root /usr/bin/vmagent-prod

cat> /etc/systemd/system/vmagent.service <<EOF
[Unit]
Description=vmagent is a tiny agent which helps you collect metrics from various sources and store them in VictoriaMetrics.
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
EnvironmentFile=-/etc/victoriametrics/vmagent/vmagent.conf
ExecStart=/usr/bin/vmagent-prod \$ARGS
ExecStop=/bin/kill -s SIGTERM \$MAINPID
ExecReload=/bin/kill -HUP \$MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
WorkingDirectory=/var/lib/vmagent-remotewrite-data
ReadWritePaths=/var/lib/vmagent-remotewrite-data
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vmagent
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=yes
EOF

cat> /etc/victoriametrics/vmagent/vmagent.conf <<EOF
ARGS="-promscrape.config=/etc/victoriametrics/vmagent/scrape.yml --remoteWrite.url=https://cloud-15.victoriametrics.com/api/v1/write --remoteWrite.bearerToken=b1b569bb-4e91-453d-8fc6-77c9a6ffaf10 -remoteWrite.tmpDataPath=/var/lib/vmagent-remotewrite-data -promscrape.suppressScrapeErrors -influxListenAddr=127.0.0.1:8089 -influx.databaseNames=proxmox"
EOF

cat> /etc/victoriametrics/vmagent/scrape.yml <<EOF
global:
  scrape_interval: 10s
  scrape_timeout: 30s

scrape_configs:
  - job_name: 'vmagent'
    static_configs:
      - targets: ['127.0.0.1:8429']
EOF

chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data
chown -R victoriametrics:victoriametrics /etc/victoriametrics/vmagent

systemctl enable vmagent.service
systemctl restart vmagent.service
ps aux | grep vmagent

3. Configure Proxmox VE to send metrics to vmagent

Login as a root@PAM or user@pve with Administrator permissions to Proxmox VE:

On Datacenter menu choose Metric Server and add InfluxDB remote write store:

Set parameters as to match this screenshot:

Configure Firewall on Datacenter level to allow metrics from 127.0.0.1 on 8089/UDP port to be accepted:

4. Verify metrics sending

Run the following command to check that vmagent actually receiving the write requests from Proxmox:

for i in {1..10}; do curl -sg http://127.0.0.1:8429/metrics | grep 'vm_ingestserver_requests_total{type="influx", name="write", net="udp"}'; sleep 2; done

The expected output is:

vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 1004

From your instance’s Dashboard tab at Managed VictoriaMetrics we should be able to see the changes in the graphs:

An increase in Active time series and Datapoints ingestion rate would mean that instance is receiving write requests from vmagent.

5. Check the obtained result in your browser

To see collected metrics in Grafana we need to add a new datasource pointed to the instance at Managed VictoriaMetrics. Go to the Access tab at Managed VictoriaMetrics instance, press on the Example button and choose the Reading tab. Please see the screenshot below:

You can see the details from screenshot above to create a new datasource in Grafana. See the screenshot below:

Now you need to import Proxmox dashboard to Grafana. Open http://your_grafana_domain/dashboards and press the Import button:

Enter dashboard’s ID 13307 and press the Load button on the right:

Choose vm-dbaas as a datasource and press the Import button.

The expected result will resemble the following screenshot:

If this is what you see then everything is working and you can observe the data on the dashboard!

5. Final thoughts

• We have set up an instance at Managed VictoriaMetrics
• We have set up and configured vmagent on Proxmox VE
• We have configured Proxmox VE to send metrics to the vmagent via UDP
• We have configured Grafana to work with Managed VictoriaMetrics instance
• We visualized resource usage in Proxmox VE via Grafana

Please comment below if you have any questions, or feel free to contact us.