In this blog post we’re going to walk you through how to monitor Proxmox VE via Managed VictoriaMetrics and its vmagent, including a step by step guide on how to set up, configure and visualise this environment.
Proxmox VE is a complete, open-source server management platform for enterprise virtualization. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage and networking functionality, on a single platform. With the integrated web-based user interface users can manage VMs and containers, high availability for clusters, or the integrated disaster recovery tools with ease.
Managed VictoriaMetrics allows users running VictoriaMetrics at AWS without the need to perform typical DevOps tasks such as proper configuration, monitoring, logs collection, access protection, software updates, backups, etc.
The guide covers:
Preconditions:
If you don’t have the Managed VictoriaMetrics account, please follow the guide.
To read more about Managed VictoriaMetrics see the announcement blogpost
Open https://dbaas.victoriametrics.com/instances and create a new VictoriaMetrics instance.
Configure the instance with parameters that best suit your case:
Once the instance is created and provisioned, you will get an email notification. Click on the created instance to see configuration details:
Go to the “Access” tab, press on the Example button and choose the “Writing” section to see the authorization token for access to this instance:
Copy the parameters from vmagent’s config field - you’ll need them later for vmagent configuration.
The following script will install vmagent on Proxmox VE server. Please note, that remoteWrite.bearerToken
and remoteWrite.url
fields need to be updated with corresponding values from the previous step:
Script should be running by root user or by user with sudo rights
#!/bin/bash
set -e
apt update && apt install -y curl wget net-tools jq
mkdir -p /etc/victoriametrics/vmagent
mkdir -p /var/lib/vmagent-remotewrite-data
if ! getent passwd victoriametrics >/dev/null 2>&1; then
adduser --system --home /var/lib/vmagent-remotewrite-data --group victoriametrics
fi
chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data
VM_VERSION=`curl -sg "https://api.github.com/repos/VictoriaMetrics/VictoriaMetrics/tags" | jq -r '.[0].name'`
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/vmutils-amd64-${VM_VERSION}.tar.gz -O /tmp/vmutils.tar.gz
cd /tmp && tar -xzvf /tmp/vmutils.tar.gz vmagent-prod
mv /tmp/vmagent-prod /usr/bin
chmod +x /usr/bin/vmagent-prod
chown root:root /usr/bin/vmagent-prod
cat> /etc/systemd/system/vmagent.service <<EOF
[Unit]
Description=vmagent is a tiny agent which helps you collect metrics from various sources and store them in VictoriaMetrics.
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
EnvironmentFile=-/etc/victoriametrics/vmagent/vmagent.conf
ExecStart=/usr/bin/vmagent-prod \$ARGS
ExecStop=/bin/kill -s SIGTERM \$MAINPID
ExecReload=/bin/kill -HUP \$MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
WorkingDirectory=/var/lib/vmagent-remotewrite-data
ReadWritePaths=/var/lib/vmagent-remotewrite-data
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vmagent
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=yes
EOF
cat> /etc/victoriametrics/vmagent/vmagent.conf <<EOF
ARGS="-promscrape.config=/etc/victoriametrics/vmagent/scrape.yml --remoteWrite.url=https://cloud-15.victoriametrics.com/api/v1/write --remoteWrite.bearerToken=b1b569bb-4e91-453d-8fc6-77c9a6ffaf10 -remoteWrite.tmpDataPath=/var/lib/vmagent-remotewrite-data -promscrape.suppressScrapeErrors -influxListenAddr=127.0.0.1:8089 -influx.databaseNames=proxmox"
EOF
cat> /etc/victoriametrics/vmagent/scrape.yml <<EOF
global:
scrape_interval: 10s
scrape_timeout: 30s
scrape_configs:
- job_name: 'vmagent'
static_configs:
- targets: ['127.0.0.1:8429']
EOF
chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data
chown -R victoriametrics:victoriametrics /etc/victoriametrics/vmagent
systemctl enable vmagent.service
systemctl restart vmagent.service
ps aux | grep vmagent
Login as a root@PAM
or user@pve
with Administrator permissions to Proxmox VE:
On Datacenter
menu choose Metric Server
and add InfluxDB remote write store:
Set parameters as to match this screenshot:
Configure Firewall
on Datacenter
level to allow metrics from 127.0.0.1
on 8089/UDP
port to be accepted:
Run the following command to check that vmagent actually receiving the write requests from Proxmox:
for i in {1..10}; do curl -sg http://127.0.0.1:8429/metrics | grep 'vm_ingestserver_requests_total{type="influx", name="write", net="udp"}'; sleep 2; done
The expected output is:
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 1004
From your instance’s Dashboard
tab at Managed VictoriaMetrics we should be able to see the changes in the graphs:
An increase in Active time series
and Datapoints ingestion rate
would mean that instance is receiving write requests from vmagent.
To see collected metrics in Grafana we need to add a new datasource pointed to the instance at Managed VictoriaMetrics. Go to the Access
tab at Managed VictoriaMetrics instance, press on the Example
button and choose the Reading
tab. Please see the screenshot below:
You can see the details from screenshot above to create a new datasource in Grafana. See the screenshot below:
Now you need to import Proxmox dashboard to Grafana. Open http://your_grafana_domain/dashboards
and press the Import
button:
Enter dashboard’s ID 13307 and press the Load
button on the right:
Choose vm-dbaas
as a datasource and press the Import
button.
The expected result will resemble the following screenshot:
If this is what you see then everything is working and you can observe the data on the dashboard!
Please comment below if you have any questions, or feel free to contact us.