Monitoring Proxmox VE via Managed VictoriaMetrics and vmagent
Monitoring Proxmox VE via Managed VictoriaMetrics and vmagent
In this blog post we’re going to walk you through how to monitor Proxmox VE via Managed VictoriaMetrics and its vmagent, including a step by step guide on how to set up, configure and visualise this environment.
Proxmox VE is a complete, open-source server management platform for enterprise virtualization. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage and networking functionality, on a single platform. With the integrated web-based user interface users can manage VMs and containers, high availability for clusters, or the integrated disaster recovery tools with ease.
Managed VictoriaMetrics allows users to run VictoriaMetrics on AWS without the need to perform typical DevOps tasks such as proper configuration, monitoring, logs collection, access protection, software updates, backups, etc.
The guide covers:
- How to setup a deployment on Managed VictoriaMetrics
- How to setup vmagent on Proxmox VE Hypervisor server
- How to configure Proxmox’ built-in monitoring to work with vmagent
- How to visualize stored data
Preconditions:
- Proxmox VE ver 7.0+
- Managed VictoriaMetrics account
- Terminal with curl installed
- Installed Grafana
- Grafana dashboard for Proxmox VE
1. Setup Managed VictoriaMetrics deployment
If you don’t have the Managed VictoriaMetrics account yet, just sign up here – it’s free.
To read more about Managed VictoriaMetrics see the announcement blogpost
Open https://cloud.victoriametrics.com/deployments and create a new VictoriaMetrics deployment.
Configure the deployment with parameters that best suit your case:
Once the deployment is created and provisioned, you will get an email notification. Click on the created deployment to see configuration details:
Go to the “Access” tab, press on the Example button and choose the “Writing” section to see the authorization token for access to this deployment:
Copy the parameters from vmagent’s config field - you’ll need them later for vmagent configuration.
2. Install vmagent
The following script will install vmagent on Proxmox VE server. Please note, that remoteWrite.bearerToken
and remoteWrite.url
fields need to be updated with corresponding values from the previous step:
Script should be running by root user or by user with sudo rights
#!/bin/bash
set -e
apt update && apt install -y curl wget net-tools jq
mkdir -p /etc/victoriametrics/vmagent
mkdir -p /var/lib/vmagent-remotewrite-data
if ! getent passwd victoriametrics >/dev/null 2>&1; then
adduser --system --home /var/lib/vmagent-remotewrite-data --group victoriametrics
fi
chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data
VM_VERSION=`curl -sg "https://api.github.com/repos/VictoriaMetrics/VictoriaMetrics/tags" | jq -r '.[0].name'`
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/${VM_VERSION}/vmutils-amd64-${VM_VERSION}.tar.gz -O /tmp/vmutils.tar.gz
cd /tmp && tar -xzvf /tmp/vmutils.tar.gz vmagent-prod
mv /tmp/vmagent-prod /usr/bin
chmod +x /usr/bin/vmagent-prod
chown root:root /usr/bin/vmagent-prod
cat> /etc/systemd/system/vmagent.service <<EOF
[Unit]
Description=vmagent is a tiny agent which helps you collect metrics from various sources and store them in VictoriaMetrics.
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
User=victoriametrics
Group=victoriametrics
StartLimitBurst=5
StartLimitInterval=0
Restart=on-failure
RestartSec=1
EnvironmentFile=-/etc/victoriametrics/vmagent/vmagent.conf
ExecStart=/usr/bin/vmagent-prod \$ARGS
ExecStop=/bin/kill -s SIGTERM \$MAINPID
ExecReload=/bin/kill -HUP \$MAINPID
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
WorkingDirectory=/var/lib/vmagent-remotewrite-data
ReadWritePaths=/var/lib/vmagent-remotewrite-data
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=vmagent
PrivateTmp=yes
ProtectHome=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=yes
EOF
cat> /etc/victoriametrics/vmagent/vmagent.conf <<EOF
ARGS="-promscrape.config=/etc/victoriametrics/vmagent/scrape.yml --remoteWrite.url=https://cloud-15.victoriametrics.com/api/v1/write --remoteWrite.bearerToken=b1b569bb-4e91-453d-8fc6-77c9a6ffaf10 -remoteWrite.tmpDataPath=/var/lib/vmagent-remotewrite-data -promscrape.suppressScrapeErrors -influxListenAddr=127.0.0.1:8089 -influx.databaseNames=proxmox"
EOF
cat> /etc/victoriametrics/vmagent/scrape.yml <<EOF
global:
scrape_interval: 10s
scrape_timeout: 30s
scrape_configs:
- job_name: 'vmagent'
static_configs:
- targets: ['127.0.0.1:8429']
EOF
chown -R victoriametrics:victoriametrics /var/lib/vmagent-remotewrite-data
chown -R victoriametrics:victoriametrics /etc/victoriametrics/vmagent
systemctl enable vmagent.service
systemctl restart vmagent.service
ps aux | grep vmagent
3. Configure Proxmox VE to send metrics to vmagent
Login as a root@PAM
or user@pve
with Administrator permissions to Proxmox VE:
On Datacenter
menu choose Metric Server
and add InfluxDB remote write store:
Set parameters as to match this screenshot:
Configure Firewall
on Datacenter
level to allow metrics from 127.0.0.1
on 8089/UDP
port to be accepted:
4. Verify metrics sending
Run the following command to check that vmagent actually receiving the write requests from Proxmox:
for i in {1..10}; do curl -sg http://127.0.0.1:8429/metrics | grep 'vm_ingestserver_requests_total{type="influx", name="write", net="udp"}'; sleep 2; done
The expected output is:
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 989
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 997
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 1004
From your deployment’s Dashboard
tab at Managed VictoriaMetrics we should be able to see the changes in the graphs:
An increase in Active time series
and Datapoints ingestion rate
would mean that deployment is receiving write requests from vmagent.
5. Check the obtained result in your browser
To see collected metrics in Grafana we need to add a new datasource pointed to the deployment at Managed VictoriaMetrics. Go to the Access
tab at Managed VictoriaMetrics deployment, press on the Example
button and choose the Reading
tab. Please see the screenshot below:
You can see the details from screenshot above to create a new datasource in Grafana. See the screenshot below:
Now you need to import Proxmox dashboard to Grafana. Open http://your_grafana_domain/dashboards
and press the Import
button:
Enter dashboard’s ID 13307 and press the Load
button on the right:
Choose vm-dbaas
as a datasource and press the Import
button.
The expected result will resemble the following screenshot:
If this is what you see then everything is working and you can observe the data on the dashboard!
5. Final thoughts
- We have set up a deployment at Managed VictoriaMetrics
- We have set up and configured vmagent on Proxmox VE
- We have configured Proxmox VE to send metrics to the vmagent via UDP
- We have configured Grafana to work with Managed VictoriaMetrics deployment
- We visualized resource usage in Proxmox VE via Grafana
Please comment below if you have any questions, or feel free to contact us.