Multi-tiered Observability: A Practical Way to Handle Diverse Workloads

Observability in large companies is rarely one-size-fits-all. The VictoriaMetrics topologies guide shows why different deployment patterns are needed as scale, isolation, and reliability requirements grow. Different workloads require different trade-offs: some need long retention for audits and trend analysis, while others need higher resolution for debugging. Business-critical systems also demand dependable alerting and high availability, often with several 9s of reliability.

A single observability configuration cannot effectively support such a wide range of profiles. A better approach is to recognize these differences upfront and design an architecture that matches each workload’s needs for retention, resolution, alerting, and cost. This article presents a different take: multi-tier observability. With this, the team responsible for observability can standardize observability service profiles for the rest of the company.

The specific implementation that we show today is based on a real use case scenario kindly shared by a large trading firm using VictoriaMetrics OSS. We want to thank the engineers who approached us with their solution. We appreciate the remarkable effort they put into sharing their challenges and how they solved them, and hope to honor that effort by sharing their ideas with the larger open-source community.

Companies with a team dedicated to observability services typically start with a centralized, highly available cluster that serves all internal systems via a shared backend.

This model can work well for a long time because:

• It keeps the experience simple.
• Allows for long-term retention.
• Provides a global view that aggregates data from all systems.
• Presents a central point to configure alerts and recording rules.

As systems grow, the trade-offs of centralization become apparent. Beyond a certain point, a single shared observability backend struggles to meet all the diverse requirements. A noisy or misbehaving system can saturate capacity and introduce high cardinality issues, which in turn can cause out-of-memory issues, slow queries, or even bring down the observability backend.

Business-critical teams are most affected: their alarms fire with a lag or not at all. In general, the observability experience degrades when the backend is under heavy load, forcing operations to make compromises such as disconnecting non-critical systems, filtering data, or reducing retention.

If you are in this situation and your company expects more growth, adding more capacity isn’t likely to be the answer. This is the point where going multi-tier starts to make sense.

The key question stops being “can we store and query metrics?” and becomes “can we do it with clear tiers, predictable cost, and strong isolation between environments and workloads?”

A solution we saw applied to great effect is a three-tiered architecture with different Quality of Service (QoS) objectives, each running on its own infrastructure. Each tier provides a different service profile:

• Default tier: For standard applications where high reliability or strict SLAs are unnecessary, this tier maintains existing scraping, labeling, and alerting. It offers a familiar experience for most services without over-provisioning for non-critical needs.
• High-cardinality tier: For workloads with high-cardinality metrics or large telemetry-like experiments, development, or spammy exporters. This separation contains resource-intensive workloads and reduces impact on other tiers.
• Business critical tier: Reserved for business-critical workloads needing high availability and performance. It enforces multi-availability zone deployment, strict cardinality limits, and compliance requirements.

We can implement this architecture with vmagent and VictoriaMetrics clusters:

• vmagent: scrapes and forwards metrics to the appropriate cluster.
• VictoriaMetrics cluster: behind vmagent, storage is split across multiple VictoriaMetrics clusters, each dedicated to a specific quality‑of‑service tier. The cluster version of VictoriaMetrics has been benchmarked to be highly resource‑efficient, is horizontally scalable with separate components for ingestion, storage, and query paths, and supports multi‑node, multi‑availability‑zone topologies.

We can obtain global view by plugging Grafana into every tier via global vmselect. For alerting rules, we can plug in a separate vmalert in each tier for better isolation. The alerting notifications are then forwarded to the global Alertmanager installation.

When designing a multi-tiered setup, it’s important to define the service levels for each tier. These should align with business values and operational needs.

The following table shows a few example parameters for each tier. Adjust as necessary for your requirements.

vmagent supports limiting, relabeling, deduplication, and stream aggregation for all metric samples, scraped or pushed. This allows us to manipulate metric samples, replicate, route and shard them before storing them in the VictoriaMetrics cluster. You can read all about the process in “Life of a Sample”.

Here are a few examples of flags that let you control how data flows through vmagent:

For a VictoriaMetrics cluster, we configure vmstorage, vmselect, and vminsert to let you encode different retention, cardinality, and availability policies per tier. Here are a few notable parameter examples:

The main benefit is isolation: each tier runs in a separate system, preventing one from affecting the others.

In addition, this approach brings other benefits such as:

• Better resource allocation
• Seamless migration
• Reliable alerting

A multi-tiered approach allows us to align the value of each metric class with its actual business value.

Critical production workloads benefit from dedicated clusters configured with multi‑AZ redundancy and sufficient CPU and IO headroom. In contrast, less critical and high‑cardinality data is handled by clusters optimized for cost rather than strict SLAs. This separation dramatically reduces the risk that load tests, experimental labels, or misconfigured exporters in non‑production environments will impact production observability.

Users experience seamless migration: Grafana continues to work through VictoriaMetrics’ MetricsQL-compatible interface, so existing PromQL dashboards usually function without change. VMUI enables direct query exploration and troubleshooting.

Labels, filters, and dashboards can be organized to hide multi-tier complexity from users, allowing them to select the systems and environments relevant to them.

Alerts can scale with your observability stack, and each tier can have its own alerting policy. For example, the critical tier may have 24/7 alerting while the noisy tier can be limited to business hours.

Instead of relying solely on remote‑write paths into a distant long‑term store, alerts for critical systems can be evaluated close to where their data is ingested, with VictoriaMetrics providing consistent query performance even under heavy load.

For observability teams facing similar growth and reliability pressures, this approach offers a practical path forward.

The goal is not just to collect and query metrics; it is to do so with the right balance of stability, isolation, and efficiency for each workload class. That is the kind of observability problem VictoriaMetrics is designed to solve.

Thanks to the open-source community for sharing real-world observability challenges with us. Our users’ willingness to document challenges and solutions helps the entire community build better systems.