Open Source Tech

This article walks through how data flows from collection to storage, explaining how vmstorage processes incoming metrics, assigns unique IDs to time series, and organizes everything into different types of storage parts. The whole system is pretty clever - it uses in-memory buffers for speed, smart compression to save space, and has various watchdogs keeping an eye on things like disk space and data retention.

Through the weak package, you can create these special pointers that automatically become nil when their target memory gets collected. While they’re a bit trickier to use than regular pointers, they’re super useful for things like canonicalization maps and memory-efficient caching. The implementation is pretty clever too, using an 8-byte indirection object to make garbage collection more efficient.

VictoriaMetrics agent, or vmagent, is a lightweight tool designed to gather metrics from a number of different sources. Once it pulls in all those metrics, vmagent lets you ‘design’ them (through ‘relabeling’) or filter them down (doing things like reducing cardinality, stream aggregation, deduplication, and so on) before shipping them off to wherever you want to store them.

Go’s runtime package provides two intriguing features: Finalizers and KeepAlive, which help manage object lifecycle in unique ways. Finalizers let you attach cleanup functions to objects that run when they’re garbage collected. Meanwhile, KeepAlive serves as a tool to prevent premature object collection, especially when dealing with resources that need to stay alive longer than the compiler might expect.

The sync.Once is probably the easiest sync primitive to use, but there’s more under the hood than you might think. It’s also a good opportunity to understand how it works by juggling both atomic operations and mutexes.

Still, we haven’t really covered some other important interfaces, like Closer, Seeker, and a few others. And honestly, if you’re learning Go, you probably don’t want to leave those in the blind spot.

The io.Reader and io.Writer interfaces are probably some of the most common tools. Today, we’re kicking off the I/O series by taking a look at a lot of these readers and writers, and pointing out some common mistakes — like using io.ReadAll in ways that can backfire.

Go’s sync.Map isn’t a magic bullet for all concurrent map needs. It’s got some good tricks up its sleeve, like handling reads without locking, but it’s not always the best choice. This article dives into how sync.Map works under the hood, from its two-map system to the bottom line of expunged entries.

When you’ve got several identical values in your code, you only store one copy. Instead of having several copies of the same thing, they all just point to this one version, which is a lot more efficient. It’s a process often called ‘interning’ in programming circles.

What singleflight does is ensure that only one of those goroutines actually runs the operation, like getting the data from the database. It allows only one ‘in-flight’ (ongoing) operation for the same piece of data (known as a ‘key’) at any given moment.