The World So Far
An essential feature of Parca (and, by extension, Polar Signals Cloud) is the ability to filter and aggregate profiles by labels: key/value pairs that annotate each stack trace.
Labels can be coarse-grained (e.g., the hostname of the node) or fine-grained (e.g., the name of the running thread). For example, we can use the node, comm, and thread_comm labels to generate a profile for all processes running on the node denver, with the metrics graph grouped by process name and the icicle graph grouped by thread name:
When the process is running in Kubernetes or Docker, the set of available labels is determined by the pod or container metadata and exposed via relabeling. Additionally, some labels are generated by the system and always available if exposed via relabeling, like __meta_thread_comm.
However, the available pod metadata labels may not be fine-grained enough to surface all insights a user could want, and the built-in sub-process ones like __meta_thread_comm might not capture anything relevant: for example, in the Go ecosystem, programmers think in terms of goroutines rather than native threads, so the current OpenTelemetry trace ID is much more likely to be relevant than the name or ID of the underlying OS thread.
To improve the utility of Parca in these cases, we introduced trace ID labels for Go. But for users who are interested in dimensions other than trace ID, a more general approach was needed.
Introducing Custom Labels
As a motivating example, consider a database system that computes the result of SQL queries on behalf of users. Postgres is one of example of such a system (along with countless others).
An administrator might want to see at a glance how much CPU time is used for each database user, which can, as of the latest version of Polar Signals and parca-agent, be accomplished using custom labels: the code for the database system would need to set the value of the label "username" to the currently connected user while that user's queries are being served.
An example of a profile of a running Postgres instance with this change applied:
Here's another, even more realistic example: a profile of all Go processes, filtered by a particular OpenTelemetry span name:
We currently support Rust, Go, and any language that can link against C libraries (for example, C++). In Go, the labels are scoped to the goroutine; in the other languages, they are per-thread.
Getting Started
For any of the examples below, make sure you are running a recent version of parca-agent (at least v0.34.0) and passing it the command-line flag --collect-custom-labels.
Rust
Add the custom-labels crate as a dependency for your project:
cargo add custom-labels
Then use the custom_labels::with_labels function to set the labels while performing some work. The labels will be set for the given thread until the callback passed to with_labels returns. This example program loops forever, setting two labels randomly every ten seconds:
use std::time::{Duration, Instant};use rand::distributions::Alphanumeric;use rand::Rng;fn rand_str() -> String {String::from_utf8(rand::thread_rng().sample_iter(&Alphanumeric).take(16).collect::<Vec<_>>(),).unwrap()}fn main() {let mut last_update = Instant::now();loop {custom_labels::with_labels([("l1", rand_str()), ("l2", rand_str())], || loop {if last_update.elapsed() >= Duration::from_secs(10) {break;}});last_update = Instant::now();}}
Caveat: Rust custom labels are scoped to the system thread; they have no notion of, for example, the current Tokio task, which might move between threads arbitrarily. In the future, we plan to better support async workflows where the current thread is less meaningful.
Go
The agent is compatible with Pprof custom labels, so you don't need an external library. Labels are set using pprof.Do from the runtime/pprof package. This example program loops forever, setting two labels randomly every ten seconds:
package mainimport ("context""math/rand""runtime/pprof""time")func randomString(n int) string {letters := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")s := make([]rune, n)for i := range s {s[i] = letters[rand.Intn(len(letters))]}return string(s)}func main() {for {labels := pprof.Labels("l1", randomString(16), "l2", randomString(16))lastUpdate := time.Now()pprof.Do(context.TODO(), labels, func(context.Context) {for time.Since(lastUpdate) < 10*time.Second {}})lastUpdate = time.Now()}}
The label values will persist until the end of the callback passed to pprof.Do, and will be inherited by any goroutines transitively spawned by that function.
C-family languages (including C++)
The custom-labels repository also includes a low-level C library, which can be used without needing to depend on Rust. Clone the repository and build the library using make:
CFLAGS="-O2" make
Ensure that the resulting libcustomlabels.so library is linked by your application, and that the header file customlabels.h is available in its include path. The details of how to accomplish that will depend on the build system you use.
As in the Go and Rust examples above, this example program loops forever, setting two labels randomly every ten seconds:
#include <customlabels.h>#include <stdlib.h>#include <time.h>#define LABEL_LENGTH 16const char *generate_alphanumeric_string() {static char str[LABEL_LENGTH];char charset[] ="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";for (int i = 0; i < LABEL_LENGTH; i++) {str[i] = charset[rand() % (sizeof(charset) - 1)];}return str;}int main() {for (;;) {time_t last_update = time(NULL);// Note that the library copies in the provided// buffer, so we don't have to allocate or free memory for it.//// Also note that the strings don't need to be null-terminated;// their length is communicated to the library by the value of// custom_labels_string_t::len .custom_labels_set((custom_labels_string_t) {2, (unsigned char *)"l1"},(custom_labels_string_t) {LABEL_LENGTH, (unsigned char *)generate_alphanumeric_string()});custom_labels_set((custom_labels_string_t) {2, (unsigned char *)"l2"},(custom_labels_string_t) {LABEL_LENGTH, (unsigned char *)generate_alphanumeric_string()});// Spin for 10swhile (last_update + 10 > time(NULL));}}
As another example, the per-username Postgres profile in the screenshot above was generated by applying this patch to the Postgres source code (and modifying its configure script to link against libcustomlabels.so):
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.cindex 8bc6bea113..0e9f6e62ed 100644--- a/src/backend/tcop/postgres.c+++ b/src/backend/tcop/postgres.c@@ -81,6 +81,8 @@#include "utils/timestamp.h"#include "utils/varlena.h"+#include <customlabels.h>+/* ----------------* global variables* ----------------@@ -4230,6 +4232,12 @@ PostgresMain(const char *dbname, const char *username)Assert(dbname != NULL);Assert(username != NULL);+ if (username) {+ custom_labels_string_t key = (custom_labels_string_t){8, "username"};+ custom_labels_string_t val = (custom_labels_string_t){strlen(username), username};+ custom_labels_set(key, val);+ }+Assert(GetProcessingMode() == InitProcessing);/*
Conclusions
We hope the custom labels feature is useful, but please note that it's still experimental. Please let us know of any bugs or feature requests by opening an issue. If you'd like a similar feature in a language not mentioned here, don't forget to vote for it here. Happy profiling!