I have no formal training in type theory – I don’t have much informal training – I’m still lost when I crack open Benjamin Pierce’s Types and Programming Languages book. I do, however, have smart co-maintainers and a lot of experience banging my head against this small DSL called bpftrace.

As you probably know, bpftrace, the language, is an abstraction of BPF C code, which is itself a restricted version of standard C that runs in the Linux kernel. One of the “value adds” of bpftrace is that users don’t need explicit types. bpftrace is statically typed but the types are inferred. So, in the statement $x = 5, $x is resolved to an unsigned 8-bit integer. For the most part, this was a good language design choice as bpftrace’s goals are to be fast, terse, and flexible. However, this choice has had its issues; both externally in terms of causing strange user errors and internally with how the type system was built.

This post is about bpftrace’s type system and how it’s improved over the past 2 years.

This is a continuation of the posts on The Path to 1.0.

In the previous post on C Interop we discussed how it is being used to provide both an escape hatch for features not available in bpftrace’s core and as a means to reduce the amount of LLVM IR that needs to be written and supported. In this post we’ll look at the other mechanisms required to reduce bpftrace’s core code footprint and create powerful, reusable features in the bpftrace language itself.

This is a continuation of the posts on The Path to 1.0.

If you’ve ever written a BPF program from scratch you know there are two main components: the userspace code and the BPF/Kernel code. The former is used to load, attach, and process data from the latter. The BPF/Kernel code is, for the most part, a C program that is a subset of standard C with verifier constraints, meaning certain things are disallowed in order to protect the kernel from crashes or extended delays in scheduling.

This is a continuation of the posts on The Path to 1.0.

Coming in the next release is the ability to import bpftrace scripts into other bpftrace scripts. This new functionality comes on the heels of bpftrace macros, which allow you to factor out common code. Imports were the next logical step to support larger and more complex bpftrace scripts. Instead of shoving all your common code into the top of one script, you can now import them to preserve readability and sharability.

If you have used bpftrace you know how useful it can be for debugging complex issues. There are many bpftrace scripts that yield tons of valuable information about your system in just one line of code. Historically, bpftrace has focused on powerful, terse, but narrowly scoped, features to solve specific problems but as these features multiplied, so did the complexity and the inability to compose them together.

Imagine you write a bpftrace program which needs to access a data structure of some kernel data type, say struct task_struct. In order to generate correct offsets for accessing the struct fields, bpftrace needs to know the layout of the type on the running kernel. Historically, this could be achieved by providing the correct kernel headers to the program using the #include directive. With the coming of BTF (BPF Type Information), this is no longer necessary as bpftrace is able to automatically extract the types layout from BTF. Therefore, for a vast majority of use-cases, including headers is not only unnecessary, but can also lead to unexpected problems and should be avoided, if possible. In this blog post, we will look into the reasons why that is the case and show that the less headers a bpftrace program includes, the more portable it is across kernel versions.

It may seem strange that a language that prides itself on terseness didn’t have a way to reduce duplicate code. These days almost all popular, general-purpose programming languages provide at least one mechanism for this: functions, macros, gotos, etc. But bpftrace is a domain-specific language (DSL); known for one-liners. But it seems people have started to write long bpftrace programs (have you seen bpfsnake?) and, as a result, started to crave the ability to not repeat themselves in order to reduce errors, reading, and writing.

This post is about the journey to adding macros to bpftrace.

A mysterious CPU spike in ClickHouse Cloud on GCP led to months of debugging, revealing a deeper issue within the Linux kernel’s memory management. This is an
article written by Sergei Trifonov featuring bpftrace.

Full Article

I spent a few weeks earlier this year [tracking down][1] a set of flaky end-to-end tests where bpftrace would occasionally cease to print output. I had gotten as far as figuring out std::cout had [badbit][0] set after a write but had run out of ideas on how to debug it. At the time, because I could not reproduce it locally, I had assumed it was an oddity with pipes and CI and given up.

Except bugs never go away. They only lay dormant.

Ever wondered what gets written into the big global bit bucket, /dev/null? No? On busy, active systems it is not only interesting to see what is writen to this file but it may actually be extremely useful for debugging and troubleshooting. This is simply because developers frequently redirect stderr to /dev/null either in applications or in scripts and, while this may be the correct thing to do most of the time, it can sometimes obscure interesting runtime behaviour.