Imagine you write a bpftrace program which needs to access a data structure of some kernel data type, say struct task_struct. In order to generate correct offsets for accessing the struct fields, bpftrace needs to know the layout of the type on the running kernel. Historically, this could be achieved by providing the correct kernel headers to the program using the #include directive. With the coming of BTF (BPF Type Information), this is no longer necessary as bpftrace is able to automatically extract the types layout from BTF. Therefore, for a vast majority of use-cases, including headers is not only unnecessary, but can also lead to unexpected problems and should be avoided, if possible. In this blog post, we will look into the reasons why that is the case and show that the less headers a bpftrace program includes, the more portable it is across kernel versions.

It may seem strange that a language that prides itself on terseness didn’t have a way to reduce duplicate code. These days almost all popular, general-purpose programming languages provide at least one mechanism for this: functions, macros, gotos, etc. But bpftrace is a domain-specific language (DSL); known for one-liners. But it seems people have started to write long bpftrace programs (have you seen bpfsnake?) and, as a result, started to crave the ability to not repeat themselves in order to reduce errors, reading, and writing.

This post is about the journey to adding macros to bpftrace.

A mysterious CPU spike in ClickHouse Cloud on GCP led to months of debugging, revealing a deeper issue within the Linux kernel’s memory management. This is an
article written by Sergei Trifonov featuring bpftrace.

Full Article

I spent a few weeks earlier this year [tracking down][1] a set of flaky end-to-end tests where bpftrace would occasionally cease to print output. I had gotten as far as figuring out std::cout had [badbit][0] set after a write but had run out of ideas on how to debug it. At the time, because I could not reproduce it locally, I had assumed it was an oddity with pipes and CI and given up.

Except bugs never go away. They only lay dormant.

Ever wondered what gets written into the big global bit bucket, /dev/null? No? On busy, active systems it is not only interesting to see what is writen to this file but it may actually be extremely useful for debugging and troubleshooting. This is simply because developers frequently redirect stderr to /dev/null either in applications or in scripts and, while this may be the correct thing to do most of the time, it can sometimes obscure interesting runtime behaviour.