Snooping /dev/null

Ever wondered what gets written into the big global bit bucket, /dev/null? No? On busy, active systems it is not only interesting to see what is writen to this file but it may actually be extremely useful for debugging and troubleshooting. This is simply because developers frequently redirect stderr to /dev/null either in applications or in scripts and, while this may be the correct thing to do most of the time, it can sometimes obscure interesting runtime behaviour.

So, we simply want to catch any data written to the /dev/null file. The /dev/null file is actually a character special device file and the operations for the "device" are provided by the kernel devlist array. Looking at the null_fops struct we can see that write operations are serviced by the write_null() function so let's capture the data coming into that function. The function is trivial as it is there to simply "absorb" data being written to it:

static ssize_t write_null(struct file *file, const char __user *buf,
			  size_t count, loff_t *ppos)
{
	return count;
}

The data of interest to us is contained in the buf parameter - the second parameter to write_null(). The __user part of the type declaration here says that this pointer is a userland pointer and therefore shouldn't be dereferenced directly by kernel code. This means that we'll have to treat this specially in our bpftrace script to get the results we want.

The script to capture writes to /dev/null is extremely short but a few things need explanation:

config = {
  max_strlen = 32768;
}

kprobe:vmlinux:write_null*
{
  printf("%s: %s\n", comm,  str(arg1));
}

• The default string size is currently 64 bytes so this means that printing a buffer with the %s format modifier will truncate the outout at 64 bytes. Alot of work has been done recently on increasing that limit and it's nearly all landed but until that time we need to manually tune the maximum string size parameter (max_strlen) to 32KB which is the upper bound on string sizes.
• The wildcard match on the kprobe:vmlinux:write_null probe is an unfortunate necessity for me on my system as the write_null function has been optimised by the compiler and renamed to be kprobe:vmlinux:write_null.llvm.13919501563405862786() 😥. The wildcard means it should just work for anyone else as well (although it assumes that we only match the one function...).
• The uptr() function marks the buf pointer as user address space pointer and this ensures that bpftrace will use the correct mechanism to access the pointer. Without explicit use of uptr() here the script will appear to work but will just error silently when attemoting to access the pointer. For further information see the "Address Spaces" section in the documentation (XXX link).

Here's some sample output on my test system:

$ sudo bpftrace devnull.bt
Attaching 1 probe...

tput: tput:
tput: No value for $TERM and no -T specified
tput:

ethtool: netlink error: Operation not permitted

proxy2_counters: /etc/fbagent/commandcollectors/configerator/proxy2_counters.sh: line 41: FBAGENTC: command not found

Line 41 of the proxy2_counters.sh script is:

  fbagent_output=$(FBAGENTC getRecentCollectedValues 2>/dev/null)

Oops. The $FBAGENTC variable isn't being quoted correctly and therefore we see the command not found output shown above. This breaking change has gone unnoticed for an embarrassingly long period of time because the script still appears to work: the fbagent_output variable containing stdout from the $FBAGENTC command invocation is used to control logic in the script that dictates output and therefore things appeared to be OK. Functionally the script was broken though for a certain class of systems. The fix was obviously trivial but this goes to highlight that functional breakage can easily be introduced and we need all the help we can get in locating such errors.