Version: 0.22

Set the buffer mode for stdout.

Run COMMAND as a child process. When the child terminates bpftrace will also terminate, as if 'exit()' had been called. If bpftrace terminates before the child process does the child process will be terminated with a SIGTERM. If used, 'USDT' probes will only be attached to the child process. To avoid a race condition when using 'USDTs', the child is stopped after 'execve' using 'ptrace(2)' and continued when all 'USDT' probes are attached. The child process runs with the same privileges as bpftrace itself (usually root).

Unless otherwise specified, bpftrace does not perform any implicit filtering. Therefore, if you are only interested in events in COMMAND, you may want to filter based on the child PID. The child PID is available to programs as the 'cpid' builtin. For example, you could add the predicate /pid == cpid/ to probes with userspace context.

Enable debug mode. For more details see the Debug Output section.

Terminate execution right after attaching all the probes. Useful for testing that the script can be parsed, loaded, and attached, without actually running it.

Execute PROGRAM instead of reading the program from a file or stdin.

Set the output format.

The JSON output is compatible with NDJSON and JSON Lines, meaning each line of the streamed output is a single blob of valid JSON.

Print the help summary.

Add the directory DIR to the search path for C headers. This option can be used multiple times. For more details see the Preprocessor Options section.

Add FILENAME as an include for the pre-processor. This is equal to adding '#include FILENAME' at the top of the program. This option can be used multiple times. For more details see the Preprocessor Options section.

Print detailed information about features supported by the kernel and the bpftrace build.

Errors from bpf-helpers(7) are silently ignored by default which can lead to strange results.

This flag enables the detection of errors (except for errors from 'probe_read_*' BPF helpers). When errors occur bpftrace will log an error containing the source location and the error code:

Same as '-k' but also includes the errors from 'probe_read_*' BPF helpers.

List all probes that match the SEARCH pattern. If the pattern is omitted all probes will be listed. This pattern supports wildcards in the same way that probes do. E.g. '-l kprobe:*file*' to list all 'kprobes' with 'file' in the name. This can be used with a program, which will list all probes in that program. For more details see the Listing Probes section.

Suppress all warning messages created by bpftrace.

Write bpftrace tracing output to FILENAME instead of stdout. This doesn’t include child process (-c option) output. Errors are still written to stderr.

Attach to the process with PID. If the process terminates, bpftrace will also terminate. When using USDT probes, uprobes, and uretprobes they will be attached to only this process. For listing uprobes/uretprobes set the target to '*' and the process’s address space will be searched for the symbols.

Keep messages quiet.

Some calls, like 'system', are marked as unsafe as they can have dangerous side effects ('system("rm -rf")') and are disabled by default. This flag allows their use.

Activate usdt semaphores based on file path.

Print bpftrace version information.

Enable verbose messages. For more details see the Verbose Output section.

bpftrace supports accessing one-dimensional arrays like those found in C.

Constructing arrays from scratch, like int a[] = {1,2,3} in C, is not supported. They can only be read into a variable from a pointer.

The [] operator is used to access elements.

Both single line and multi line comments are supported.

Conditional expressions are supported in the form of if/else statements and the ternary operator.

The ternary operator consists of three operands: a condition followed by a ?, the expression to execute when the condition is true followed by a : and the expression to execute if the condition is false.

Both the ifTrue and ifFalse expressions must be of the same type, mixing types is not allowed.

The ternary operator can be used as part of an assignment.

If/else statements, like the one in C, are supported.

To improve script portability, you can set bpftrace Config Variables via the config block, which can only be placed at the top of the script before any probes (even BEGIN).

The names of the config variables can be in the format of environment variables or their lowercase equivalent without the BPFTRACE_ prefix. For example,BPFTRACE_STACK_MODE, STACK_MODE, and stack_mode are equivalent.

Note: Environment variables for the same config take precedence over those set inside a script config block.

List of All Config Variables

The following fundamental types are provided by the language. Note: Integers are by default represented as 64 bit signed but that can be changed by either casting them or, for scratch variables, explicitly specifying the type upon declaration.

Filters (also known as predicates) can be added after probe names. The probe still fires, but it will skip the action unless the filter is true.

Floating-point numbers are not supported by BPF and therefore not by bpftrace.

Identifiers must match the following regular expression: [_a-zA-Z][_a-zA-Z0-9]*

Integer and string literals are supported.

Integer literals can be defined in the following formats:

Octal literals have to be prefixed with a 0 e.g. 0123. Hexadecimal literals start with either 0x or 0X e.g. 0x10. Scientific literals are written in the <m>e<n> format which is a shorthand for m*10^n e.g. $i = 2e3;. Note that scientific literals are integer only due to the lack of floating point support e.g. 1e-3 is not valid.

To improve the readability of big literals an underscore _ can be used as field separator e.g. 1_000_123_000.

Integer suffixes as found in the C language are parsed by bpftrace to ensure compatibility with C headers/definitions but they’re not used as size specifiers.123UL, 123U and 123LL all result in the same integer type with a value of 123.

Character literals are not supported at this time, and the corresponding ASCII code must be used instead:

String literals can be defined by enclosing the character string in double quotes e.g. $str = "Hello world";.

Strings support the following escape sequences:

Pointers in bpftrace are similar to those found in C.

C like structs are supported by bpftrace. Fields are accessed with the . operator. Fields of a pointer to a struct can be accessed with the -> operator.

Custom structs can be defined in the preamble.

Constructing structs from scratch, like struct X var = {.f1 = 1} in C, is not supported. They can only be read into a variable from a pointer.

bpftrace has support for immutable N-tuples (n > 1). A tuple is a sequence type (like an array) where, unlike an array, every element can have a different type.

Tuples are a comma separated list of expressions, enclosed in brackets, (1,2)Individual fields can be accessed with the . operator. Tuples are zero indexed like arrays are.

Integer and pointer types can be converted using explicit type conversion with an expression like:

Integer casts to a higher rank are sign extended. Conversion to a lower rank is done by zeroing leading bits.

It is also possible to cast between integers and integer arrays using the same syntax:

Both the cast and the destination type must have the same size. When casting to an array, it is possible to omit the size which will be determined automatically from the size of the cast value.

Integers are internally represented as 64 bit signed. If you need another representation, you may cast to the supported Data Types.

bpftrace knows two types of variables, 'scratch' and 'map'.

'scratch' variables are kept on the BPF stack and their names always start with a $, e.g. $myvar. 'scratch' variables cannot be accessed outside of their lexical block e.g.

'scratch' variables can also declared before or during initialization with let e.g.

If no assignment is specified variables will initialize to 0.

You can also specify the type in the declaration e.g.

'map' variables use BPF 'maps'. These exist for the lifetime of bpftrace itself and can be accessed from all action blocks and user-space. Map names always start with a @, e.g. @mymap.

All valid identifiers can be used as name.

The data type of a variable is automatically determined during first assignment and cannot be changed afterwards.

These are the positional parameters to the bpftrace program, also referred to as command line arguments. If the parameter is numeric (entirely digits), it can be used as a number. If it is non-numeric, it must be used as a string in the str() call. If a parameter is used that was not provided, it will default to zero for numeric context, and "" for string context. Positional parameters may also be used in probe argument and will be treated as a string parameter.

If a positional parameter is used in str(), it is interpreted as a pointer to the actual given string literal, which allows to do pointer arithmetic on it. Only addition of a single constant, less or equal to the length of the supplied string, is allowed.

$# returns the number of positional arguments supplied.

This allows scripts to be written that use basic arguments to change their behavior. If you develop a script that requires more complex argument processing, it may be better suited for bcc instead, which supports Python’s argparse and completely custom argument processing.

Script example, bsize.d:

When run with a 65536 argument:

It has passed the argument in as $1 and used it as a filter.

With no arguments, $1 defaults to zero:

bswap reverses the order of the bytes in integer n. In case of 8 bit integers, n is returned without being modified. The return type is an unsigned integer of the same width as n.

buf reads length amount of bytes from address data. The maximum value of length is limited to the BPFTRACE_MAX_STRLEN variable. For arrays the length is optional, it is automatically inferred from the signature.

buf is address space aware and will call the correct helper based on the address space associated with data.

The buffer object returned by buf can safely be printed as a hex encoded string with the %r format specifier.

Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \x00). The %rx format specifier can be used to print everything in hex form, including ASCII characters. The similar %rh format specifier prints everything in hex form without \x and with spaces between bytes (e.g. 0a fe).

async

Dump the contents of the named file to stdout.cat supports the same format string and arguments that printf does. If the file cannot be opened or read an error is printed to stderr.

compile time

cgroupid retrieves the cgroupv2 ID of the cgroup available at path.

Convert cgroup id to cgroup path. This is done asynchronously in userspace when the cgroup_path value is printed, therefore it can resolve to a different value if the cgroup id gets reassigned. This also means that the returned value can only be used for printing.

A string literal may be passed as an optional second argument to filter cgroup hierarchies in which the cgroup id is looked up by a wildcard expression (cgroup2 is always represented by "unified", regardless of where it is mounted).

The currently mounted hierarchy at /sys/fs/cgroup is used to do the lookup. If the cgroup with the given id isn’t present here (e.g. when running in a Docker container), the cgroup path won’t be found (unlike when looking up the cgroup path of a process via /proc/…​/cgroup).

async

Terminate bpftrace, as if a SIGTERM was received. The END probe will still trigger (if specified) and maps will be printed. An optional exit code can be provided.

Or

async

join joins all the string array arr with sep as separator into one string. This string will be printed to stdout directly, it cannot be used as string value.

The concatenation of the array members is done in BPF and the printing happens in userspace.

compile time

Get the address of the kernel symbol name.

You can find all kernel symbols at /proc/kallsyms.

Marks ptr as a kernel address space pointer. See the address-spaces section for more information on address-spaces. The pointer type is left unchanged.

These are implemented using BPF stack maps.

Sampling only three frames from the stack (limit = 3):

You can also choose a different output format. Available formats are bpftrace, perf, and raw (no symbolication):

async

Retrieve the name of the function that contains address addr. The address to name mapping happens in user-space.

The ksym_t type can be printed with the %s format specifier.

Create a buffer that holds a macaddress as read from macThis buffer can be printed in the canonical string format using the %s format specifier.

Returns a timestamp in nanoseconds, as given by the requested kernel clock. Defaults to boot if no clock is explicitly requested.

ntop returns the string representation of an IPv4 or IPv6 address.ntop will infer the address type (IPv4 or IPv6) based on the addr type and size. If an integer or char[4] is given, ntop assumes IPv4, if a char[16] is given, ntop assumes IPv6. You can also pass the address type (e.g. AF_INET) explicitly as the first parameter.

compile time

Returns offset of the field offset bytes in struct. Similar to kernel offsetof operator. Note that subfields are not yet supported.

unsafe

Kernel 4.16

Helper bpf_override

When using override the probed function will not be executed and instead rc will be returned.

This feature only works on kernels compiled with CONFIG_BPF_KPROBE_OVERRIDE and only works on functions tagged ALLOW_ERROR_INJECTION.

bpftrace does not test whether error injection is allowed for the probed function, instead if will fail to load the program into the kernel:

Kernel 5.10

Helper bpf_d_path

Return full path referenced by struct path pointer in argument. If size is set, the path will be clamped by size otherwise BPFTRACE_MAX_STRLEN is used.

This function can only be used by functions that are allowed to, these functions are contained in the btf_allowlist_d_path set in the kernel.

sync

Get the address of the percpu kernel symbol name for CPU cpu. When cpu is omitted, the current CPU is used.

The second variant may return NULL if cpu is higher than the number of available CPUs. Therefore, it is necessary to perform a NULL-check on the result when accessing fields of the pointed structure, otherwise the BPF program will be rejected.

async

print prints a the value, which can be a map or a scalar value, with the default formatting for the type.

Prints:

Declared maps and histograms are automatically printed out on program termination.

Note that maps are printed by reference while scalar values are copied. This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it.

The printing of maps supports the optional top and div arguments.top limits the printing to the top N entries with the highest integer values

The div argument scales the values prior to printing them. Scaling values before storing them can result in rounding errors. Consider the following program:

With the following sequence as numbers for arg0: 134, 377, 111, 99. The total is 721 which rounds to 72 when scaled by 10 but the program would print 70 due to the rounding of individual values.

Changing the print call to print(@, 5, 2) will take the top 5 values and scale them by 2:

async

printf() formats and prints data. It behaves similar to printf() found in C and many other languages.

The format string has to be a constant, it cannot be modified at runtime. The formatting of the string happens in user space. Values are copied and passed by value.

bpftrace supports all the typical format specifiers like %llx and %hhu. The non-standard ones can be found in the table below:

printf() can also symbolize enums as strings. User defined enums as well as enums defined in the kernel are supported. For example:

yields:

Colors are supported too, using standard terminal escape sequences:

compile time

pton converts a text representation of an IPv4 or IPv6 address to byte array.pton infers the address family based on . or : in the given argument.pton comes in handy when we need to select packets with certain IP addresses.

Get the contents of the register identified by name. Valid names depend on the CPU architecture.

unsafe

Kernel 5.3

Helper bpf_send_signal

Probe types: k(ret)probe, u(ret)probe, USDT, profile

Send a signal to the process being traced. The signal can either be identified by name, e.g. SIGSTOP or by ID, e.g. 19 as found in kill -l.

compile time

Returns size of the argument in bytes. Similar to C/C++ sizeof operator. Note that the expression does not get evaluated.

Kernel 5.5

Helper bpf_skb_output

Write sk_buff skb 's data section to a PCAP file in the path, starting from offset to offset + length.

The PCAP file is encapsulated in RAW IP, so no ethernet header is included. The data section in the struct skb may contain ethernet header in some kernel contexts, you may set offset to 14 bytes to exclude ethernet header.

Each packet’s timestamp is determined by adding nsecs and boot time, the accuracy varies on different kernels, see nsecs.

This function returns 0 on success, or a negative error in case of failure.

Environment variable BPFTRACE_PERF_RB_PAGES should be increased in order to capture large packets, or else these packets will be dropped.

Usage

Helper probe_read_str, probe_read_{kernel,user}_str

str reads a NULL terminated (\0) string from data. The maximum string length is limited by the BPFTRACE_MAX_STRLEN env variable, unless length is specified and shorter than the maximum. In case the string is longer than the specified length only length - 1 bytes are copied and a NULL byte is appended at the end.

When available (starting from kernel 5.5, see the --info flag) bpftrace will automatically use the kernel or user variant of probe_read_{kernel,user}_str based on the address space of data, see [Address-spaces] for more information.

strcontains compares whether the string haystack contains the string needle. If needle is contained 1 is returned, else zero is returned.

bpftrace doesn’t read past the length of the shortest string.

Convert errno code to string. This is done asynchronously in userspace when the strerror value is printed, hence the returned value can only be used for printing.

async

Format the nanoseconds since boot timestamp timestamp_ns according to the format specified by fmt. The time conversion and formatting happens in user space, therefore the timestamp value returned can only be used for printing using the %s format specifier.

bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.

bpftrace also supports the following format string extensions:

strncmp compares up to n characters string s1 and string s2. If they’re equal 0 is returned, else a non-zero value is returned.

bpftrace doesn’t read past the length of the shortest string.

The use of the == and != operators is recommended over calling strncmp directly.

unsafeasync

system lets bpftrace run the specified command (fork and exec) until it completes and print its stdout. The command is run with the same privileges as bpftrace and it blocks execution of the processing threads which can lead to missed events and delays processing of async events.

Note how the async time and printf first print every second until the interval:s:10 probe hits, then they print every 10 seconds due to bpftrace blocking on sleep.

system supports the same format string and arguments that printf does.

async

Format the current wall time according to the format specifier fmt and print it to stdout. Unlike strftime() time() doesn’t send a timestamp from the probe, instead it is the time at which user-space processes the event.

bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.

Does not work with ASLR, see issue #75

The uaddr function returns the address of the specified symbol. This lookup happens during program compilation and cannot be used dynamically.

The default return type is uint64*. If the ELF object size matches a known integer size (1, 2, 4 or 8 bytes) the return type is modified to match the width (uint8*, uint16*, uint32* or uint64* resp.). As ELF does not contain type info the type is always assumed to be unsigned.

Marks ptr as a user address space pointer. See the address-spaces section for more information on address-spaces. The pointer type is left unchanged.

These are implemented using BPF stack maps.

Sampling only three frames from the stack (limit = 3):

You can also choose a different output format. Available formats are bpftrace, perf, and raw (no symbolication):

Note that for these examples to work, bash had to be recompiled with frame pointers.

async

Equal to ksym but resolves user space symbols.

If ASLR is enabled, user space symbolication only works when the process is running at either the time of the symbol resolution or the time of the probe attachment. The latter requires BPFTRACE_CACHE_USER_SYMBOLS to be set to PER_PID, and might not work with older versions of BCC. A similar limitation also applies to dynamically loaded symbols.

async

Removes a watchpoint

Calculate the running average of n between consecutive calls.

Internally this keeps two values in the map: value count and running total. The average is computed in user-space when printing by dividing the total by the count. However, you can get the average in kernel space in expressions likeif (@y == 5) but this is expensive as bpftrace needs to iterate over all the cpus to collect and sum BOTH count and total.

async

Clear all keys/values from map m.

Count how often this function is called.

Using @=count() is conceptually similar to @++. The difference is that the count() function uses a map type optimized for performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads can be expensive as bpftrace needs to iterate over all the cpus to collect and sum these values.

Note: This differs from "raw" writes (e.g. @++) where multiple writers to a shared location might lose updates, as bpftrace does not generate any atomic instructions for ++.

Example one:

Example two:

Delete a single key from a map. For scalar maps (e.g. no explicit keys), the key is omitted and is equivalent to calling clear. For map keys that are composed of multiple values (e.g. @mymap[3, "hello"] = 1 - remember these values are represented as a tuple) the syntax would be: delete(@mymap, (3, "hello"));

The, now deprecated, API (supported in version ⇐ 0.21.x) of passing map arguments with the key is still supported: e.g. delete(@mymap[3, "hello"]);.

Return true (1) if the key exists in this map. Otherwise return false (0). Error if called with a map that has no keys (aka scalar map). Return value can also be used for scratch variables and map keys/values.

Create a log2 histogram of n using $2^k$ buckets per power of 2, 0 ⇐ k ⇐ 5, defaults to 0.

Prints:

Return the number of elements in the map.

Create a linear histogram of n.lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size. Values in the range (-inf, min) and (max, inf) get their get their own bucket too, bringing the total amount of buckets created to M+2.

Prints:

Update the map with n if n is bigger than the current value held. Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).

Note: this is different than the typical userspace max() in that bpftrace’s max()only takes a single argument. The logical "other" argument to compare to is the value in the map the "result" is being assigned to.

For example, compare the two logically equivalent samples (C++ vs bpftrace):

In C++:

In bpftrace:

Also note that bpftrace takes care to handle the unset case. In other words, there is no default value. The first value you pass to max() will always be returned.

Update the map with n if n is smaller than the current value held. Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).

See max() above for how this differs from the typical userspace min().

stats combines the count, avg and sum calls into one.

Calculate the sum of all n passed.

Using @=sum(5) is conceptually similar to @+=5. The difference is that the sum() function uses a map type optimized for performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads can be expensive as bpftrace needs to iterate over all the cpus to collect and sum these values.

Note: This differs from "raw" writes (e.g. @+=5) where multiple writers to a shared location might lose updates, as bpftrace does not generate any implicit atomic operations.

Example one:

Example two:

async

Set all values for all keys to zero.

These are special built-in events provided by the bpftrace runtime.BEGIN is triggered before all other probes are attached.END is triggered after all other probes are detached.

Note that specifying an END probe doesn’t override the printing of 'non-empty' maps at exit. To prevent printing all used maps need be cleared in the END probe:

These are special built-in events provided by the bpftrace runtime. The trigger function is called by the bpftrace runtime when the bpftrace process receives specific events, such as a SIGUSR1 signal. When multiple signal handlers are attached to the same signal, only the first one is used.

These are the pre-defined hardware events provided by the Linux kernel, as commonly traced by the perf utility. They are implemented using performance monitoring counters (PMCs): hardware resources on the processor. There are about ten of these, and they are documented in the perf_event_open(2) man page. The event names are:

The count option specifies how many events must happen before the probe fires (sampling interval). If count is left unspecified a default value is used.

This will fire once for every 1,000,000 cache misses.

The interval probe fires at a fixed interval as specified by its time spec. Interval fires on one CPU at a time, unlike profile probes.

This prints the rate of syscalls per second.

Warning this feature is experimental and may be subject to interface changes.

These are eBPF iterator probes that allow iteration over kernel objects. Iterator probe can’t be mixed with any other probe, not even another iterator. Each iterator probe provides a set of fields that could be accessed with the ctx pointer. Users can display the set of available fields for each iterator via -lv options as described below.

It’s possible to pin an iterator by specifying the optional probe ':pin' part, that defines the pin file. It can be specified as an absolute or relative path to /sys/fs/bpf.

fentry/fexit probes attach to kernel functions similar to kprobe and kretprobe. They make use of eBPF trampolines which allow kernel code to call into BPF programs with near zero overhead. Originally, these were called kfunc and kretfunc but were later renamed to fentry and fexit to match how these are referenced in the kernel and to prevent confusion with BPF Kernel Functions. The original names are still supported for backwards compatibility.

fentry/fexit probes make use of BTF type information to derive the type of function arguments at compile time. This removes the need for manual type casting and makes the code more resilient against small signature changes in the kernel. The function arguments are available in the args struct which can be inspected by doing verbose listing (see Listing Probes). These arguments are also available in the return probe (fexit), unlike kretprobe.

The fget function takes one argument as file descriptor and you can access it via args.fd and the return value is accessible via retval:

kprobes allow for dynamic instrumentation of kernel functions. Each time the specified kernel function is executed the attached BPF programs are ran.

Function arguments are available through the argN for register args. Arguments passed on stack are available using the stack pointer, e.g. $stack_arg0 = (int64)reg("sp") + 16. Whether arguments passed on stack or in a register depends on the architecture and the number or arguments used, e.g. on x86_64 the first 6 non-floating point arguments are passed in registers and all following arguments are passed on the stack. Note that floating point arguments are typically passed in special registers which don’t count as argN arguments which can cause confusion. Consider a function with the following signature:

Due to d being a floating point, x is accessed through arg1 where one might expect arg2.

bpftrace does not detect the function signature so it is not aware of the argument count or their type. It is up to the user to perform Type conversion when needed, e.g.

Here arg0 was cast as a (struct path *), since that is the first argument to vfs_open. The struct support is the same as bcc and based on available kernel headers. This means that many, but not all, structs will be available, and you may need to manually define structs.

If the kernel has BTF (BPF Type Format) data, all kernel structs are always available without defining them. For example:

You can optionally specify a kernel module, either to include BTF data from that module, or to specify that the traced function should come from that module.

See BTF Support for more details.

kprobe s are not limited to function entry, they can be attached to any instruction in a function by specifying an offset from the start of the function.

kretprobe s trigger on the return from a kernel function. Return probes do not have access to the function (input) arguments, only to the return value (through retval). A common pattern to work around this is by storing the arguments in a map on function entry and retrieving in the return probe:

Profile probes fire on each CPU on the specified interval. These operate using perf_events (a Linux kernel facility, which is also used by the perf command).

The hook point triggered by tracepoint and rawtracepoint is the same.tracepoint and rawtracepoint are nearly identical in terms of functionality. The only difference is in the program context.rawtracepoint offers raw arguments to the tracepoint while tracepoint applies further processing to the raw arguments. The additional processing is defined inside the kernel.

Tracepoint arguments are available via the argN builtins. Each arg is a 64-bit integer. The available arguments can be found in the relative path of the kernel source code include/trace/events/. For example:

These are the pre-defined software events provided by the Linux kernel, as commonly traced via the perf utility. They are similar to tracepoints, but there is only about a dozen of these, and they are documented in the perf_event_open(2) man page. If the count is not provided, a default is used.

The event names are:

This roughly counts who is causing page faults, by sampling the process name for every one in one hundred faults.

Tracepoints are hooks into events in the kernel. Tracepoints are defined in the kernel source and compiled into the kernel binary which makes them a form of static tracing. Unlike kprobe s, new tracepoints cannot be added without modifying the kernel.

The advantage of tracepoints is that they generally provide a more stable interface than kprobe s do, they do not depend on the existence of a kernel function.

Tracepoint arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.

Alternatively members for each tracepoint can be listed from their /format file in /sys.

Apart from the filename member, we can also print flags, mode, and more. After the "common" members listed first, the members are specific to the tracepoint.

uprobe s or user-space probes are the user-space equivalent of kprobe s. The same limitations that apply kprobe and kretprobe also apply to uprobe s and uretprobe s, namely: arguments are available via the argN and sargN builtins and can only be accessed with a uprobe (sargN is more common for older versions of golang). retval is the return value for the instrumented function and can only be accessed with a uretprobe.

What does arg0 of readline() in /bin/bash contain? I don’t know, so I’ll need to look at the bash source code to find out what its arguments are.

When tracing libraries, it is sufficient to specify the library name instead of a full path. The path will be then automatically resolved using /etc/ld.so.cache:

If the traced binary has DWARF included, function arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.

When tracing C++ programs, it’s possible to turn on automatic symbol demangling by using the :cpp prefix:

It is important to note that for uretprobe s to work the kernel runs a special helper on user-space function entry which overrides the return address on the stack. This can cause issues with languages that have their own runtime like Golang:

Where probe_namespace is optional if probe_name is unique within the binary.

You can target the entire host (or an entire process’s address space by using the -p arg) by using a single wildcard in place of the binary_path/library_path:

Please note that if you use wildcards for the probe_name or probe_namespace and end up targeting multiple USDTs for the same probe you might get errors if you also utilize the USDT argument builtin (e.g. arg0) as they could be of different types.

Arguments are available via the argN builtins:

bpftrace also supports USDT semaphores. If both your environment and bpftrace support uprobe refcounts, then USDT semaphores are automatically activated for all processes upon probe attachment (and --usdt-file-activation becomes a noop). You can check if your system supports uprobe refcounts by running:

If your system does not support uprobe refcounts, you may activate semaphores by passing in -p $PID or --usdt-file-activation. --usdt-file-activation looks through /proc to find processes that have your probe’s binary mapped with executable permissions into their address space and then tries to attach your probe. Note that file activation occurs only once (during attach time). In other words, if later during your tracing session a new process with your executable is spawned, your current tracing session will not activate the new process. Also note that --usdt-file-activation matches based on file path. This means that if bpftrace runs from the root host, things may not work as expected if there are processes execved from private mount namespaces or bind mounted directories. One workaround is to run bpftrace inside the appropriate namespaces (i.e. the container).

This feature is experimental and may be subject to interface changes. Memory watchpoints are also architecture dependent.

These are memory watchpoints provided by the kernel. Whenever a memory address is written to (w), read from (r), or executed (x), the kernel can generate an event.

In the first form, an absolute address is monitored. If a pid (-p) or a command (-c) is provided, bpftrace takes the address as a userspace address and monitors the appropriate process. If not, bpftrace takes the address as a kernel space address.

In the second form, the address present in argN when function is entered is monitored. A pid or command must be provided for this form. If synchronous (watchpoint), a SIGSTOP is sent to the tracee upon function entry. The tracee will be SIGCONTed after the watchpoint is attached. This is to ensure events are not missed. If you want to avoid the SIGSTOP + SIGCONT use asyncwatchpoint.

Note that on most architectures you may not monitor for execution while monitoring read or write.

Print the call stack every time the jiffies variable is updated:

"hit" and exit when the memory pointed to by arg1 of increment is written to:

Default: PER_PROGRAM if ASLR disabled or -c option given, PER_PID otherwise.

Default: 1

C++ symbol demangling in userspace stack traces is enabled by default.

This feature can be turned off by setting the value of this environment variable to 0.

Default: 0

For user space symbols, symbolicate lazily/on-demand (1) or symbolicate everything ahead of time (0).

Default: 1000000

Log size in bytes.

Default: 512

This is the maximum number of BPF programs (functions) that bpftrace can generate. The main purpose of this limit is to prevent bpftrace from hanging since generating a lot of probes takes a lot of resources (and it should not happen often).

Default: 10000

Maximum bytes read by cat builtin.

Default: 4096

This is the maximum number of keys that can be stored in a map. Increasing the value will consume more memory and increase startup times. There are some cases where you will want to, for example: sampling stack traces, recording timestamps for each page, etc.

Default: 512

This is the maximum number of probes that bpftrace can attach to. Increasing the value will consume more memory, increase startup times, and can incur high performance overhead or even freeze/crash the system.

Default: 64

The maximum length (in bytes) for values created by str(), buf() and path().

This limit is necessary because BPF requires the size of all dynamically-read strings (and similar) to be declared up front. This is the size for all strings (and similar) in bpftrace unless specified at the call site. There is no artificial limit on what you can tune this to. But you may be wasting resources (memory and cpu) if you make this too high.

Default: 0

Maximum number of levels of nested field accesses for tracepoint args. 0 is unlimited.

Default: warn

Controls handling of probes with multiple kprobe or uprobe attach points which cannot be attached to some functions because they do not exist in the kernel or in the traced binary.

The possible options are: - error - always fail on missing probes - warn - print a warning but continue execution - ignore - silently ignore missing probes

Default: 32

The maximum size (in bytes) of individual objects that will be stored on the BPF stack. If they are larger than this limit they will be stored in pre-allocated memory.

This exists because the BPF stack is limited to 512 bytes and large objects make it more likely that we’ll run out of space. bpftrace can store objects that are larger than the on_stack_limit in pre-allocated memory to prevent this stack error. However, storing in pre-allocated memory may be less memory efficient. Lower this default number if you are still seeing a stack memory error or increase it if you’re worried about memory consumption.

Default: 64

Number of pages to allocate per CPU perf ring buffer. The value must be a power of 2. If you’re getting a lot of dropped events bpftrace may not be processing events in the ring buffer fast enough. It may be useful to bump the value higher so more events can be queued up. The tradeoff is that bpftrace will use more memory.

Default: bpftrace

Output format for ustack and kstack builtins. Available modes/formats:

This can be overwritten at the call site.

Default: ..

Trailer to add to strings that were truncated. Set to empty string to disable truncation trailers.

Default: 1

Controls whether maps are printed on exit. Set to 0 in order to change the default behavior and not automatically print maps at program exit.

Default: dwarf if bpftrace is compiled with LLDB, symbol_table otherwise

Choose how bpftrace will resolve all uprobe symbol locations.

Available options:

If the DebugInfo was rewritten by a post-linkage optimisation tool (like BOLT or AutoFDO), it might yield an incorrect address for a probe location. This config can force using the Symbol Table, for when the DebugInfo returns invalid addresses.

Default: None

The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details.

Default: 0

Outputs bpftrace’s runtime debug messages to the trace_pipe. This feature can be turned on by setting the value of this environment variable to 1.

Default: /lib/modules/$(uname -r)

Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel build.

Default: /lib/modules/$(uname -r)

bpftrace requires kernel headers for certain features, which are searched for in this directory.

Default: None

This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details.

The -d STAGE option produces debug output. It prints the output of the bpftrace execution stage given by the STAGE argument. The option can be used multiple times (with different stage names) and the special value all prints the output of all the supported stages. The option also takes multiple stages in one invocation as comma separated values.

Note: This is primarily used for bpftrace developers.

The supported options are:

Probe listing is the method to discover which probes are supported by the current system. Listing supports the same syntax as normal attachment does and alternatively can be combined with -e or filename args to see all the probes that a program would attach to.

The verbose flag (-v) can be specified to inspect arguments (args) for providers that support it:

The -I option can be used to add directories to the list of directories that bpftrace uses to look for headers. Can be defined multiple times.

The --include option can be used to include headers by default. Can be defined multiple times. Headers are included in the order they are defined, and they are included before any other include in the program being executed.

The -v option prints more information about the program as it is run:

Kernel and user pointers live in different address spaces which, depending on the CPU architecture, might overlap. Trying to read a pointer that is in the wrong address space results in a runtime error. This error is hidden by default but can be enabled with the -kk flag:

bpftrace tries to automatically set the correct address space for a pointer based on the probe type, but might fail in cases where it is unclear. The address space can be changed with the kptrs and uptr functions.

If the kernel version has BTF support, kernel types are automatically available and there is no need to include additional headers to use them. It is not recommended to mix definitions from multiple sources (ie. BTF and header files). If your program mixes definitions, bpftrace will do its best but can easily get confused due to redefinition conflicts. Prefer to exclusively use BTF as it can never get out of sync on a running system. BTF is also less susceptible to parsing failures (C is constantly evolving). Almost all current linux deployments will support BTF.

To allow users to detect this situation in scripts, the preprocessor macro BPFTRACE_HAVE_BTF is defined if BTF is detected. See tools/ for examples of its usage.

Requirements for using BTF for vmlinux:

Additional requirements for using BTF for kernel modules:

See kernel documentation for more information on BTF.

bpftrace parses header files using libclang, the C interface to Clang. Thus environment variables affecting the clang toolchain can be used. For example, if header files are included from a non-default directory, the CPATH or C_INCLUDE_PATH environment variables can be set to allow clang to locate the files. See clang documentation for more information on these environment variables and their usage.

bpftrace can be used to create some powerful one-liners and some simple tools. For complex tools, which may involve command line options, positional parameters, argument processing, and customized output, consider switching to bcc. bcc provides Python (and other) front-ends, enabling usage of all the other Python libraries (including argparse), as well as a direct control of the kernel BPF program. The down side is that bcc is much more verbose and laborious to program. Together, bpftrace and bcc are complimentary.

An expected development path would be exploration with bpftrace one-liners, then and ad hoc scripting with bpftrace, then finally, when needed, advanced tooling with bcc.

As an example of bpftrace vs bcc differences, the bpftrace xfsdist.bt tool also exists in bcc as xfsdist.py. Both measure the same functions and produce the same summary of information. However, the bcc version supports various arguments:

The bcc version is 131 lines of code. The bpftrace version is 22.

There are three invocation modes for bpftrace built-in functions.

While BPF in the kernel can do a lot there are still things that can only be done from user space, like the outputting (printing) of data. The way bpftrace handles this is by sending events from the BPF program which user-space will pick up some time in the future (usually in milliseconds). Operations that happen in the kernel are 'synchronous' ('sync') and those that are handled in user space are 'asynchronous' ('async')

The asynchronous behaviour can lead to some unexpected behavior as updates can happen before user space had time to process the event. The following situations may occur:

One example is updating a map value in a tight loop:

Maps are printed by reference not by value and as the value gets updated right after the print user-space will likely only see the final value once it processes the event:

Therefore, when you need precise event statistics, it is recommended to use synchronous functions (e.g. count() and hist()) to ensure more reliable and accurate results.

By default when a bpftrace program exits it will print all maps to stdout. If you don’t want this, you can either override the print_maps_on_exit configuration option or you can specify an END probe and clear the maps you don’t want printed.

For example, these two scripts are equivalent and will print nothing on exit:

In the above example, systemd-run will not finish until bpftrace has attached its probes, so you can be sure that all following commands will be traced. To stop tracing, run systemctl stop bpftrace.

To debug early boot issues, bpftrace can be invoked via a systemd service ordered before the service that needs to be traced. A basic unit file to run bpftrace before another service looks as follows::

Similarly to the systemd-run example, the service to be traced will not start until bpftrace started by the systemd unit has attached its probes.

For bpftrace PER_CPU types (search this document for "PER_CPU"), you may coerce (and thus force a more expensive synchronous read) the type to an integer using a cast or by doing a comparison. This is useful for when you need an integer during comparisons, printf(), or other.

For example: