22.4 DTrace

DTrace originated from Sun Microsystems' Solaris operating system and is open-sourced under the CDDL license. FreeBSD has built-in support starting from the GENERIC kernel, implemented through kernel modules.

Warning

Due to the changes in commit 1ca7542a1b31, the content in this section only applies to 16.0-CURRENT.

Overview

DTrace can be used to diagnose system performance issues, investigate and debug unexpected behavior in the FreeBSD kernel and user-space programs, support running pre-written scripts, and use the D language to write custom tools to meet specific analysis needs.

The FreeBSD implementation provides full support for kernel DTrace and experimental support for user-space DTrace. Through the user-space DTrace pid provider, you can trace function boundaries in user-space programs and insert static probes in user-space programs for subsequent tracing. Some Ports, such as databases/postgresql18-server and lang/php85, provide DTrace options to enable static probes.

Partial source code from the Makefile of Port databases/postgresql18-server:

DTRACE_CONFIGURE_ENABLE=dtrace
DTRACE_LDFLAGS=         -lelf
DTRACE_INSTALL_TARGET=  install

Partial source code from the Makefile of Port lang/php85:

OPTIONS_DEFINE+=CGI CLI DEBUG DTRACE EMBED FPM IPV6 LINKTHR \
                                MYSQLND NOASLR PCRE PHPDBG ZTS JIT

The official guide for DTrace is maintained by the illumos project and is available at illumos DTrace Guide.

Note

FreeBSD's DTrace implementation differs from DTrace on illumos/macOS, and some probes and D scripts are not directly portable across platforms. User-space DTrace (USDT) is an experimental feature on FreeBSD.

Implementation Differences

Although DTrace in FreeBSD is similar to Solaris, there are some differences. The main difference is that DTrace in FreeBSD is implemented by a set of kernel modules. Since FreeBSD 10.0-RELEASE, the required modules are automatically loaded when dtrace is run.

FreeBSD uses the kernel option DDB_CTF to support loading CTF data from kernel modules and the kernel itself. CTF is Solaris's Compact C Type Format, used to encapsulate simplified debug information, similar to DWARF or traditional stabs. CTF data is added to binary files by the ctfconvert and ctfmerge build tools. The ctfconvert tool parses the DWARF ELF debug sections generated by the compiler, while ctfmerge merges the CTF ELF sections from object files into an executable or shared library.

The providers offered by FreeBSD differ slightly from Solaris. The dtmalloc provider is notable as it can trace malloc by type in the FreeBSD kernel. Some providers that exist in Solaris do not exist in FreeBSD. Additionally, providers that exist in both may be incompatible, with different probe argument types. Therefore, D scripts written for Solaris may need modification to run on FreeBSD, and vice versa.

Due to security mechanism differences, only the root user can use DTrace on FreeBSD. Solaris has some low-level security checks that FreeBSD has not yet implemented. As a result, the /dev/dtrace/dtrace device file is strictly restricted to root user access.

DTrace itself is released under the Common Development and Distribution License (CDDL). The FreeBSD kernel that supports DTrace is under the BSD license, but the DTrace kernel modules are subject to the CDDL when distributed or loaded in binary form (see the license at sys/cddl/contrib/opensolaris/OPENSOLARIS.LICENSE, and the specific source code at sys/cddl/dev/dtrace/dtrace_cddl.h).

Enabling DTrace Support

DTrace support is built into the GENERIC kernel. Since FreeBSD 10.0-RELEASE, the required modules are automatically loaded when dtrace is run. To ensure all DTrace modules are loaded (some scripts may depend on providers that are not auto-loaded), you can manually execute:

kldload dtraceall

You can install the DTrace toolkit sysutils/dtrace-toolkit.

Install using pkg:

# pkg install dtrace-toolkit

Or install using Ports:

# cd /usr/ports/sysutils/dtrace-toolkit/
# make install clean

This toolkit includes pre-built scripts for collecting system information, including checking open files, memory, CPU usage, and more. FreeBSD also includes several scripts in the base system, located in /usr/share/dtrace, and in the source tree at share/dtrace/.

Note

The scripts in /usr/share/dtrace are specifically ported for FreeBSD. Not all scripts in the DTrace toolkit can run directly on FreeBSD; some scripts require modifications to work.

The DTrace toolkit uses DTrace's own scripting language, the D language, whose syntax is similar to C++. This section does not provide an in-depth explanation of the D language. For an overview, please refer to the FreeBSD dtrace(1) manual page. The D language is also covered in detail in the illumos Dynamic Tracing Guide.

To statically compile DTrace into the kernel, add the following lines to your custom kernel configuration file and recompile the kernel:

options         KDTRACE_HOOKS
options         DDB_CTF
makeoptions     DEBUG=-g
makeoptions     WITH_CTF=1

AMD64 architecture users should also add the following line:

options         KDTRACE_FRAME

This option provides support for dtrace_fbt. Although DTrace can still run without this option, function boundary tracing functionality will be limited.

To add DTrace support for modules outside the kernel, add the following line to the module's Makefile:

CFLAGS+= -DKDTRACE_HOOKS

This flag enables DTrace hooks during compilation, facilitating debugging and monitoring of the module. After modification, you need to recompile the module to activate DTrace functionality.

Using DTrace

DTrace scripts consist of one or more probes, each of which is a monitorable point associated with an action. When the probe condition is met, the associated action is executed. For example, an action may occur when a file is opened, a process starts, or a line of code is executed. Actions can record information or modify context variables. Reading and writing context variables allows probes to share information, enabling collaborative analysis of relationships between different events.

To view all probes, execute the following command as root:

# dtrace -l | more
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4        fbt            kernel                camstatusentrycomp entry
    5        fbt            kernel                camstatusentrycomp return
    6        fbt            kernel            cam_compat_handle_0x18 entry
    7        fbt            kernel            cam_compat_handle_0x18 return

...omitted below...

Each probe has an ID, a PROVIDER (such as dtrace or fbt), a MODULE, and a FUNCTION NAME.

Regular users cannot run DTrace:

$ dtrace -l
dtrace: failed to initialize dtrace: DTrace requires additional privileges

Example: The hotkernel script identifies the functions that consume the most kernel time.

# cd /usr/local/share/dtrace-toolkit
# ./hotkernel
Sampling... Hit Ctrl-C to end.

Press Ctrl+C to stop execution. After the script finishes, it displays kernel functions and their timing information, sorted by time in ascending order:

dtrace: buffer size lowered to 1m
dtrace: aggregation size lowered to 1m
^C
FUNCTION                                                COUNT   PCNT
kernel`vm_page_alloc_domain_iter                            1   0.0%
kernel`cpu_idle                                             7   0.0%
kernel`em_update_stats_counters                             8   0.0%
kernel`spinlock_exit                                       33   0.0%
kernel`acpi_cpu_c1                                     128203 100.0%

This script also supports kernel modules. Run it with the -m option:

# cd /usr/local/share/dtrace-toolkit
# ./hotkernel -m
Sampling... Hit Ctrl-C to end.
dtrace: buffer size lowered to 1m
dtrace: aggregation size lowered to 1m
^C
MODULE                                                  COUNT   PCNT
kernel                                                  80153 100.0%

Example: The procsystime script captures and prints the system call times for a specified process ID (PID) or process name. For example, after starting a new /bin/sh instance, run:

# ./procsystime -n sh
Tracing... Hit Ctrl-C to end...
^C

Elapsed Times for processes sh,

         SYSCALL          TIME (ns)
         geteuid               4203
       sigreturn              14858
        __sysctl              61708
           write             640999
            read         2507609226

As shown above, the read system call takes the longest time, while geteuid takes the shortest.

Example: The hotopen script built into the base system periodically outputs a table listing which UIDs are opening files.

# cd /usr/share/dtrace
# ./hotopen
dtrace: buffer size lowered to 1m
dtrace: aggregation size lowered to 1m
Files opened per UID in the last second.
2026 May  2 19:20:52
uid 0   count 2
2026 May  2 19:20:53
uid 0   count 2
2026 May  2 19:20:54
uid 0   count 1
^C
2026 May  2 19:20:55
uid 0   count 1

This allows you to quickly determine whether a specific user is causing the system to enter a "thrashing" state through frequent open/close calls.