> For the complete documentation index, see [llms.txt](https://book.bsdcn.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://book.bsdcn.org/ask/flat/chapter-22-development-tools/di-22.4-jie-dtrace.md). # 22.4 DTrace DTrace originated from Sun Microsystems' Solaris operating system and is open-sourced under the CDDL license. FreeBSD has built-in support starting from the GENERIC kernel, implemented through kernel modules. > **Warning** > > Due to the changes in commit [1ca7542a1b31](https://cgit.freebsd.org/src/commit/?id=1ca7542a1b31fffefaac5a2ff45ef23f908b33c2), the content in this section only applies to 16.0-CURRENT. ## Overview DTrace can be used to diagnose system performance issues, investigate and debug unexpected behavior in the FreeBSD kernel and user-space programs, support running pre-written scripts, and use the D language to write custom tools to meet specific analysis needs. The FreeBSD implementation provides full support for kernel DTrace and experimental support for user-space DTrace. Through the user-space DTrace `pid` provider, you can trace function boundaries in user-space programs and insert static probes in user-space programs for subsequent tracing. Some Ports, such as **databases/postgresql18-server** and **lang/php85**, provide DTrace options to enable static probes. Partial source code from the Makefile of Port databases/postgresql18-server: ```makefile DTRACE_CONFIGURE_ENABLE=dtrace DTRACE_LDFLAGS= -lelf DTRACE_INSTALL_TARGET= install ``` Partial source code from the Makefile of Port lang/php85: ```makefile OPTIONS_DEFINE+=CGI CLI DEBUG DTRACE EMBED FPM IPV6 LINKTHR \ MYSQLND NOASLR PCRE PHPDBG ZTS JIT ``` The official guide for DTrace is maintained by the illumos project and is available at [illumos DTrace Guide](https://illumos.org/books/dtrace/bookinfo.html). > **Note** > > FreeBSD's DTrace implementation differs from DTrace on illumos/macOS, and some probes and D scripts are not directly portable across platforms. User-space DTrace (USDT) is an experimental feature on FreeBSD. ## Implementation Differences Although DTrace in FreeBSD is similar to Solaris, there are some differences. The main difference is that DTrace in FreeBSD is implemented by a set of kernel modules. Since FreeBSD 10.0-RELEASE, the required modules are automatically loaded when dtrace is run. FreeBSD uses the kernel option `DDB_CTF` to support loading CTF data from kernel modules and the kernel itself. `CTF` is Solaris's Compact C Type Format, used to encapsulate simplified debug information, similar to `DWARF` or traditional stabs. `CTF` data is added to binary files by the ctfconvert and ctfmerge build tools. The `ctfconvert` tool parses the `DWARF` `ELF` debug sections generated by the compiler, while `ctfmerge` merges the `CTF` `ELF` sections from object files into an executable or shared library. The providers offered by FreeBSD differ slightly from Solaris. The `dtmalloc` provider is notable as it can trace malloc by type in the FreeBSD kernel. Some providers that exist in Solaris do not exist in FreeBSD. Additionally, providers that exist in both may be incompatible, with different probe argument types. Therefore, D scripts written for Solaris may need modification to run on FreeBSD, and vice versa. Due to security mechanism differences, only the `root` user can use DTrace on FreeBSD. Solaris has some low-level security checks that FreeBSD has not yet implemented. As a result, the **/dev/dtrace/dtrace** device file is strictly restricted to `root` user access. DTrace itself is released under the Common Development and Distribution License (CDDL). The FreeBSD kernel that supports DTrace is under the BSD license, but the DTrace kernel modules are subject to the CDDL when distributed or loaded in binary form (see the license at **sys/cddl/contrib/opensolaris/OPENSOLARIS.LICENSE**, and the specific source code at **sys/cddl/dev/dtrace/dtrace\_cddl.h**). ## Enabling DTrace Support DTrace support **is built into the GENERIC** kernel. Since FreeBSD 10.0-RELEASE, the required modules are automatically loaded when dtrace is run. To ensure all DTrace modules are loaded (some scripts may depend on providers that are not auto-loaded), you can manually execute: ```sh kldload dtraceall ``` You can install the DTrace toolkit sysutils/dtrace-toolkit. Install using pkg: ```sh # pkg install dtrace-toolkit ``` Or install using Ports: ```sh # cd /usr/ports/sysutils/dtrace-toolkit/ # make install clean ``` This toolkit includes pre-built scripts for collecting system information, including checking open files, memory, CPU usage, and more. FreeBSD also includes several scripts in the base system, located in **/usr/share/dtrace**, and in the source tree at **share/dtrace/**. > **Note** > > The scripts in **/usr/share/dtrace** are specifically ported for FreeBSD. Not all scripts in the DTrace toolkit can run directly on FreeBSD; some scripts require modifications to work. The DTrace toolkit uses DTrace's own scripting language, the D language, whose syntax is similar to C++. This section does not provide an in-depth explanation of the D language. For an overview, please refer to the FreeBSD dtrace(1) manual page. The D language is also covered in detail in the [illumos Dynamic Tracing Guide](https://www.illumos.org/books/dtrace/bookinfo.html). To statically compile DTrace into the kernel, add the following lines to your custom kernel configuration file and recompile the kernel: ```ini options KDTRACE_HOOKS options DDB_CTF makeoptions DEBUG=-g makeoptions WITH_CTF=1 ``` AMD64 architecture users should also add the following line: ```ini options KDTRACE_FRAME ``` This option provides support for `dtrace_fbt`. Although DTrace can still run without this option, function boundary tracing functionality will be limited. To add DTrace support for modules outside the kernel, add the following line to the module's Makefile: ```cpp CFLAGS+= -DKDTRACE_HOOKS ``` This flag enables DTrace hooks during compilation, facilitating debugging and monitoring of the module. After modification, you need to recompile the module to activate DTrace functionality. ## Using DTrace DTrace scripts consist of one or more **probes**, each of which is a monitorable point associated with an action. When the probe condition is met, the associated action is executed. For example, an action may occur when a file is opened, a process starts, or a line of code is executed. Actions can record information or modify context variables. Reading and writing context variables allows probes to share information, enabling collaborative analysis of relationships between different events. To view all probes, execute the following command as root: ```sh # dtrace -l | more ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR 4 fbt kernel camstatusentrycomp entry 5 fbt kernel camstatusentrycomp return 6 fbt kernel cam_compat_handle_0x18 entry 7 fbt kernel cam_compat_handle_0x18 return ...omitted below... ``` Each probe has an `ID`, a `PROVIDER` (such as `dtrace` or `fbt`), a `MODULE`, and a `FUNCTION NAME`. Regular users cannot run DTrace: ```sh $ dtrace -l dtrace: failed to initialize dtrace: DTrace requires additional privileges ``` Example: The **hotkernel** script identifies the functions that consume the most kernel time. ```sh # cd /usr/local/share/dtrace-toolkit # ./hotkernel Sampling... Hit Ctrl-C to end. ``` Press Ctrl+C to stop execution. After the script finishes, it displays kernel functions and their timing information, sorted by time in ascending order: ```sh dtrace: buffer size lowered to 1m dtrace: aggregation size lowered to 1m ^C FUNCTION COUNT PCNT kernel`vm_page_alloc_domain_iter 1 0.0% kernel`cpu_idle 7 0.0% kernel`em_update_stats_counters 8 0.0% kernel`spinlock_exit 33 0.0% kernel`acpi_cpu_c1 128203 100.0% ``` This script also supports kernel modules. Run it with the `-m` option: ```sh # cd /usr/local/share/dtrace-toolkit # ./hotkernel -m Sampling... Hit Ctrl-C to end. dtrace: buffer size lowered to 1m dtrace: aggregation size lowered to 1m ^C MODULE COUNT PCNT kernel 80153 100.0% ``` Example: The **procsystime** script captures and prints the system call times for a specified process `ID` (`PID`) or process name. For example, after starting a new **/bin/sh** instance, run: ```sh # ./procsystime -n sh Tracing... Hit Ctrl-C to end... ^C Elapsed Times for processes sh, SYSCALL TIME (ns) geteuid 4203 sigreturn 14858 __sysctl 61708 write 640999 read 2507609226 ``` As shown above, the read system call takes the longest time, while geteuid takes the shortest. Example: The **hotopen** script built into the base system periodically outputs a table listing which UIDs are opening files. ```sh # cd /usr/share/dtrace # ./hotopen dtrace: buffer size lowered to 1m dtrace: aggregation size lowered to 1m Files opened per UID in the last second. 2026 May 2 19:20:52 uid 0 count 2 2026 May 2 19:20:53 uid 0 count 2 2026 May 2 19:20:54 uid 0 count 1 ^C 2026 May 2 19:20:55 uid 0 count 1 ``` This allows you to quickly determine whether a specific user is causing the system to enter a "thrashing" state through frequent open/close calls. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://book.bsdcn.org/ask/flat/chapter-22-development-tools/di-22.4-jie-dtrace.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.