18.1 Linux Compatibility Layer Architecture

FreeBSD's Linux compatibility layer is often mistaken for a virtual machine.

This compatibility layer does not introduce significant performance overhead; in some scenarios, certain software even outperforms native Linux environments. The compatibility layer is neither an instruction set emulator nor a binary translation layer, but rather a system-level implementation of the Linux Application Binary Interface (ABI).

What is the Linuxulator

FreeBSD's compatibility mechanism for Linux applications relies on a core component called Linuxulator. The literal meaning of "Linuxulator" is "Linux Emulator," a name that is easily confused with traditional instruction set emulators. Linuxulator is not a traditional emulator, nor is it an independent FreeBSD user-space program; it is merely an informal designation in the official FreeBSD documentation for a specific kernel module, whose official identifier is linux.

Linuxulator is fundamentally different from WSL2's virtual machine approach and Wine's Windows API compatibility implementation layer.

Specifically, its core principle is as follows: the FreeBSD kernel can identify and intercept Linux process system call requests, map them to functionally equivalent FreeBSD system calls, and respond to these requests using the FreeBSD kernel's implementation.

Tip

In other words, Linuxulator uses the FreeBSD kernel's system calls to handle Linux process system calls.

Through the Linuxulator module, the FreeBSD kernel simulates Linux kernel behavior at the system call interface, but actual process scheduling and execution are still handled by the FreeBSD kernel; the code that processes system calls is also a native FreeBSD implementation.

Linux processes running through Linuxulator are treated as standard FreeBSD processes within the FreeBSD kernel, indistinguishable from native processes.

What is the Linux Compatibility Layer

There is no real Linux kernel in the FreeBSD system; the Linux kernel version number declared by FreeBSD serves only as an identifier and does not impose actual kernel functionality constraints; this version number can even be set to an arbitrary value, such as 255.255.

Different Linux software has different minimum kernel version requirements, and the system call interfaces they depend on and use also differ. For example, if the declared Linux kernel version is too low, the Arch Linux chroot environment may fail to initialize properly and return a kernel too old error message.

The following is compiled from an email sent by Terry Lambert ([e-mail]) to the FreeBSD mailing list (message ID: [e-mail]), introducing how the Linux binary compatibility layer works. The original text may have been lost; this section has been appropriately supplemented and updated.

User executes command (shell)
         |
         v
    +------------+
    | execve(2)  |
    +------------+
         |
         v
  +------------------------+
  | Check file Magic Number |
  +------------------------+
         |
         v
Invoke corresponding loader (ELF, a.out, PE...)
         |
         v
Check ELF Note segment Brand
         |
         v
Linux Brand
         |
         v
Linux ABI Loader
         |
         v
Thread PCB / syscall context switches to Linux system call table
e.g. sys/amd64/linux/linux_sysent.c
         |
         v
Lookup dependency binaries /compat/linux/... -> fallback /...
         |
         v
Execute Linux binary program

FreeBSD has an abstraction layer called the "execution class loader," which is embedded directly in the processing logic of the execve(2) system call.

Traditionally, UNIX loaders rely on checking magic numbers to determine file format. If no known binary format (such as ELF or the now-obsolete a.out) can be matched, the kernel returns an ENOEXEC error. At this point, the calling Shell takes over the file and attempts to interpret and execute it as a script of that Shell type.

Exercise

$ file 701.pdf
701.pdf: PDF document, version 1.4
$ hexdump -C -n 32 701.pdf
00000000 25 50 44 46 2d 31 2e 34 0d 0a 25 a1 b3 c5 d7 0d |%PDF-1.4..%.....|
00000010 0a 34 31 20 30 20 6f 62 6a 0d 0a 3c 3c 2f 44 65 |.41 0 obj..<</De|
00000020

Based on the above information, read the source code /contrib/file/magic/Magdir/pdf, and consider how the file command works.

sys/sys/elf_common.h example snippet, determining whether a file is an ELF file:

#define	ELFMAG0		0x7f
#define	ELFMAG1		'E'
#define	ELFMAG2		'L'
#define	ELFMAG3		'F'
#define	ELFMAG		"\177ELF"	/* ELF magic string */
#define	SELFMAG		4		/* ELF magic string size */

FreeBSD does not hardcode a single loader; instead, it maintains an array of loaders (execsw). The system tries different loaders in sequence, including the #! (Shebang) loader for processing scripts. In other words, in modern FreeBSD, the parsing path for most scripts is completed in kernel mode, rather than relying on Shell error fallback.

To support the Linux ABI, after identifying the standard ELF magic number (through the magic numbers defined in sys/sys/elf_common.h), FreeBSD's ELF loader further validates the dedicated Brand tag in the ELF Note segment. Since Linux and native FreeBSD programs share the same underlying ELF format, and the ELF header e_ident[EI_OSABI] field of Linux binaries is typically ELFOSABI_NONE (unlike SVR4/Solaris which uses dedicated values like ELFOSABI_SOLARIS), the ELF Note Brand tag becomes the decisive characteristic for distinguishing Linux from FreeBSD ABI targets.

sys/compat/linux/linux_elf.c example snippet, used to further determine whether the ELF is a Linux program.

bool
linux_trans_osrel(const Elf_Note *note, int32_t *osrel)
{
    const Elf32_Word *desc;   // Data pointer pointing to the note description segment
    uintptr_t p;               // Temporary pointer for calculating the description segment start position

    // note + 1 points to the position after the note structure (typically the start of the name field)
    p = (uintptr_t)(note + 1);
    // Skip the name field, aligned to Elf32_Addr
    p += roundup2(note->n_namesz, sizeof(Elf32_Addr));

    // desc points to the description segment that actually stores version information
    desc = (const Elf32_Word *)p;

    // Check whether the first element of the description segment is the Linux ABI identifier
    if (desc[0] != GNU_ABI_LINUX)
        return (false);

    /*
     * For Linux, we encode the operating system version number as follows:
     *  (version << 16) | (major << 8) | minor
     * Specific macro definitions are in linux_mib.h
     */
    *osrel = LINUX_KERNVER(desc[1], desc[2], desc[3]);

    return (true);  // Successfully parsed
}

sys/compat/linux/linux_mib.h example:

#define LINUX_KVERSION       5       // Kernel major version number (version), here indicating Linux kernel major version is 5
#define LINUX_KPATCHLEVEL    15      // Kernel minor version number (major), here indicating minor version is 15
#define LINUX_KSUBLEVEL      0       // Kernel revision number (minor), here indicating revision number is 0

// Encode the Linux kernel version number as a single integer
// Formula: upper 16 bits for version, middle 8 bits for major, lower 8 bits for minor
// Example: LINUX_KERNVER(5,15,0) => (5 << 16) + (15 << 8) + 0 = 0x00050F00
#define LINUX_KERNVER(a,b,c) (((a) << 16) + ((b) << 8) + (c))

After confirming the Linux Brand, the loader modifies the sysentvec pointer in the process execution context, switching the default system call table to the Linux ABI system call table. Thereafter, all system calls initiated by this process are indexed through this table. The associated signal trampoline and trap vectors are also switched accordingly. This system call table is provided by the kernel module; for amd64, its entries are generated by sys/amd64/linux/linux_sysent.c, which maps Linux system call numbers to corresponding kernel wrapper functions.

sys/compat/linux/linux_util.c code defining the default root path:

char linux_emul_path[MAXPATHLEN] = "/compat/linux";
// Define a string variable storing the path of the Linux compatibility environment, initial value is "/compat/linux"

SYSCTL_STRING(_compat_linux, OID_AUTO, emul_path, CTLFLAG_RWTUN,
    linux_emul_path, sizeof(linux_emul_path),
    "Linux runtime environment path");
// Register a sysctl node, making this path viewable or modifiable via sysctl
// Parameter explanation:
// _compat_linux      -> Parent node where sysctl resides
// OID_AUTO           -> System automatically assigns an OID
// emul_path          -> sysctl name
// CTLFLAG_RWTUN      -> Readable and writable, and can be set via tunable at system startup
// linux_emul_path    -> Bound variable
// sizeof(linux_emul_path) -> Variable length
// "Linux runtime environment path" -> Description information

The above code is essentially the default definition for sysctl compat.linux.emul_path.

At the file system level, the Linux compatibility layer implements a fallback root path redirection mechanism. When a Linux process requests a path lookup, the system first attempts to locate the file under the /compat/linux/ path; if not found, it falls back to the host system's native path. Linux programs can thus seamlessly load their dedicated shared libraries while still accessing FreeBSD system resources when necessary. Combined with sysctl compat.linux.osname (default value "Linux") and compat.linux.osrelease, and by providing customized tools such as uname(1) in /compat/linux, the environment disguise is fully realized.

sys/compat/linux/linux_util.c code for setting the fallback root directory:

int
linux_pwd_onexec(struct thread *td)
{
    struct nameidata nd;  // Data structure for describing file lookup operations
    int error;            // Stores the error code returned by function calls

    NDINIT(&nd, LOOKUP, FOLLOW, UIO_SYSSPACE, linux_emul_path);  // Initialize nd to look up linux_emul_path
    error = namei(&nd);  // Execute lookup
    if (error != 0) {
        pwd_altroot(td, NULL);  // Lookup failed, call pwd_altroot to set to NULL
        return (0);
    }
    NDFREE_PNBUF(&nd);          // Free path buffer in nd
    pwd_altroot(td, nd.ni_vp);  // Lookup succeeded, pass ni_vp to pwd_altroot
    vrele(nd.ni_vp);            // Release ni_vp reference
    return (0);
}

sys/sys/namei.h partial definition of the fallback mechanism:

struct nameidata {
    // ...part omitted...

    /*
     * Arguments to lookup.
     */
    struct vnode *ni_startdir;  /* Starting directory */
    struct vnode *ni_rootdir;   /* Logical root directory */
    struct vnode *ni_topdir;    /* Logical top directory */
    int ni_dirfd;               /* Starting directory file descriptor used by *at functions */
    int ni_lcf;                 /* Local call flags */

    // ...part omitted...
};

// ...part omitted...

#define namei_setup_rootdir(ndp, cnp, pwd) do {                      \
    if (__predict_true((cnp->cn_flags & ISRESTARTED) == 0))          \
        ndp->ni_rootdir = pwd->pwd_adir;  /* If not a restart call, use pwd_adir as root directory */ \
    else                                                               \
        ndp->ni_rootdir = pwd->pwd_rdir;  /* Otherwise use pwd_rdir as root directory */ \
} while (0)
#endif

// ...part omitted...

The FreeBSD kernel provides native-level support for the Linux ABI. The vast majority of system calls (such as VFS, VM, and IPC operations) directly share the FreeBSD kernel's implementation at the kernel level. The difference between the two lies only in argument marshaling at the system call boundary: FreeBSD programs use native glue functions, while Linux programs connect through compatibility layer wrappers.

Strictly speaking, this is a system-call-level ABI translation implementation, not an instruction-set-level "emulation." The CPU directly executes the native machine code of Linux binaries, with no intermediate instruction translation overhead. Early literature used the term "emulation" due to the limitations of the technical terminology at the time, which is not an accurate description of its technical essence.

Mounting the Linux Compatibility Layer File Systems

Based on the linux_start() snippet defined in the source code file libexec/rc/rc.d/linux:

	_emul_path="$(sysctl -n compat.linux.emul_path)"

	if [ -x ${_emul_path}/sbin/ldconfigDisabled ]; then
		_tmpdir=`mktemp -d -t linux-ldconfig`
		${_emul_path}/sbin/ldconfig -C ${_tmpdir}/ld.so.cache
		if ! cmp -s ${_tmpdir}/ld.so.cache ${_emul_path}/etc/ld.so.cache; then
			cat ${_tmpdir}/ld.so.cache > ${_emul_path}/etc/ld.so.cache
		fi
		rm -rf ${_tmpdir}
	fi

	#...irrelevant parts omitted...

	if checkyesno linux_mounts_enable; then
		linux_mount linprocfs "${_emul_path}/proc" -o nocover
		linux_mount linsysfs "${_emul_path}/sys" -o nocover
		linux_mount devfs "${_emul_path}/dev" -o nocover
		linux_mount fdescfs "${_emul_path}/dev/fd" -o nocover,linrdlnk
		linux_mount tmpfs "${_emul_path}/dev/shm" -o nocover,mode=1777
	fi

sysctl -n compat.linux.emul_path will output the current path of the Linux compatibility layer; the default value was mentioned above as "/compat/linux/". When building the compatibility layer under the /compat/linux/ path, the relevant file system paths can be automatically mounted and ldd dependencies refreshed. When building manually, the sysctl variable compat.linux.emul_path must be pointed to the custom path to avoid excessive manual configuration.

Warning

compat.linux.emul_path is a global sysctl variable that can only point to one path at a time. If multiple compatibility layers are installed, the later-installed compatibility layer will overwrite the emul_path setting of the earlier-installed one, causing the earlier-installed compatibility layer to malfunction. When multiple compatibility layers coexist, you must manually switch emul_path or use different startup configurations.

The linux_mounts_enable="YES" variable is read by the libexec/rc/rc.d/linux service script and then executes the actual file system mount operations. It is enabled by default, defined in the default rc file libexec/rc/rc.conf, and can be overridden in /etc/rc.conf.

Why Using the Linux Compatibility Layer is Not a Philosophy of Suffering

The practice of loading modules via kldload linux should not be questioned. As Xunzi said: "The gentleman is not different by nature; he is good at making use of things." The same applies to using the Linux compatibility layer. Similar technologies include using Wine or CrossOver on Linux, and even ReactOS; as well as Linux compatibility layers and Android compatibility layers on the Windows platform, all of which have been widely adopted.

References

• Xunzi. Xunzi[M]. Beijing: Zhonghua Book Company, 1954.
• FreeBSD Project. linux(4) -- Linux ABI support[EB/OL]. [2026-04-17]. https://man.freebsd.org/cgi/man.cgi?query=linux&sektion=4. Linux binary compatibility layer manual page.

Exercises

1. Read the linux_file.c file in the sys/compat/linux/ directory of the FreeBSD source code, analyze the mapping mechanism from Linux system calls to FreeBSD system calls, and select 3 key system calls to trace their processing flow.
2. Modify compat.linux.osrelease to 2 different Linux kernel version numbers, and test the compatibility behavior of Linux software under different version numbers.