Skip to content
Standard libraryZig 0.17.0Official documentation
Sourcestd/os/linux/tls.zig

Source line 1

This file implements the two TLS variants [1] used by ELF-based systems. Note that, in reality, Variant I has two sub-variants.

It is important to understand that the term TCB (Thread Control Block) is overloaded here. Official ABI documentation uses it simply to mean the ABI TCB, i.e. a small area of ABI-defined data, usually one or two words (see the AbiTcb type below). People will also often use TCB to refer to the libc TCB, which can be any size and contain anything. (One could even omit it!) We refer to the latter as the Zig TCB; see the ZigTcb type below.

[1] https://www.akkadia.org/drepper/tls.pdf

std.os.linux.tls.__tls_get_addr

function

Source line 734

fn __tls_get_addr(ti: *const TlsIndex) callconv(.c) *anyopaque

Parameters

ti: *const TlsIndex

std.os.linux.tls.__tls_get_offset

function

Source line 748

fn __tls_get_offset() callconv(.naked) noreturn

std.os.linux.tls.AbiTcb

type

Source line 144

Per-thread storage for the ELF TLS ABI.

std.os.linux.tls.alignBackward

function

Source line 605

Inline because TLS is not set up yet.

inline fn alignBackward(addr: usize, alignment: usize) usize

Parameters

addr: usize
alignment: usize

std.os.linux.tls.alignForward

function

Source line 600

Inline because TLS is not set up yet.

inline fn alignForward(addr: usize, alignment: usize) usize

Parameters

addr: usize
alignment: usize

std.os.linux.tls.alignPtrCast

function

Source line 610

Inline because TLS is not set up yet.

inline fn alignPtrCast(comptime T: type, ptr: [*]u8) *T

Parameters

T: type
ptr: [*]u8

std.os.linux.tls.area_desc

variable

Source line 218

AreaDesc

std.os.linux.tls.AreaDesc

container

Source line 190

Describes a process's TLS area. The area encompasses the DTV, both TCBs, and the TLS block, with the exact layout of these being dependent primarily on current_variant.

Fields

size: usize
alignment: usize
dtv: struct {
    /// Offset into the TLS area.
    offset: usize,
}
abi_tcb: struct {
    /// Offset into the TLS area.
    offset: usize,
}
block: struct {
    /// The initial data to be copied into the TLS block. Note that this may be smaller than
    /// `size`, in which case any remaining data in the TLS block is simply left uninitialized.
    init: []const u8,
    /// Offset into the TLS area.
    offset: usize,
    /// This is the effective size of the TLS block, which may be greater than `init.len`.
    size: usize,
}
gdt_entry_number: usize

Only used on the 32-bit x86 architecture (not x86_64, nor x32).

std.os.linux.tls.assert

alias

Source line 16

Alias of std.debug.assert

std.os.linux.tls.builtin

constant

Source line 17

std.os.linux.tls.computeAreaDesc

function

Source line 483

fn computeAreaDesc(phdrs: []elf.ElfN.Phdr) void

Parameters

phdrs: []elf.ElfN.Phdr

std.os.linux.tls.current_dtv_offset

constant

Source line 126

Usually only used by the modified Variant I.

std.os.linux.tls.current_tp_offset

constant

Source line 111

The offset value for the modified Variant I.

std.os.linux.tls.current_variant

constant

Source line 66

Variant

std.os.linux.tls.Dtv

container

Source line 183

Dynamic Thread Vector as specified in the ELF TLS ABI. Ordinarily, there is a block pointer per dynamically-loaded module, but since we only support static TLS, we only need one block pointer.

Fields

len: usize = 1
tls_block: [*]u8

std.os.linux.tls.elf

alias

Source line 14

Alias of std.elf

std.os.linux.tls.getThreadPointer

function

Source line 383

pub fn getThreadPointer() usize

std.os.linux.tls.initStatic

function

Source line 654

Computes the layout of the static TLS area, allocates the area, initializes all of its fields, and assigns the architecture-specific value to the TP register.

pub fn initStatic(phdrs: []elf.ElfN.Phdr) void

Parameters

phdrs: []elf.ElfN.Phdr

std.os.linux.tls.linux

alias

Source line 19

Alias of std.os.linux

std.os.linux.tls.main_thread_area_buffer

variable

Source line 650

The main motivation for the size chosen here is to be larger than total amount of thread-local variables for most programs. Putting this allocation in the ELF like this is equivalent to moving the mmap call below into the kernel, avoiding syscall overhead.

[0x1000]u8

std.os.linux.tls.math

alias

Source line 15

Alias of std.math

std.os.linux.tls.mem

alias

Source line 13

Alias of std.mem

std.os.linux.tls.mmap_tls

function

Source line 682

inline fn mmap_tls(length: usize) usize

Parameters

length: usize

std.os.linux.tls.native_arch

constant

Source line 18

std.os.linux.tls.page_size_min

alias

Source line 20

Alias of std.heap.page_size_min

std.os.linux.tls.prepareArea

function

Source line 616

Initializes all the fields of the static TLS area and returns the computed architecture-specific value of the TP register.

pub fn prepareArea(area: []u8) usize

Parameters

area: []u8

std.os.linux.tls.setThreadPointer

function

Source line 220

pub fn setThreadPointer(addr: usize) void

Parameters

addr: usize

std.os.linux.tls.std

alias

Source line 12

Alias of std

std.os.linux.tls.TlsIndex

type

Source line 730

std.os.linux.tls.Variant

container

Source line 29

Represents an ELF TLS variant.

In all variants, the TP and the TLS blocks must be aligned to the p_align value in the PT.TLS ELF program header. Everything else has natural alignment.

The location of the DTV does not actually matter. For simplicity, we put it in the TLS area, but there is no actual ABI requirement that it reside there.

Fields

I_original

The original Variant I:


DTV Zig TCB ABI TCB TLS Blocks

----------------^----------------------- -- The TP register points here.

The layout in this variant necessitates separate alignment of both the TP and the TLS blocks.

The first word in the ABI TCB points to the DTV. For some architectures, there may be a second word with an unspecified meaning.

I_modified

The modified Variant I:


DTV Zig TCB ABI TCB TLS Blocks

------------------------------^------------- -- The TP register points here (*inside* the TLS blocks).

The offset from the start of the TLS blocks to the TP register is current_tp_offset. It may be zero, in which case the TP register points to the start of the TLS blocks.

The first (and only) word in the ABI TCB points to the DTV.

II

Variant II:


TLS Blocks ABI TCB Zig TCB DTV

-------------^-------------------------- -- The TP register points here.

The first (and only) word in the ABI TCB points to the ABI TCB itself.

std.os.linux.tls.ZigTcb

container

Source line 177

Per-thread storage for Zig's use. Currently unused.

Fields

dummy: usize
Source code

std/os/linux/tls.zig

//! This file implements the two TLS variants [1] used by ELF-based systems. Note that, in reality,
//! Variant I has two sub-variants.
//!
//! It is important to understand that the term TCB (Thread Control Block) is overloaded here.
//! Official ABI documentation uses it simply to mean the ABI TCB, i.e. a small area of ABI-defined
//! data, usually one or two words (see the `AbiTcb` type below). People will also often use TCB to
//! refer to the libc TCB, which can be any size and contain anything. (One could even omit it!) We
//! refer to the latter as the Zig TCB; see the `ZigTcb` type below.
//!
//! [1] https://www.akkadia.org/drepper/tls.pdf

const std = @import("std");
const mem = std.mem;
const elf = std.elf;
const math = std.math;
const assert = std.debug.assert;
const builtin = @import("builtin");
const native_arch = builtin.cpu.arch;
const linux = std.os.linux;
const page_size_min = std.heap.page_size_min;

/// Represents an ELF TLS variant.
///
/// In all variants, the TP and the TLS blocks must be aligned to the `p_align` value in the
/// `PT.TLS` ELF program header. Everything else has natural alignment.
///
/// The location of the DTV does not actually matter. For simplicity, we put it in the TLS area, but
/// there is no actual ABI requirement that it reside there.
const Variant = enum {
    /// The original Variant I:
    ///
    /// ----------------------------------------
    /// | DTV | Zig TCB | ABI TCB | TLS Blocks |
    /// ----------------^-----------------------
    ///                 `-- The TP register points here.
    ///
    /// The layout in this variant necessitates separate alignment of both the TP and the TLS
    /// blocks.
    ///
    /// The first word in the ABI TCB points to the DTV. For some architectures, there may be a
    /// second word with an unspecified meaning.
    I_original,
    /// The modified Variant I:
    ///
    /// --------------------------------------------
    /// | DTV | Zig TCB | ABI TCB |   TLS Blocks   |
    /// ------------------------------^-------------
    ///                               `-- The TP register points here (*inside* the TLS blocks).
    ///
    /// The offset from the start of the TLS blocks to the TP register is `current_tp_offset`. It
    /// may be zero, in which case the TP register points to the start of the TLS blocks.
    ///
    /// The first (and only) word in the ABI TCB points to the DTV.
    I_modified,
    /// Variant II:
    ///
    /// ----------------------------------------
    /// | TLS Blocks | ABI TCB | Zig TCB | DTV |
    /// -------------^--------------------------
    ///              `-- The TP register points here.
    ///
    /// The first (and only) word in the ABI TCB points to the ABI TCB itself.
    II,
};

const current_variant: Variant = switch (native_arch) {
    .aarch64,
    .aarch64_be,
    .alpha,
    .arc,
    .arceb,
    .arm,
    .armeb,
    .csky,
    .hppa,
    .microblaze,
    .microblazeel,
    .sh,
    .sheb,
    .thumb,
    .thumbeb,
    .xtensa,
    .xtensaeb,
    => .I_original,
    .loongarch32,
    .loongarch64,
    .m68k,
    .mips,
    .mipsel,
    .mips64,
    .mips64el,
    .or1k,
    .powerpc,
    .powerpcle,
    .powerpc64,
    .powerpc64le,
    .riscv32,
    .riscv64,
    => .I_modified,
    .hexagon,
    .s390x,
    .sparc,
    .sparc64,
    .x86,
    .x86_64,
    => .II,
    else => @compileError("undefined TLS variant for this architecture"),
};

/// The offset value for the modified Variant I.
const current_tp_offset = switch (native_arch) {
    .m68k,
    .mips,
    .mipsel,
    .mips64,
    .mips64el,
    .powerpc,
    .powerpcle,
    .powerpc64,
    .powerpc64le,
    => 0x7000,
    else => 0,
};

/// Usually only used by the modified Variant I.
const current_dtv_offset = switch (native_arch) {
    .m68k,
    .mips,
    .mipsel,
    .mips64,
    .mips64el,
    .powerpc,
    .powerpcle,
    .powerpc64,
    .powerpc64le,
    => 0x8000,
    .riscv32,
    .riscv64,
    => 0x800,
    else => 0,
};

/// Per-thread storage for the ELF TLS ABI.
const AbiTcb = switch (current_variant) {
    .I_original, .I_modified => switch (native_arch) {
        .aarch64,
        .aarch64_be,
        .alpha,
        .arm,
        .armeb,
        .hppa,
        .microblaze,
        .microblazeel,
        .sh,
        .sheb,
        .thumb,
        .thumbeb,
        .xtensa,
        .xtensaeb,
        => extern struct {
            /// This is offset by `current_dtv_offset`.
            dtv: usize,
            _reserved: ?*anyopaque,
        },
        else => extern struct {
            /// This is offset by `current_dtv_offset`.
            dtv: usize,
        },
    },
    .II => extern struct {
        /// This is self-referential.
        self: *AbiTcb,
    },
};

/// Per-thread storage for Zig's use. Currently unused.
const ZigTcb = struct {
    dummy: usize,
};

/// Dynamic Thread Vector as specified in the ELF TLS ABI. Ordinarily, there is a block pointer per
/// dynamically-loaded module, but since we only support static TLS, we only need one block pointer.
const Dtv = extern struct {
    len: usize = 1,
    tls_block: [*]u8,
};

/// Describes a process's TLS area. The area encompasses the DTV, both TCBs, and the TLS block, with
/// the exact layout of these being dependent primarily on `current_variant`.
const AreaDesc = struct {
    size: usize,
    alignment: usize,

    dtv: struct {
        /// Offset into the TLS area.
        offset: usize,
    },

    abi_tcb: struct {
        /// Offset into the TLS area.
        offset: usize,
    },

    block: struct {
        /// The initial data to be copied into the TLS block. Note that this may be smaller than
        /// `size`, in which case any remaining data in the TLS block is simply left uninitialized.
        init: []const u8,
        /// Offset into the TLS area.
        offset: usize,
        /// This is the effective size of the TLS block, which may be greater than `init.len`.
        size: usize,
    },

    /// Only used on the 32-bit x86 architecture (not x86_64, nor x32).
    gdt_entry_number: usize,
};

pub var area_desc: AreaDesc = undefined;

pub fn setThreadPointer(addr: usize) void {
    @setRuntimeSafety(false);
    @disableInstrumentation();

    switch (native_arch) {
        .x86 => {
            var user_desc: linux.user_desc = .{
                .entry_number = area_desc.gdt_entry_number,
                .base_addr = addr,
                .limit = 0xfffff,
                .flags = .{
                    .seg_32bit = 1,
                    .contents = 0, // Data
                    .read_exec_only = 0,
                    .limit_in_pages = 1,
                    .seg_not_present = 0,
                    .useable = 1,
                },
            };
            const rc = @call(.always_inline, linux.syscall1, .{ .set_thread_area, @intFromPtr(&user_desc) });
            assert(rc == 0);

            const gdt_entry_number = user_desc.entry_number;
            // We have to keep track of our slot as it's also needed for clone()
            area_desc.gdt_entry_number = gdt_entry_number;
            // Update the %gs selector
            asm volatile ("movl %[gs_val], %%gs"
                :
                : [gs_val] "r" (gdt_entry_number << 3 | 3),
            );
        },
        .x86_64 => {
            const rc = @call(.always_inline, linux.syscall2, .{ .arch_prctl, linux.ARCH.SET_FS, addr });
            assert(rc == 0);
        },
        .aarch64, .aarch64_be => {
            asm volatile (
                \\ msr tpidr_el0, %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .alpha => {
            asm volatile (
                \\ wruniq
                :
                : [addr] "{$16}" (addr),
            );
        },
        .arc, .arceb => {
            // We apparently need to both set r25 (TP) *and* inform the kernel...
            asm volatile (
                \\ mov r25, %[addr]
                :
                : [addr] "r" (addr),
            );
            const rc = @call(.always_inline, linux.syscall1, .{ .arc_settls, addr });
            assert(rc == 0);
        },
        .arm, .armeb, .thumb, .thumbeb => {
            const rc = @call(.always_inline, linux.syscall1, .{ .set_tls, addr });
            assert(rc == 0);
        },
        .m68k => {
            const rc = @call(.always_inline, linux.syscall1, .{ .set_thread_area, addr });
            assert(rc == 0);
        },
        .hexagon => {
            asm volatile (
                \\ ugp = %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .hppa => {
            asm volatile (
                \\ ble 0xe0(%%sr2, %%r0)
                \\  nop
                :
                : [addr] "{r26}" (addr),
                : .{ .r31 = true });
        },
        .loongarch32, .loongarch64 => {
            asm volatile (
                \\ move $tp, %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .riscv32, .riscv64 => {
            asm volatile (
                \\ mv tp, %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .csky, .mips, .mipsel, .mips64, .mips64el => {
            const rc = @call(.always_inline, linux.syscall1, .{ .set_thread_area, addr });
            assert(rc == 0);
        },
        .microblaze, .microblazeel => {
            asm volatile (
                \\ ori r21, %[addr], 0
                :
                : [addr] "r" (addr),
            );
        },
        .or1k => {
            asm volatile (
                \\ l.ori r10, %[addr], 0
                :
                : [addr] "r" (addr),
            );
        },
        .powerpc, .powerpcle => {
            asm volatile (
                \\ mr 2, %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .powerpc64, .powerpc64le => {
            asm volatile (
                \\ mr 13, %[addr]
                :
                : [addr] "r" (addr),
            );
        },
        .s390x => {
            asm volatile (
                \\ lgr %%r0, %[addr]
                \\ sar %%a1, %%r0
                \\ srlg %%r0, %%r0, 32
                \\ sar %%a0, %%r0
                :
                : [addr] "r" (addr),
                : .{ .r0 = true });
        },
        .sh, .sheb => {
            asm volatile (
                \\ ldc %[addr], gbr
                :
                : [addr] "r" (addr),
            );
        },
        .sparc, .sparc64 => {
            asm volatile (
                \\ mov %[addr], %%g7
                :
                : [addr] "r" (addr),
            );
        },
        .xtensa, .xtensaeb => {
            asm volatile (
                \\ wur %[addr], threadptr
                :
                : [addr] "a" (addr),
            );
        },
        else => @compileError("Unsupported architecture"),
    }
}

pub fn getThreadPointer() usize {
    @setRuntimeSafety(false);
    @disableInstrumentation();

    return switch (native_arch) {
        .aarch64, .aarch64_be => asm (
            \\ mrs %[ret], tpidr_el0
            : [ret] "=r" (-> usize),
        ),
        .alpha => asm (
            \\ rduniq
            : [ret] "={$0}" (-> usize),
        ),
        .arc, .arceb => asm (
            \\ mov %[ret], r25
            : [ret] "=r" (-> usize),
        ),
        .arm, .armeb, .thumb, .thumbeb => asm (
            \\ mrc p15, 0, %[ret], c13, c0, 3
            : [ret] "=r" (-> usize),
        ),
        .csky => asm (
            \\ mov %[ret], r31
            : [ret] "=r" (-> usize),
        ),
        .hexagon => asm (
            \\ %[ret] = ugp
            : [ret] "=r" (-> usize),
        ),
        .hppa => asm (
            \\ mfctl %%cr27, %[ret]
            : [ret] "=r" (-> usize),
        ),
        .loongarch32, .loongarch64 => asm (
            \\ move %[ret], $tp
            : [ret] "=r" (-> usize),
        ),
        .m68k => linux.syscall0(.get_thread_area),
        .mips, .mipsel, .mips64, .mips64el => asm (
            \\ rdhwr %[ret], $29
            : [ret] "=r" (-> usize),
        ),
        .microblaze, .microblazeel => asm (
            \\ ori %[ret], r21, 0
            : [ret] "=r" (-> usize),
        ),
        .or1k => asm (
            \\ l.ori %[ret], r10, 0
            : [ret] "=r" (-> usize),
        ),
        .riscv32, .riscv64 => asm (
            \\ mv %[ret], tp
            : [ret] "=r" (-> usize),
        ),
        .powerpc, .powerpcle => asm (
            \\ mr %[ret], 2
            : [ret] "=r" (-> usize),
        ),
        .powerpc64, .powerpc64le => asm (
            \\ mr %[ret], 13
            : [ret] "=r" (-> usize),
        ),
        .s390x => asm (
            \\ ear %[ret], %%a0
            \\ sllg %[ret], %[ret], 32
            \\ ear %[ret], %%a1
            : [ret] "=r" (-> usize),
        ),
        .sh, .sheb => asm (
            \\ stc gbr, %[ret]
            : [ret] "=r" (-> usize),
        ),
        .sparc, .sparc64 => asm (
            \\ mov %%g7, %[ret]
            : [ret] "=r" (-> usize),
        ),
        .x86 => asm (
            \\ movl %%gs:0, %[ret]
            : [ret] "=r" (-> usize),
        ),
        .x86_64 => switch (@sizeOf(usize)) {
            8 => asm (
                \\ movq %%fs:0, %[ret]
                : [ret] "=r" (-> usize),
            ),
            // On x32, usize is 32 bits.
            4 => asm (
                \\ movl %%fs:0, %[ret]
                : [ret] "=r" (-> usize),
            ),
            else => comptime unreachable,
        },
        .xtensa, .xtensaeb => asm (
            \\ rur %[ret], threadptr
            : [ret] "=r" (-> usize),
        ),
        else => @compileError("Unsupported architecture"),
    };
}

fn computeAreaDesc(phdrs: []elf.ElfN.Phdr) void {
    @setRuntimeSafety(false);
    @disableInstrumentation();

    var tls_phdr: ?*elf.ElfN.Phdr = null;
    var img_base: usize = 0;

    for (phdrs) |*phdr| {
        switch (phdr.type) {
            .PHDR => img_base = @intFromPtr(phdrs.ptr) - phdr.vaddr,
            .TLS => tls_phdr = phdr,
            else => {},
        }
    }

    var align_factor: usize = undefined;
    var block_init: []const u8 = undefined;
    var block_size: usize = undefined;

    if (tls_phdr) |phdr| {
        align_factor = phdr.@"align";

        // The effective size in memory is represented by `memsz`; the length of the data stored
        // in the `PT.TLS` segment is `filesz` and may be less than the former.
        block_init = @as([*]u8, @ptrFromInt(img_base + phdr.vaddr))[0..phdr.filesz];
        block_size = phdr.memsz;
    } else {
        align_factor = @alignOf(usize);

        block_init = &[_]u8{};
        block_size = 0;
    }

    // Offsets into the allocated TLS area.
    var dtv_offset: usize = undefined;
    var abi_tcb_offset: usize = undefined;
    var block_offset: usize = undefined;

    // Compute the total size of the ABI-specific data plus our own `ZigTcb` structure. All the
    // offsets calculated here assume a well-aligned base address.
    const area_size = switch (current_variant) {
        .I_original => blk: {
            var l: usize = 0;
            dtv_offset = l;
            l += @sizeOf(Dtv);
            // Add some padding here so that the TP (`abi_tcb_offset`) is aligned to `align_factor`
            // and the `ZigTcb` structure can be found by simply subtracting `@sizeOf(ZigTcb)` from
            // the TP.
            const delta = (l + @sizeOf(ZigTcb)) & (align_factor - 1);
            if (delta > 0)
                l += align_factor - delta;
            l += @sizeOf(ZigTcb);
            abi_tcb_offset = l;
            l += alignForward(@sizeOf(AbiTcb), align_factor);
            block_offset = l;
            l += block_size;
            break :blk l;
        },
        .I_modified => blk: {
            var l: usize = 0;
            dtv_offset = l;
            l += @sizeOf(Dtv);
            // In this variant, the TLS blocks must begin immediately after the end of the ABI TCB,
            // with the TP pointing to the beginning of the TLS blocks. Add padding so that the TP
            // (`abi_tcb_offset`) is aligned to `align_factor` and the `ZigTcb` structure can be
            // found by subtracting `@sizeOf(AbiTcb) + @sizeOf(ZigTcb)` from the TP.
            const delta = (l + @sizeOf(ZigTcb) + @sizeOf(AbiTcb)) & (align_factor - 1);
            if (delta > 0)
                l += align_factor - delta;
            l += @sizeOf(ZigTcb);
            abi_tcb_offset = l;
            l += @sizeOf(AbiTcb);
            block_offset = l;
            l += block_size;
            break :blk l;
        },
        .II => blk: {
            var l: usize = 0;
            block_offset = l;
            l += alignForward(block_size, align_factor);
            // The TP is aligned to `align_factor`.
            abi_tcb_offset = l;
            l += @sizeOf(AbiTcb);
            // The `ZigTcb` structure is right after the `AbiTcb` with no padding in between so it
            // can be easily found.
            l += @sizeOf(ZigTcb);
            // It doesn't really matter where we put the DTV, so give it natural alignment.
            l = alignForward(l, @alignOf(Dtv));
            dtv_offset = l;
            l += @sizeOf(Dtv);
            break :blk l;
        },
    };

    area_desc = .{
        .size = area_size,
        .alignment = align_factor,

        .dtv = .{
            .offset = dtv_offset,
        },

        .abi_tcb = .{
            .offset = abi_tcb_offset,
        },

        .block = .{
            .init = block_init,
            .offset = block_offset,
            .size = block_size,
        },

        .gdt_entry_number = @as(usize, @bitCast(@as(isize, -1))),
    };
}

/// Inline because TLS is not set up yet.
inline fn alignForward(addr: usize, alignment: usize) usize {
    return alignBackward(addr + (alignment - 1), alignment);
}

/// Inline because TLS is not set up yet.
inline fn alignBackward(addr: usize, alignment: usize) usize {
    return addr & ~(alignment - 1);
}

/// Inline because TLS is not set up yet.
inline fn alignPtrCast(comptime T: type, ptr: [*]u8) *T {
    return @ptrCast(@alignCast(ptr));
}

/// Initializes all the fields of the static TLS area and returns the computed architecture-specific
/// value of the TP register.
pub fn prepareArea(area: []u8) usize {
    @setRuntimeSafety(false);
    @disableInstrumentation();

    // Clear the area we're going to use, just to be safe.
    @memset(area, 0);

    // Prepare the ABI TCB.
    const abi_tcb = alignPtrCast(AbiTcb, area.ptr + area_desc.abi_tcb.offset);
    switch (current_variant) {
        .I_original, .I_modified => abi_tcb.dtv = @intFromPtr(area.ptr + area_desc.dtv.offset),
        .II => abi_tcb.self = abi_tcb,
    }

    // Prepare the DTV.
    const dtv = alignPtrCast(Dtv, area.ptr + area_desc.dtv.offset);
    dtv.len = 1;
    dtv.tls_block = area.ptr + current_dtv_offset + area_desc.block.offset;

    // Copy the initial data.
    @memcpy(area[area_desc.block.offset..][0..area_desc.block.init.len], area_desc.block.init);

    // Return the corrected value (if needed) for the TP register. Overflow here is not a problem;
    // the pointer arithmetic involving the TP is done with wrapping semantics.
    return @intFromPtr(area.ptr) +% switch (current_variant) {
        .I_original, .II => area_desc.abi_tcb.offset,
        .I_modified => area_desc.block.offset +% current_tp_offset,
    };
}

/// The main motivation for the size chosen here is to be larger than total
/// amount of thread-local variables for most programs. Putting this allocation
/// in the ELF like this is equivalent to moving the `mmap` call below into the
/// kernel, avoiding syscall overhead.
var main_thread_area_buffer: [0x1000]u8 align(page_size_min) = undefined;

/// Computes the layout of the static TLS area, allocates the area, initializes all of its fields,
/// and assigns the architecture-specific value to the TP register.
pub fn initStatic(phdrs: []elf.ElfN.Phdr) void {
    @setRuntimeSafety(false);
    @disableInstrumentation();

    computeAreaDesc(phdrs);

    const area = blk: {
        // Fast path for the common case where the TLS data is really small, avoid an allocation and
        // use our local buffer.
        if (area_desc.alignment <= page_size_min and area_desc.size <= main_thread_area_buffer.len) {
            break :blk main_thread_area_buffer[0..area_desc.size];
        }

        const begin_addr = mmap_tls(area_desc.size + area_desc.alignment - 1);
        if (@call(.always_inline, linux.errno, .{begin_addr}) != .SUCCESS) @trap();

        const area_ptr: [*]align(page_size_min) u8 = @ptrFromInt(begin_addr);

        // Make sure the slice is correctly aligned.
        const begin_aligned_addr = alignForward(begin_addr, area_desc.alignment);
        const start = begin_aligned_addr - begin_addr;
        break :blk area_ptr[start..][0..area_desc.size];
    };

    const tp_value = prepareArea(area);
    setThreadPointer(tp_value);
}

inline fn mmap_tls(length: usize) usize {
    const prot: linux.PROT = .{ .READ = true, .WRITE = true };
    const flags: linux.MAP = .{ .TYPE = .PRIVATE, .ANONYMOUS = true };

    if (@hasField(linux.SYS, "mmap2")) {
        return @call(.always_inline, linux.syscall6, .{
            .mmap2,
            0,
            length,
            @as(u32, @bitCast(prot)),
            @as(u32, @bitCast(flags)),
            @as(usize, @bitCast(@as(isize, -1))),
            0,
        });
    } else {
        // The s390x mmap() syscall existed before Linux supported syscalls with 5+ parameters, so
        // it takes a single pointer to an array of arguments instead.
        return if (native_arch == .s390x) @call(.always_inline, linux.syscall1, .{
            .mmap,
            @intFromPtr(&[_]usize{
                0,
                length,
                @as(u32, @bitCast(prot)),
                @as(u32, @bitCast(flags)),
                @as(usize, @bitCast(@as(isize, -1))),
                0,
            }),
        }) else @call(.always_inline, linux.syscall6, .{
            .mmap,
            0,
            length,
            @as(u32, @bitCast(prot)),
            @as(u32, @bitCast(flags)),
            @as(usize, @bitCast(@as(isize, -1))),
            0,
        });
    }
}

comptime {
    assert(!builtin.link_libc); // otherwise libc should control TLS

    if (builtin.output_mode == .Exe and builtin.link_mode == .static) {
        // This is a static executable without libc, so it is our job to provide the TLS accessor
        // function for the GD and LD models. This function is unlikely to actually be used, since
        // the linker should be able to relax every TLS access to the LE model and therefore
        // eliminate all calls to this function, but that isn't guaranteed.
        const Fns = struct {
            const TlsIndex = switch (native_arch) {
                .x86_64 => extern struct { module: u64, offset: u64 }, // Even for x32...
                else => extern struct { module: usize, offset: usize }, // ...but not MIPS N32!
            };
            fn __tls_get_addr(ti: *const TlsIndex) callconv(.c) *anyopaque {
                comptime assert(native_arch != .s390x);

                assert(ti.module == 1); // The executable's module ID is always 1
                const tp = getThreadPointer();
                const block: [*]u8 = switch (current_variant) {
                    .I_original => @ptrFromInt(tp -% area_desc.abi_tcb.offset +% area_desc.block.offset),
                    .I_modified => @ptrFromInt(tp -% current_tp_offset),
                    // The `.I_original` approach would also work for `.II`, but there is an
                    // alternative strategy which is one less operation:
                    .II => @ptrFromInt(tp -% area_desc.block.size),
                };
                return block[@intCast(ti.offset)..];
            }
            fn __tls_get_offset() callconv(.naked) noreturn {
                comptime assert(native_arch == .s390x);

                // We receive the module's GOT pointer in r12 and the GOT offset in r2.
                asm volatile (
                    \\ la %%r1, 0(%%r12, %%r2)
                    \\ lg %%r2, 8(%%r1)
                    \\ lgrl %%r0, %[block_size]
                    \\ sgr %%r2, %%r0
                    \\ br %%r14
                    :
                    : [block_size] "s" (&area_desc.block.size),
                );
            }
        };

        if (native_arch == .s390x)
            @export(&Fns.__tls_get_offset, .{ .name = "__tls_get_offset" })
        else
            @export(&Fns.__tls_get_addr, .{ .name = "__tls_get_addr" });
    }
}