beamlet, the Elixir VM

beamlet is a BEAM interpreter in safe Rust: it runs Erlang and Elixir code compiled by the standard compiler, OTP 28 with Elixir 1.20. Every session and every agent on Redoubt is one beamlet VM. The VM gets from its embedder, through one Rust trait (Platform), exactly the services it is granted (a clock, a console, random bytes, code, files, programs) and nothing else, and it treats every .beam file, literal and message as hostile input. beamlet runs on the host and boots the shell on Redoubt with verified modules from the userland disk, and its files are 9P files in its namespace. Native launching on Redoubt remains planned.

Purpose

A session needs a language, a standard library and a prompt, and it needs them without C, without a POSIX kernel underneath, and small enough to audit. The BEAM gives Elixir and Erlang, OTP’s libraries, the compilers, supervision and message passing; beamlet gives the BEAM in a form Redoubt can trust: its own crates hold no unsafe, a loader checks everything before code runs, limits fail closed, and one narrow boundary is where the operating system comes in. The code is in userland/otp.

How to use it

On the host, beamlet is a command:

$ cargo run -p beamlet -- -pa ebin my_module start
$ cargo run -p beamlet -- --root ./sandbox -pa ebin my_module start     # expose one directory
$ cargo run -p beamlet -- --mount /data=./data:ro --root ./sandbox ...   # add a read-only mount
$ cargo run -p beamlet -- --exec --schedulers 4 ...                      # grant programs; 4 threads

Without --root, file calls fail with enotsup; without --exec, opening a port to a program fails with eacces. The VM’s environment starts empty, with HOME set to / when there is a --root (--env NAME[=VALUE] adds to it), so the host’s is not visible. Tests: cargo test in userland/otp runs the unit and hostile-input tests; the differential suites (tools/difftest, tools/elixir-tests) need the pinned OTP and Elixir toolchains installed.

On Redoubt there is no command to run: the steward starts a session’s VM when a person logs in (sessions), and a launcher starts an agent’s (agents). What the code in the VM sees is ordinary Elixir: File.read!/1, IO.puts/1, :gen_tcp.connect/3.

What it can and cannot do

Loading hostile code

The VM crate (beamlet-vm) is #![forbid(unsafe_code)], and so are beamlet-re and beamlet-crypto (userland/otp/vm/src/lib.rs).

• One compiler. The loader accepts bytecode from OTP 28 only. It refuses deprecated or unknown opcodes, old atom tables and anything else another version would need compatibility code for.
• Checked before it runs. The loader checks every table index, label, register number and literal before any code runs (userland/otp/vm/src/loader.rs). A malformation that survives loading is bad_code at run time: the process that ran it dies, uncatchably. Nothing in the VM panics on bad input. Every truncation of real .beam files is refused, and 20,000 mutated modules per run load or fail without a panic or a hang.
• Deep terms do not recurse. Copying, comparing, printing and collecting use work lists, never Rust recursion, so a million-level nested term is fine. The external term format limits nesting to 256.
• Safe decoding names no code. binary_to_term(Bin, [safe]) creates no atoms, decodes an export fun only of a function exported now (a loaded module’s export, or a BIF), as BEAM does, and decodes no local fun at all, where BEAM decodes one whose atoms exist: a fun is code to run, with free variables the sender chose, and safe mode is for data from outside (userland/otp/vm/src/etf.rs).

Limits inside one VM

One VM is one trust domain, but a buggy or hostile Erlang process must not take the rest of the VM down. Every limit fails closed: the offender ends, and nothing is lost silently (userland/otp/vm/src/vm.rs).

• Mailbox (max_mailbox, 2^20 messages): a message that would overflow a mailbox kills the receiver with {system_limit, message_queue}, untrappably. Dropping it would break protocols silently, like a TCP stream with a hole in it.
• Process memory (max_heap_words, 2^27 words, and max_heap_size, which a process can only lower): checked at the end of each slice; a process over it is collected first and killed only if what is live is still over. A resource whose native declares its size counts that size as the memory of each process whose heap holds it, toward max_heap_size as heap words do: a screen buffer; an atomics or counters array, its cells at 8 bytes each, up to 2^24 of them; and a zlib stream, its queued input (at most 256 MB), the output it has not handed out, its codec’s state (a deflater’s includes the tables its library keeps behind boxes of its own, measured against the library by a test) and the term stashed in it, declared afresh by every native that changes them. A process past its limit through any of them is ended as for heap growth. Only a process’s heap counts one toward its limit, and a holder that is not the one resizing it counts the new size from its next collection. A resource in ETS or persistent_term counts toward their limits, below. One in a message not yet received counts toward nobody’s: the screen buffer’s limit of four a process and the mailbox’s limit bound a buffer there, but an atomics array or a zlib stream sent and dropped by its sender is bounded only by the budget’s page limit, below.
• ETS (max_ets_words, 2^27 words for all tables together): an insert past it raises system_limit, as does an update_counter or update_element that would take it there. A table’s heir data ({heir, Pid, Data}) is kept with the table and counts as an object does: an ets:new or ets:setopts that would take ETS past the limit with it raises system_limit. A resource an object holds counts at its live size, once for each object holding it: a zlib stream that grows after its insert counts its growth at once, so the next insert past the limit is refused (the growth itself is bounded by the grower’s own heap limit). ETS keeps its total as objects enter and leave and resources resize, never by scanning: an insert, delete or replacement costs one step for each resource on the objects it adds or removes, a resize one step, and deleting a table one step for each resource its objects hold.
• persistent_term (max_persistent_words, 2^27 words): its keys, and every value ever put, each resource at its live size as in ETS. A put past it raises system_limit, as does a put of a new key past 2^16 keys. A value replaced or erased is never freed: it stays in its literal chunk, where BEAM frees it once no process refers to it, so its words count for good. An erased key’s own words are released. A value’s parts that are already literals (another persistent value, a module’s constant) are not copied, and count only where they were first put. persistent_term:info() reports the count as its memory.
• CPU: reductions preempt every process, including a loop of plain jumps with no calls. A native is not preempted, so what one may compute is bounded before it runs: crypto:mod_pow and finite-field Diffie-Hellman hold the modulus (P), the base and the exponent each to 10,000 bits, OpenSSL’s OPENSSL_DH_MAX_MODULUS_BITS, and an RSA key to 8,192 bits, its D and primes to N’s width; within the bound every call answers as BEAM’s does, and a wider operand is refused with badarg before any arithmetic, where the unbounded call on a megabyte’s would hold its scheduler thread, and every process on it, for days.
• Fixed limits, each system_limit: 2^20 atoms of at most 255 characters, 2^16 processes, a stack of 2^24 slots, bignums of 2^24 bits, binaries of 2^30 bits.
• A term is equal to itself without being read. Comparing a term with itself (==, =:=, a map key, a sort) answers before a byte of it is touched, as BEAM’s eq does: sound for every Erlang term, since the language has no NaN and so no term that differs from itself. It matters at the prompt: erl_eval compares every binding with itself on each call of an interpreted fun, so a loop typed at the prompt over a 256 KiB binary once cost a scan of it a step (a_term_equal_to_itself_is_not_scanned).

These limits are measurements, not an allocator: one native that allocates a lot at once is caught afterwards. The hard backstop is the embedder’s allocator, and on Redoubt the session budget’s page limit (R6 (charging)). The image also caps beamlet’s heap (init), but at the budget’s edge, where the budget binds first: that cap is there for the bench’s measurement. On Redoubt the platform lowers max_heap_words, max_ets_words and max_persistent_words to a sixteenth of the VM’s budget each, which it reads from its startup block’s budget_pages (init): the launcher that made the budget fills it, init from the manifest’s budget.pages, the steward with the VM’s share of its session’s budget, so no argument has to be kept equal to the budget by hand, and a manifest that names one is refused by the bench (host:testbench::no_manifest_names_budget_pages). Each counts the VM’s own 8-byte words, as the VM counts a process (two for a 16-byte term), so a limit is the same bytes on rv32 as on rv64. A flooding process peaks at about four times its heap limit, the old heap, the collector’s copy and its growth, so the budget must be at least twice what the VM uses with no Erlang process running; then one flooding process, the tables or persistent_term meets its limit while the VM still has pages. A session’s sixteenth is 696 pages, 2,850,816 bytes; the shell at its prompt holds 3,696 bytes in ETS and 36,112 counted in persistent_term, about a thousandth and a hundredth of it. Several flooding at once, or a native’s single large allocation, reach the backstop instead, which ends the VM, and init restarts it. It is a server like any other under init’s restart rule: a VM that cannot stay up (a start module that fails every time, a startup block that tells it no budget, exit code 4) is restarted until the limit, and then the machine reboots (init). CPU between VMs is the kernel’s to share, by budget weight (scheduling).

What the VM holds at its prompt

Started with the argument report_memory, which the image never passes, the VM prints what it holds the first time it waits for console input with nothing else to run, for the shell at its prompt, one console line a row (memory.rs). It is the VM’s own count: each allocation as the runtime’s heap holds it (a power-of-two block up to 2 KiB, whole pages above), every waiting process collected first; B-tree nodes are not counted. beamlet adds its runtime heap’s pages held at that moment and its record’s peak, and the bench’s scan of that record (the memory budget) is the independent total, read once the case has typed one command. In pages, each row rounded to whole pages on its own:

A module’s instructions are 8-byte entries over one array of its operands, 16 bytes each on either width, a list operand’s items in the same array, so decoded code is the same size on both widths and still about two thirds of the count. The five largest modules are unicode_util, erl_parse, Elixir.Enum, Elixir.Kernel and string. What is not accounted is free small blocks, B-tree nodes and the platform’s buffers; the peak above what is held is the boot pack, which is held until the prompt, and a load’s transient, and the scan’s peak is higher than the prompt’s because the first command loads more modules.

• Loaded code keeps no spare room. The loader shrinks a module’s instructions and its operand array to their counts once decoded, where the decoding’s doubling left up to twice as many.
• A literal chunk keeps no spare room. A module’s constants are shrunk to their count before they join the shared table (about 350 pages on either width).
• Loading stays eager, but for the shell’s commands. The shell’s start loads what its prompt needs. The largest modules are the ones evaluating any line needs, so loading them on first call would move their pages to the first command, not save them. A session calls few of its commands, so a command’s module is loaded when the command is first called (the shell), and the boot pack holds none of them.

The image budgets the VM twice the largest peak the scan finds across its memory cases, in beamlet-footprint, the one case that scans a shell’s VM now that the steward starts the others’ shells, plus what the VM holds outside its heap, rounded up to 128 pages: 11,264 pages from the largest peak, 5,556 on rv64 at one hart, which its heap cap of 11,220 holds twice with 108 to spare, and with which the single VM boots in 512 MiB (budgets). Outside the heap are 44 pages: its two schedulers’ 18-page stacks (the second’s above a guard page of address space alone, which costs no page; its peak at the prompt is 21,496 to 25,200 bytes on rv64 over three runs, stack beamlet thread 3 in beamlet-footprint’s scan) and thread pages, up to five pages of endpoints its locks make, and its process’s own page. The share moves up a step of 128 when the peak passes what the cap holds twice: the rv64 peak may reach 5,610 pages before it moves again (budgets). The shell’s machine mode is 19 of the pages at the prompt, the driver’s and the loop’s code. The line editor under the shell’s driver is loaded at the prompt: OTP’s group, edlin, edlin_key, group_history, prim_tty, shell, gen_statem, sys and kernel, with Redoubt.Term and the driver. The shell’s protocols are not consolidated, but nothing at the prompt, nor a plain line, dispatches a protocol on a struct, so Elixir’s Protocol and Enumerable load only for a line that does.

That rule sizes the idle VM, and a compile is a workload within it: beamlet-compile-footprint types a three-line defmodule at the prompt under the same manifest, calls the module, and judges that the compile fits the session’s cap with at least an eighth of it to spare (heap_fits, the memory budget). The compiler’s modules stay loaded, so its heap peak is 9,135 pages on rv64 and 8,798 on rv32 of the 11,220 cap, 2,085 and 2,422 pages under it (an eighth is 1,402). It runs one scheduler, so every process is on the first thread, the one the scan paints: with two, the compile runs on whichever scheduler holds the shell’s evaluator, and the second thread’s stack is not measured. Its first thread’s stack peaks at 34,280 bytes on rv64 and 25,832 on rv32, under beamlet-footprint’s idle 36,136, so the 18-page declaration stands. It was 88,888 and 81,056 while term_to_binary(T, [compressed]), which the compiler calls for a chunk, kept miniz_oxide’s whole compressor, 65,712 bytes of it a code buffer, as a local of its own function: one native frame larger than the stack left under the interpreter’s, which defmodule at the prompt ended with a store page fault below the first scheduler’s stack on rv64 and rv32 alike, and zlib:deflateInit/2 built the same temporary on the stack before boxing it (on rv32 most times, not every time). Redoubt’s patch to the crate boxes the code buffer under with-alloc, as the crate does on wasm32 (patched crates), so the state is a few hundred bytes wherever it is built and the frame is gone. Residual: a session that has compiled has spent its flood headroom on the compiler’s code. The sixteenth limits are static, so one flooding process after a compile reaches the budget’s backstop before its own limit, an allocation fails, the VM ends and init restarts it: a logout, within the session’s own budget and no other principal’s, where before the compile the process would have been killed alone (beamlet-compile-flood, both widths: the allocation that fails is about 1.2 MiB on rv64 and 2.4 MiB on rv32, from one run of each). The follow-up is the VM deriving its limits from the pages not held by loaded code, recomputed after a load, which keeps the guarantee at any footprint without resizing a session.

Residual: QEMU’s default 256 MiB is out of reach. Its system budget leaves the VM about 2,700 pages beside the other servers, and the VM holds more than that at its prompt (the table above), most of it decoded code and the shared literal table. Fitting needs code kept in its on-disk form or loaded a function at a time.

The Platform boundary

Everything the VM gets from outside comes through the Platform trait (userland/otp/vm/src/platform.rs):

• load_module is a lookup, not a gate. It asks the platform for a module before examining the code path. Found uses those bytes, so no directory shadows a system module (packages); Absent searches the code path’s directories in order, including those added with code:add_patha/1; Refused stops without touching the code path (userland/otp/vm/src/vm.rs, locate_module). On Redoubt a name the userland volume’s erofsd answers not_found to is absent, while any other refusal at the open or on the read (corrupt, a short or long file, a device error) is refused, with one console line naming the file and the error’s name (R75 (verified userland)). load_app carries the same three outcomes: beamlet:app_spec/1 makes one platform attempt and returns the bytes on Found, or error on Absent and Refused. Residual: the bundle’s protocols are not consolidated when it is built, so a protocol consolidated in a session’s own directory is not used and protocol dispatch stays the slower, unconsolidated kind; behaviour is the same. Consolidating them when the bundle is built is not done, at the cost of that speed and of Elixir’s Protocol module once a line dispatches on a struct: a defimpl typed at the prompt for a protocol already consolidated would be ignored. Code in the VM can also load any bytes it holds with code:load_binary/3 (userland/otp/vm/src/bif/info.rs), through the same loader checks. So what confines loaded code is not how it arrived but what the VM holds: every module, however loaded, reaches only what the Platform grants. Loading one’s own bytecode acts within one’s own authority.
• Files are the platform’s. OTP’s own file, file_server and file_io_server run unchanged over a Files trait, path-based and POSIX-shaped: open, read, pread, pwrite, seek, info, list_dir, rename, delete and the rest; a 9P client implementing it is Redoubt’s, below. Names resolve inside the VM, relative to its own working directory, with . and .. resolved lexically, so no name climbs above /. What / is, is the platform’s choice, and the platform must still refuse what the VM cannot see, such as a symbolic link out of a mount. An open file belongs to the Erlang process that opened it and closes when it exits; a VM has at most 1024 open files.
• Programs are a large grant. A program is outside the VM altogether, so programs defaults to none. The host CLI grants it only with --exec, and the programs it then starts are host processes with the user’s own rights, not sandboxed.
• The host embedding exposes one directory with --root through cap-std, and more with --mount, read-only if asked; a symbolic link resolves within the VM’s own name space, and a link that would leave a mount is refused.

What runs on it

Where beamlet implements something, it behaves as the real BEAM does, and the differential suite checks it: each test runs on BEAM and on beamlet and the printed results must be identical.

• OTP and Elixir unchanged. OTP’s stdlib, logger, file, ssl (TLS 1.2 and 1.3) and ssh run unmodified, and so do Elixir’s standard library, its compiler, OTP’s Erlang compiler and IEx, whose transcript matches BEAM’s. Every live OTP 28 opcode is implemented except on_load.
• erts’s Erlang where it is code. Of BEAM’s preloaded modules, those whose Erlang is useful load from erts like any other: erlang, erts_internal, persistent_term, atomics and counters, their NIF stubs answered by natives, and prim_eval, whose receive is BEAM assembly: it runs a receive that erl_eval evaluates, such as one typed at the shell’s prompt. The rest stand on BEAM’s C runtime (the boot process, ports, sockets, tracing) and never load; a call to one is undef unless a native answers.
• A fun’s identity is its module’s checksum. A fun read from the external format with the loaded module’s checksum is that code’s fun; one with any other checksum keeps the identity it came with, is written back byte for byte, and a call to it is badfun, as on BEAM. Three artifacts of BEAM’s fun table are not reproduced: the order of two funs that differ only in their checksum (here the loaded code’s sorts first, then by checksum); the OldIndex a second decode of one unknown checksum is written back with (BEAM keeps the first decode’s; here each fun keeps its own); and a fun decoded before its code is loaded, which stays apart from that code once it loads with the fun’s checksum (BEAM’s table then makes the two equal; here they compare unequal).
• Processes as on BEAM. Links, monitors, aliases, exit signals, registered names, timers and ETS, on one or more scheduler threads with per-process heaps and copying garbage collection.
• Regular expressions (beamlet-re) run in linear time for every pattern, so a hostile pattern cannot backtrack for ever. A pattern either means what it means in PCRE or fails to compile; backreferences and general lookaround do not compile.
• Crypto (beamlet-crypto) implements OTP’s crypto natives in pure Rust, so crypto.erl, public_key, ssl and ssh run on it. It takes randomness only from Platform::random, and a failure to get it fails the operation. Its rsa crate has a known timing side channel in decryption; side channels are a stated wall, not one the design closes.
• One X25519 and Ed25519 for the whole box. They are ed25519-compact’s, the crate the loader, keyd and sshd use, built from the same vendored bytes, and the RustCrypto primitives beamlet shares with the servers are built from them too (vendored dependencies). Where it departs from OpenSSL, and so from the real BEAM: an all-zero Ed25519 seed is refused as a bad key, where the crate would panic; and a signature is checked cofactored, as the loader’s and sshd’s are, so one whose R differs by a point of small order verifies. That forges nothing without the key, but signatures are malleable, and code that takes a signature as an identifier must not.
• Compression is OTP’s zlib, whose natives run on miniz_oxide: deflate and inflate in raw, zlib and gzip formats, so :zlib, :zip and compressed external terms work unchanged. Each stream bounds what it holds queued, so a hostile archive cannot make one call allocate without limit, and what it holds counts toward its holder’s heap limit (limits inside one VM).
• A compiled pattern is a compiled matcher. binary:compile_pattern/1 builds a matcher the VM keeps behind the reference it returns ({bm, Ref} for one pattern, {ac, Ref} for several, as on BEAM): the patterns indexed by their first byte, the longest first, so binary:match, matches and split compare a byte of the haystack only against the patterns that start with it, and a search costs the haystack’s length whatever the pattern count, the leftmost and then longest match still winning. An uncompiled pattern is compiled for the call, as on BEAM; the haystack is read in place; what a compiled pattern holds counts toward its holders’ heap limit. On the host, 88 patterns over 64 KiB cost what one does, 0.44 ns a byte, where comparing every pattern at every byte cost 11 ns with 8 patterns and 127 ns with 88 (the_search_is_flat_in_the_pattern_count).
• Not supported: NIFs and port drivers (foreign code runs as a separate program), distribution, hot code upgrade, and any OTP version but the pinned one.

The console on a host

On a host, beamlet’s command line puts the terminal in raw mode from the moment a process of the VM first reads the console, so that every byte typed reaches the VM as it is (Ctrl+C included), and restores it on every exit: a result, an exception, a halt, a signal, a panic included (userland/otp/cli/src/tty.rs). Output processing stays on, so a line written past the shell’s encoder still lands where a line does on a host; the encoder ends its lines with CR LF itself, as it must on Redoubt. console_size is the terminal’s size, read afresh at each call. A change of the terminal’s size is not delivered on a host (console_resized answers nothing there); the VM hands a platform’s change to the console’s reader, as on Redoubt, and the shell reads the size at each prompt (the shell). Console input goes to one Erlang process, the shell’s driver, which takes it with beamlet:console_subscribe/0; a second subscription is refused, so no code run at the prompt can take the keyboard, or the interrupt key with it, from the driver.

The console, the clock and randomness

The first part of beamlet’s platform on Redoubt, beamlet-redoubt (userland/otp/redoubt), serves the console, the clock and randomness over the client library and the runtime’s calls. /dev/cons is opened once, if the namespace holds it (a lease’s does not: below), and its reads and writes go through the VM’s hub (asynchronous underneath): one read is out at a time, held by the server until there is typing, and one write, a page at most, with what the VM wrote after it waiting in order behind it, since the console’s share is a page a badge. So neither a read that waits nor a slow console holds the VM’s thread, until 64 KiB wait unwritten, when a write waits in place as one to a slow terminal does. When the VM ends, what it wrote reaches the console before its last line, waiting at most 2 s. After the console’s end, idle still sleeps until its deadline. When the process reading the console exits, the VM says so (console_listening) and no read goes out until another process reads it: typing waits at the console, what came in before is held for the next reader, and idle does not return for it, so a VM with nobody at the console sleeps rather than spins. The console’s size is asked of its server at each call, and from the first read one thread keeps a resize out there and hands each change to the VM (console_resized), whose idle returns for it. There is no wall clock, so system_time_us is None. ./shell --fake runs the shell on it.

• The size is still the VM’s own call. It is a typed call, which no hub carries, so a console that stops answering a size query stops the VM.
• Randomness is the kernel’s. On the fake kernel it is seeded from the host for a person’s run, and fixed for a test’s, so a test repeats. On the machine it is the kernel’s own.

beamlet on Redoubt

Status: built · partly tested: programs are not built · tested: bench:session-tcp-echo, bench:session-tcp-loopback, host:beamlet-redoubt::a_connect_is_clone_ctl_and_data_and_the_peers_end_is_closed, host:beamlet-redoubt::active_mode_delivers_what_arrives_and_the_end, host:beamlet-redoubt::refusals_are_named_and_a_failed_socket_is_given_back, host:beamlet-redoubt::a_listener_accepts_through_ctl, host:beamlet-redoubt::a_read_the_server_times_out_is_asked_again, host:beamlet-redoubt::without_a_net_the_in_vm_loopback_still_serves, bench:beamlet-natives, bench:beamlet-reduction-rate, bench:beamlet-schedulers-mttcg, bench:beamlet-schedulers-one-hart, bench:shell-compile-module, bench:shell-compile, host:beamlet-vm::two_schedulers_lose_nothing_across_the_system_lock, host:beamlet-vm::schedulers_go_offline_and_online_again, host:beamlet-redoubt::two_schedulers_lose_nothing_across_the_system_lock, host:beamlet-redoubt::four_schedulers_lose_nothing_either, host:beamlet-redoubt::without_the_argument_the_vm_has_one_scheduler, host:beamlet-redoubt::the_count_is_one_decimal_argument_capped_and_one_otherwise, host:beamlet-redoubt::the_io_report_counts_the_schedulers, host:beamlet-redoubt::without_a_console_writes_are_dropped_reads_end_and_the_size_is_unknown, host:beamlet-redoubt::a_vm_without_a_console_runs_its_module_to_exit, bench:beamlet-boot, bench:beamlet-console, bench:beamlet-files, bench:boot-profile, bench:boot-profile-unverified, bench:pack-outside-module, bench:pack-bad-truncated, bench:pack-bad-wrong-length, bench:pack-bad-wrong-name, bench:beamlet-heap-flood, bench:beamlet-budget-flood, bench:userland-boot, bench:userland-bad-start, bench:lease-bad-start, bench:userland-read-only, bench:verity-flipped-tree, bench:verity-wrong-root, host:beamlet-redoubt::a_module_is_its_file_and_a_failed_read_is_refused, host:beamlet-redoubt::not_found_at_the_open_is_absent_and_every_other_error_is_refused_by_name, host:beamlet-redoubt::verified_module_lookup_propagates_found_absent_and_refused, host:beamlet-redoubt::verified_application_lookup_propagates_found_absent_and_refused, host:beamlet-redoubt::a_packed_module_comes_from_the_pack_and_any_other_from_the_volume, host:beamlet-redoubt::a_pack_with_a_bad_entry_is_refused_whole, host:testbench::the_boot_pack_is_deterministic_sorted_and_only_of_the_objects, host:beamlet-vm::the_search_is_flat_in_the_pattern_count, host:beamlet-vm::the_matcher_finds_what_every_pattern_at_every_byte_finds

On Redoubt, beamlet is a native program. Its built Platform adapter uses the client library (native programs) for the console, files, verified code lookup and the system natives, and the kernel’s calls for the clock and randomness. The program adapter remains planned.

TCP is Plan 9’s /net, served by ipd (ipd): gen_tcp works unchanged over a backend in Erlang, beamlet_tcp, embedded in the VM (vm/lib/beamlet_tcp.erl), and the Rust side only moves bytes. When the namespace holds a /net, a socket is a process that reads /net/tcp/clone for its number, writes connect(addr, port) to its ctl in the wire encoding (the generated codec, wire), reads ctl until the state is established, and moves bytes through data: each a file operation through the VM’s hub like any other (files), a reader process per socket for what arrives, writes from the socket process. Framing, packet modes and the active-mode mailbox stay in Erlang; the hub’s etimedout on a wait is asked again, and 0 bytes is the peer’s end (tcp_closed, or {error, closed} to a recv). There is no backpressure: the reader drains the socket as bytes arrive and a passive socket’s unread bytes accumulate in the socket process, bounded only by the VM’s budget. close is ctl close, and a socket process that dies has its socket aborted by its reader, so nothing of ipd’s outlives it.

• Every socket is ipd’s then, loopback addresses included. The loopback inside the VM is not consulted: a connect to 127.0.0.1 or to the box’s own address is ipd’s refusal (R59 (never the box’s own addresses)), and a refusal keeps the name ipd gave it, by the error table’s last column (wire): eacces for the box’s own addresses, for a labelled caller (R60 (a sink refuses labels)) and for an address or port outside the scope; a connect the stack could not complete (nobody answering, a reset) reads back as the closed state, econnrefused. session-tcp-loopback connects to the box’s own sshd at its loopback and its listed address and the capture shows no SYN to either; session-tcp-echo echoes through the bench’s peer in each packet mode.
• A session’s sockets are its share of ipd’s admission (sizing): a State unit each, as its /net connection is one, within its account’s bucket divided among the account’s sessions, and a socket closed by the session keeps its unit through TCP’s TIME-WAIT (10 s). Past the share a connect is {error, emfile}, ipd’s too_many. The image’s ipd gives alice’s account 32 sockets, 4 + 32 = 36 units, so her session with her console session beside it has 36 / 3 = 12, its connection and 11 sockets: eleven connections opened and closed in quick succession all succeed, and the twelfth while they linger is emfile (session-tcp-churn). A principal without an override has the defaults, 12 / 3 = 4 beside a console session, so 3 sockets, and a client that opens and closes connections in quick succession reuses one instead, as session-tcp-echo does.
• Addresses are IPv4 tuples. A name is {error, nxdomain} until the resolver (name-scoped connections); an IPv6 tuple is eafnosupport.
• listen is ipd’s listen rule (scopes): a scope without one, as a session’s in the image, answers {error, eacces}; one with it listens, at most 8 in the backlog, and accept is a read of the listener’s ctl that ipd holds until a connection arrives, the new socket’s remote its peer name.
• Without a /net (a lease, a VM on the fake kernel) a loopback network inside the one VM stands in: listen registers a port, connect to a loopback address finds it, and any other address is {error, enetunreach}, since the VM has no network (agents). One VM is one trust domain, so the stand-in adds no authority: it reaches only the VM’s own processes.
• Not inet. sockname of a socket of ipd’s is {error, enotsup}: ipd does not tell a socket its own address and port, and nothing is invented; a listener’s is {0,0,0,0} and its port.

The waits fit together because a server parks a request strictly shorter than its clients wait for it: ipd holds a data or ctl read 8 s against the hub’s 10 s (ipd, files), so a read the hub gives up has already been answered timeout, the client asks again, and bytes arriving between wait in the socket’s buffer for the next read; the host test a_read_the_server_times_out_is_asked_again runs the backend against a /net that keeps the rule and feeds the bytes after the timeout.

The size is asked afresh because the only console whose size changes is an SSH channel, and a cached size would answer a redraw with the size from before the change; a caller that wants to be told of a change uses the parked resize call (the shell).

On the machine, beamlet is the program beamlet, started like any other with a console, a budget, and a connection to the userland disk’s erofsd, a named handle its argument endpoint= names (erofsd:system). Its waiter threads are the runtime’s thread::spawn.

• schedulers=N gives the VM N scheduler threads, at most 8; absent or malformed, one. The count is the image’s choice: the steward gives a session’s VM two (the steward). The kernel does not tell a program how many harts there are, so the VM cannot follow the machine; a call that says so is a later package’s, if an image ever needs one. The first scheduler is beamlet’s own thread, and each other is a thread of the runtime’s thread::scope with an 18-page stack, the size of a session VM’s first: twice the largest first-thread peak the memory cases find, the idle shell’s 36,136 bytes (beamlet-footprint) over a compile’s 34,280 (beamlet-compile-footprint), now that the compressor term_to_binary(T, [compressed]) runs is a few hundred bytes and not a 65 KiB frame (what the VM holds at its prompt); a module compiled at the prompt over SSH runs (shell-compile-module). A helper thread’s stack sits above an unmapped guard page (the native runtime): an overflow there faults at once and the kernel ends the VM at the overflow, not later from a heap it wrote over (thread-stack-guard). The runtime paints every spawned stack, so the scan measures them: at the idle prompt the second scheduler peaks between 21,496 and 25,200 bytes on rv64 over three runs and at 21,752 on rv32, the platform’s three other threads under 1,600 (beamlet-footprint, stack beamlet thread N). A compile runs the compressor’s frames on whichever scheduler holds the process; beamlet-compile-footprint runs one scheduler so that is the painted first thread, whose 34,280 bytes above are the figure every scheduler’s 18 pages are sized from; no case compiles on a helper, so a helper’s peak under a compile is not measured. Several schedulers below.
• bind=PREFIX=HANDLE puts a named handle beamlet was handed at a prefix of its namespace (bind=/home/alice=walfsd:data): the bind/2 a session performs for itself (namespaces), for a VM init launches alone, whose namespace holds only /dev/cons. In the steward’s sessions the session’s namespace does this. It creates no authority: the handle was handed already, and a bind naming a handle it was not handed, or a prefix that is not a clean absolute path, is refused before any server is asked anything, with one line on its console.
• report_io has the platform say, when the VM ends, how many requests went through the hub and how many threads it ran (beamlet: io: N requests through the hub; threads: S schedulers, W waiters, 1 scheduler for one); bench:beamlet-files reads it.

In a boot-stats build (checked builds) beamlet says beamlet: first console read [t=N] at the VM’s first console read, with time_now in µs, and beamlet: first prompt drawn [t=N] when the shell’s driver has drawn its first prompt (beamlet:prompt_drawn/0): the driver takes the console before it draws its banner and prompt, so the second line is the boot’s time to its prompt, where a user sees the box ready, and the span the target is on; both lines are drawn beside the prompt. boot-profile and boot-profile-unverified measure it. Measured in that build under icount (shift=3, sleep on) with seed 1, in guest time (bench:boot-profile, bench:boot-profile-unverified):

On littlefs 99 % of the boot was in the VM’s 96 lookups: littlefsd found each file’s name in the volume’s root directory again two or three times for every 9P operation, 77,710 block reads of 673 distinct blocks (littlefsd). On EROFS the same 96 loads make the same 652 9P operations, and erofsd reads the volume 641 times (112 inodes, 243 directory blocks, 286 runs of a file’s blocks) in 642 calls to blkd of 3.35 MB in whole sectors; the loads take 7.1 s verified and 3.7 s unverified, and the VM’s own work and its console about 6.5 s either way. With the boot pack (below) the 96 lookups become one read of the pack, 115 9P reads of 16 KiB: the boot makes 208 9P operations, and erofsd reads the volume 208 times (16 inodes, 35 directory blocks, 157 runs of a file’s blocks) in 209 calls to blkd of 2.46 MB. (That build counted each call erofsd made to blkd, and the whole sectors it carried; erofsd’s boot-stats now counts each read it asks of its range, whatever calls the range client splits it into, and the bytes it asks for.) Reading the pack takes 3.6 s verified and 1.2 s unverified, and the VM’s work after it, decoding the modules as they are called, 8.0 s. On littlefs the prompt with the boot pack is at 152 s verified, since littlefsd finds the file again for each read.

Since the steward starts the shell as the console principal’s session (the steward), the verified first prompt on rv64 comes at 10.3 s: init has started its servers and pushed beamlet’s 4 MB public entry to bootfsd by 1.6 s, when it starts the steward; from 1.6 to 4.4 s the steward carves the session’s budget, streams its image from bootfsd into the new process in the client library’s launch batches of 64 pages (native programs), and the VM reads its boot pack; from 4.4 to 8.4 s the shell’s application starts, until its driver takes the console; from 8.4 to 10.3 s the shell starts under the driver, to its banner and first prompt. Unverified the same points are at 1.5, 2.9, 6.7 and 8.5 s. On rv32 the steward starts at 1.9 s, the boot pack is read by 5.1 s, the driver reads at 9.6 s and the prompt comes at 11.8 s; unverified at 1.9, 3.5, 8.0 and 9.9 s. The push and the stream are calls of the same cost whatever they carry, so init and the steward lend 16 pages a call: the add carries 32 KiB, a power of two, so bootfsd’s entry doubles onto 4 MiB and its heap’s peak stays half its cap. The shell’s commands are loaded when first called, so its start loads few modules. The kernel’s audits and fault reports hold no other hart, and a slice’s end pays for its own work, not for checked calls, whole budgets and three timer arms; each takes about 2 s off the prompt on both widths. The row is seed 1, whose runs differ by under 0.03 s; in a whole run of the bench the unverified prompts came within 0.03 s of it. Across seeds 1 to 4 the prompt moves by under 0.08 s; seed 5 is the slowest, at 10.9 and 8.8 s on rv64 and 12.3 and 10.2 s on rv32.

The boot-time target: in this build, the prompt within 15 s of guest time, verified and unverified, on both widths: the slowest measured prompt plus a tenth, rounded up to 5 s, here seed 5’s verified prompt on rv32, 12.3 s, so 13.5 s, rounded up to 15 s. boot-profile and boot-profile-unverified fail past it, and run in every whole run of the bench.

How fast compiled code runs. In the console session of the image’s boot, under icount (shift=3, so a microsecond is 125 guest instructions) with seed 1, Enum.reduce(1..65536, 0, &max/2), compiled Elixir calling a native through a fun on every step, takes 98,723 reductions; its fastest of 30 runs a second apart, a run the shell’s driver did not share the hart with, gives the rate (bench:beamlet-reduction-rate):

A native call no longer fills all 255 argument slots it could pass (765 instructions on rv64 for a native of two arguments), a fun’s arguments stay in the x registers instead of going through a vector, and the interpreter’s step is inlined into its loop, so the registers it saves are saved once a time slice rather than once an instruction. On the host, beamlet runs a tail-recursive loop and lists:foldl/3 at 70-200 ns a step. Of what is left on rv64, about four fifths is the interpreter’s own work, some 200 guest instructions a BEAM instruction, and the rest the kernel’s, most of it the slice’s end: about 8,800 kernel instructions on rv64 and 12,900 on rv32 each time the VM’s slice ends, against 15,000 and 20,700 before the kernel stopped re-arming the timer three times a preemption and decoding a whole budget to move one word (scheduling). No yield and no count changed: a process still yields after its 2,000 reductions or 200,000 instructions, and the kernel’s time slice still preempts the VM’s thread whatever it runs. The floor: 42,000 reductions a second, the slower width’s rate less a tenth, rounded down to a thousand; beamlet-reduction-rate fails below it, and the rate before the interpreter’s changes is below it on both widths. A session where a line has just been typed runs slower than this for a while: the shell’s driver measures the line being edited again for each piece of input it echoes, which can take as much of the hart as the line’s own code for seconds after a long line; the floor’s run is the fastest of its 30 for that reason.

Several schedulers. The VM’s schedulers share one run queue, under the lock on the VM’s shared state, and take its processes in turn (vm.rs). A process runs on one scheduler at a time, its heap, stack and collections its own, and the lock is held between time slices and by the instructions and natives that touch what is shared: the atom table, the loaded modules and loading them, ETS, persistent_term, registered names, timers, and the mailbox of a process another scheduler runs, which takes its messages when its slice ends. Loaded code and the literal chunks are read without it: each scheduler keeps the calls it has resolved, checked against counters bumped when code changes. The platform, and the hub in it, is locked after it, never before, and only a scheduler with nothing running anywhere waits in the platform’s idle. A process still yields after its 2,000 reductions or 200,000 instructions, on whichever scheduler runs it, and keeps that scheduler while nothing else waits: another is woken only for more work than the schedulers ending a slice take back themselves. The kernel’s time slice preempts each scheduler’s thread. On Redoubt the lock is the runtime’s mutex and the wait for work its condvar, which sleep on endpoints (native programs). Their endpoints are made when the VM starts, five pages with two schedulers: the system’s lock, the platform’s, the wait for work’s own, and one for each scheduler that waits; so schedulers contending allocate nothing, and a VM at its budget’s limit does not end at its first contended lock. If the kernel refuses one, the VM runs on one scheduler. A resource’s lock (an atomics array, a zlib stream, ETS’s count of what its objects hold) makes its endpoint, a page, the first time two schedulers contend for it, and gives it back with the resource; one the kernel refuses leaves that lock’s waiters polling.

The schedulers are the session process’s threads, in its one budget: the kernel charges each hart’s runner to that budget’s one pass (scheduling), so the session’s share among budgets is its weight’s as before, and on a machine whose other harts are idle it runs on as many as it has schedulers busy. Nothing new is carved, and the heap, ETS and persistent_term limits are a sixteenth of the budget each as before; the second scheduler’s stack and thread page and the locks’ endpoints are 25 of the share’s pages (above).

In the console session of the image’s boot, two Task.async loops of Enum.reduce(1..65536, 0, &max/2), awaited, timed 10 times with one scheduler online and 10 with two, alternated; the fastest of each (bench:beamlet-schedulers-mttcg, and bench:beamlet-schedulers-one-hart at 1 hart):

Across runs alone the 2-hart ratio measured 1.77 to 2.14 on rv64 (7 runs) and 1.83 to 1.94 on rv32 (5). The 2-hart times are host times: the case runs without icount, in multi-threaded TCG, where the harts are host threads that run at once. Under icount QEMU runs the harts in turn on one host thread and guest time counts every hart’s instructions, so a second hart cannot make a run faster in guest time (which cases run in guest time). Both times come from one boot, so the ratio holds beside a steady load but not beside a stalled vCPU thread, and is a verdict only alone. At 1 hart nothing runs at once, and the cost of two schedulers is the instructions their switching takes: the kernel’s slice ends alternate the two threads where one scheduler keeps the hart. So the one-hart case runs under icount (shift=3, seed 1), where the ratio is a count of instructions, the same on any host however loaded: seven boots, two beside a twelve-core load, read 0.94 on rv32 and 0.95 or 0.96 on rv64; across boots the one-scheduler side’s fastest runs lie within 100 µs of each other and the two-scheduler side’s in two clusters 2.3 ms (rv64) and 2.8 ms (rv32) apart, within 0.1 % of a 3.6 or 4.0 s run, and a boot’s ten pairs lie within 1.7 % of each other. On the host’s clock the same case read 0.81 to 0.95 across six boots alone on rv32 (0.96 twice on rv64): a boot’s ten host times of either side spread by a tenth, and the fastest of each side met by chance. Sampled beside those runs, the vCPU thread moved among the four cores of q‘s quiet set 13 to 28 times a run, and those cores’ clocks read anywhere from 0.8 to 4.7 GHz; pinned to one host core the case read 0.93 on rv32 and 0.96 on rv64, and a side’s ten times still spread by 3 to 9 %. The floor: a ratio of 1.40 at 2 harts, a fifth under the lowest measured alone, since the train’s bench runs the case beside other work, where it once measured 1.53; a VM whose schedulers ran one at a time would measure about 1.0. And 0.90 at 1 hart: two schedulers sharing one hart keep nine tenths of one’s throughput, four points under the slower width’s 0.94 in guest time, where a boot’s pairs vary by less than two. The case does not measure what the kernel charges for the two threads: by the kernel’s rule each hart’s run is charged to the session’s one budget (scheduling), which the scheduling cases judge, not this one. With two schedulers at 1 hart, beamlet-reduction-rate measures 52,936 (rv64) and 46,959 (rv32) reductions a second, the table’s rates within a fifth of a percent. The rate is judged at the 1 hart its floor was set at, under --smp N too: under icount a second hart’s instructions count in guest time, so at 2 harts main measured 45,354 (rv64) and 37,157 (rv32), under the floor on rv32, as with two schedulers (45,778 and 36,967). Before a yielding process kept its scheduler it changed threads every slice, a kernel wake-up each, and measured 43,511 on rv64.

Residual: loading a module holds the lock, so while one scheduler loads, no other starts or ends a time slice. To the shell’s first prompt, 116 loads hold it 8.2 s of guest time on rv64 (8.9 s on rv32), the longest unicode_util’s, 877 ms (969 ms). A typed call the platform makes in place (the console’s size, littlefsd’s rename) holds the platform’s lock. The runtime’s heap is one spin lock (native programs): two schedulers allocating at once take turns there, and one waiting for a holder the kernel preempted spins. A native is still not preempted, but holds only its own scheduler.

Before its VM starts, beamlet reads the volume’s boot pack, boot.pack, whole: one open, one stat for its length, one allocation of that length charged to the VM, and reads of 16 KiB in order, the lend a module’s file is read through (pack.rs). The image’s builder makes the boot pack from the modules and resources the shell loads to its prompt, which image/userland.toml names, their bytes copied from the files beside it: a magic and a version word, the entry count, an index of each entry’s name, offset and length sorted by name, then the entries’ bytes back to back. The same files and names give the same pack, byte for byte. beamlet checks the boot pack whole before using any of it: the magic and the version, each name a module’s or a resource’s file name in strictly ascending order, the entries’ bytes back to back from the index’s end to the file’s, and each module’s first atom, its own name, the one its entry names. A lookup takes a name the boot pack holds from it, a copy of its bytes that the loader decodes and drops as it decodes a file’s, and asks the volume for any other name, as above, so the order is the boot pack, then the volume’s file, then the code path. Once every entry has been taken, at the prompt, the boot pack’s allocation goes; a module it holds that is never called keeps it, at most the pack’s size. A boot pack the volume does not hold is said once on the console and the lookups go to the files; one that cannot be read, or does not check, is said once, naming the boot pack and why, and beamlet ends as below. The boot pack is per VM and read-only: no sharing of its pages between VMs, no byte cache in the steward, no decoded snapshot, no pack per profile. It is a file of the verified volume (R75 (verified userland)), never written at run time.

If the volume does not attach, because erofsd serves it as corrupt, the boot pack cannot be read or does not check, or the module it is told to start cannot load, what it does is decided by its console. With one (a session’s VM, or one init starts), it says why there and waits without exiting: a tampered disk must not become a restart loop that reboots the machine (bench:userland-bad-start, bench:pack-bad-truncated). Without one, a lease’s VM (agents), waiting would be silent, a budget held to its deadline with nothing said: it exits, and its exit code, the one thing the steward’s lease … exited line carries (the steward), is the diagnostic, 1 in a default build and the end’s own code in the test-only exit-kinds build (bench:lease-bad-start), a halt’s the status the process gave (an agent whose input could not be read, bench:lease-writes). That exit is an exit, not a fault: crash blame is init’s, for a server it restarted whose exit notice names a faulting thread’s current call (init), and a lease’s VM is the steward’s launch, so its exits count toward no lockout (R40 (blame by label set)): a sponsor’s retry of a lease that cannot start costs its own budget and nothing else (host:redoubt-steward::a_leases_exit_is_its_end_and_no_blame). A read refused by the volume’s admission for the moment is tried again before it counts as refused (the client library): a VM loading beside its account’s other VMs rides out the moment.

Each label set that runs beamlet reads its own attachment of the userland image through its own blkd, verityd and erofsd, all carrying that set (R34 (confined placement)).

The timer’s counter frequency is not needed: time_now’s microseconds serve the clock and idle’s deadlines (bench:beamlet-console).

Natives

A native is one of three kinds, and no other:

• OTP’s own, reimplemented in Rust where BEAM has C (crypto, re, zlib, the file and buffer primitives): the API is OTP’s, so the differential test holds each to the real BEAM (what runs on it).
• The system’s: what has no POSIX equivalent, the small fixed set below.
• A primitive the interpreter is too slow for, and no more than the primitive: the screen buffer (screen natives).

Anything else is Elixir. Every native’s own code holds no unsafe; it bounds the work one call does, since a native is not preempted by reductions; it reaches no panic from any argument; and it is attacked with hostile arguments, fuzzed where it parses. OTP’s own natives parse what OTP’s do (certificates, compressed streams, patterns), held to BEAM by the differential test; a native of the other two kinds never parses a large untrusted format, which goes to a native program in a budget of its own instead (native programs).

What has no POSIX equivalent reaches Elixir through a small fixed set of beamlet natives, and every server binding is pure Elixir over them:

They are the module redoubt’s, over the platform’s System (userland/otp/vm/src/platform.rs; the argument checks in bif/system.rs, the calls in userland/otp/redoubt/src/system.rs). A platform with no System, the host CLI’s, answers each {error, not_supported}; the host’s fake-redoubt runs Redoubt’s, on the fake kernel. Each native, as Erlang sees it:

• ns_lookup(Path) is {ok, Handle, Rest}: the connection the longest matching prefix of a clean absolute path names and the rest below it, or, for a name with no /, the named handle the VM was started with (keyd, budget, a session’s notices) and <<>>. A path that is not clean (.., //) is bad_name, one nothing is bound at not_found. bind(Prefix, Connection) puts a connection at a clean absolute prefix of the files’ own namespace: a path under an existing prefix shadows it for longer matches only. A handle that is not a 9P connection is attached once, then and there, each of the attach’s two calls waiting at most ATTACH_US on the VM’s thread; one whose server does not answer or does not speak 9P, or a budget, is not_a_connection. A bind at a prefix already bound replaces it; the table holds at most MAX_BINDINGS (64) entries, those the VM started with included, and a new prefix past them is system_limit, so a loop of binds fails visibly and never on the VM’s heap. ns() is [{Path, Name | nil, Handle}], the namespace’s entries in binding order, then the named handles.
• copy_file(From, To) is {ok, Bytes}: the file server copies the file From to the new file To itself, by its typed copy_file, so no byte crosses into the VM. Both paths resolve as File’s do, against the working directory; on two connections it is {error, exdev}, and the caller copies through the VM; onto a name that is there, eexist. It is a typed call, which no hub carries, made on the VM’s thread as a rename is, so the VM waits while the server copies. A platform whose servers do not copy, the host CLI’s, answers enotsup.
• call(Handle, {Words, Buffer, Handles}, TimeoutMs) is {ok, Ref}, and the reply arrives as {reply, Ref, {ok, {Words, Buffer, Handles}} | {error, Name}}. A buffer is a binary for a message that lends one and nil for an inline one; the reply’s buffer is the bytes its word 1 says at the front of the lend. No hub carries a typed call, and no scheduler waits for one: a pool of CALL_THREADS threads makes them, MAX_QUEUED more wait, and past that a call is busy; each is bounded by its timeout, at most MAX_CALL_MS, a ceiling on what a caller may ask and not any call’s default: the generated clients pass 5 s for a short call and the ceiling itself for the one call measured past it, a lease’s launch (wire), and a timeout above the ceiling is badarg before any call is made. A VM that asks for the whole of it holds one of its own call threads that long, a cost inside its own trust domain. The caller waits in receive, as for any message. send(Handle, {Words, Buffer, Handles}) is one way, waiting at most SEND_TIMEOUT_US for its receiver. The natives encode nothing of a protocol: the generated clients do (wire).
• serve(Endpoint) serves a receive right the VM holds, on a thread of its own, which keeps the endpoint open and serves it for the VM’s life (at most MAX_SERVED endpoints; one served already, by the same handle, is already_served; a second handle to the same receive right is not told apart): each call is admitted by the serving library’s admission, per (account, label set) with a share per badge (R26 (admission fairness)), and parked with the library’s deadline, REQUEST_WAIT_US (R28 (parked-call accounting)); it reaches the serving process as {request, Request, Badge, Account, Labels, {Words, Buffer, Handles}}, Request nil for a one-way send; each request or send the thread takes goes into the VM’s forwards queue and the thread is back at its receive at once, the VM taking the queue at its next poll or woken if it idles (the forwards queue). reply(Request, {Words, Buffer, Handles}) answers it once. A request the Erlang side never answers is ended by that thread at its deadline and its admission given back: the VM holds nothing for it. These are calls, not 9P, so the multiplexed connections’ rule (R77 (multiplexed requests)) is not this one’s.
• budget_create(#{pages, processes, weight, labels, account, deadline}) carves a child from the VM’s own budget, the named handle budget (no_budget without one); labels, account and deadline may be left out, and a deadline, in the clock’s microseconds, makes it a lease (deadlines). Labels left out are the VM’s own, the set labels() reads: a session is user-class, and the kernel gives a user-class caller’s child exactly its parent’s labels, so in a labelled session that default is what lets it carve, and launch, at all. Labels given go to the kernel as they are: fewer than the session’s are label_denied, more class_denied (labels on budgets). budget_destroy(Budget) destroys it and everything in it; budget_usage(Budget) is {ok, #{pages => {Limit, Used}, processes => {Limit, Used}, weight => {Limit, Carved}}}. A handle of another kind is wrong_object before the kernel is asked.
• labels() is the VM’s label set, read at its start off the first call thread’s first wake-up, which the kernel stamps with the sender’s labels as it stamps every message: the kernel’s word, not a launcher’s, and fixed.
• identity() is {ok, #{principal => Name, labels => [{Name, Id}], context => Name | nil, home => Path | nil}}: what the steward told a session’s VM of itself in its arguments (principal=, label=NAME:ID, context=, home=/PATH; sessions), read once at the start and fixed. It is the launcher’s word, information and not authority: labels() is the kernel’s, and what the session can reach is its handles. A VM started with no principal=, as one under init, gets {error, not_found}.
• launch(#{image, budget, namespace, handles, args, stack_pages, heap_pages, serve}) is {ok, Job}, and the job’s end arrives as {exit, Job, Cause, Code} (exited, faulted or killed): the client library’s launch with the image the caller read, a budget it carved, its namespace entries and named handles (at most MAX_START_HANDLES together, serve’s counted), and arguments; the platform adds the loader stub, which it carries as init does, the job’s own exit endpoint (native programs), and the budget’s pages, which it recorded when it carved the budget, into the child’s startup block (init): one source, nothing a caller types. A receive right the VM serves among them goes to the program as a send right with a badge of the launch’s own, in a budget the VM carved (below). With serve => Name the program is a server: the platform makes an endpoint, gives the child its receive right as the named handle Name, keeps none of it once the child has started, and answers {ok, Job, Connection}, the caller’s send right with the server’s own badge (below every minted one), which bind/2 attaches like any other; this is how a session starts the server of its pipes (native programs). At most MAX_JOBS run at once, each watched by a thread, started the first time one is needed, that waits for its exit notice; a launch past them is too_many, before anything is made; a job no thread takes to watch is killed, its budget destroyed, since nothing would hear its end.

Every refusal is a Redoubt name: a term of the wrong type is badarg, as for any native, and a well-formed request the platform refuses is {error, Name}, the kernel’s error by its name in the ABI (wrong_object, label_denied), a 9P error by the one table’s (wire), or the platform’s own (not_found, not_a_connection, no_budget, busy). What waits on the VM’s thread is bounded and named: a send/2, at most SEND_TIMEOUT_US; a bind of a handle not yet a connection, its attach’s two calls at most ATTACH_US each; a hand-off to a thread of the platform’s, at most HAND_US; and the kernel’s own calls for budgets and launching, which answer at once. Each native bounds its work: no list is walked past its cap, and no binary is copied past its own:

Handles are resource terms: unforgeable, collected, and never serialisable. A copy of a handle inside the VM is the same connection (one badge, one client), so passing one to another Erlang process is sharing it. A handle no process holds is closed at the next collection; a budget’s is closed and the budget lives on, since destruction is a call. A handle dies with the object it names or the budget that stamped it (objects), and the kernel gives its index to the next handle the VM gets, so the VM marks a handle gone when it destroys the budget it came through, and a collection closes nothing at a gone handle’s index. That covers the budget’s own handle, and every handle the server launched with serve in that budget minted and returned to calls on it, since its mints are stamped with that budget (a pipeline’s stage connections, which piped mints). The VM cannot tell a handle such a server minted from one it forwards, so a forwarded handle is marked gone too and is never closed: it stays in the VM’s table until the VM ends, the price of never closing a reused index. The VM carves only from its own budget, so no budget it carved is below another. What is the VM’s own goes the other way: the send right a launch with serve gives it is to an endpoint the VM made, charged to the VM’s budget, so the server’s death leaves the slot to the VM, and it is closed when the right’s last term goes. A handle the VM started with is known to the namespace for the VM’s life; one a bind made known lives with the budget it came through (the budget a launch with serve ran in, or the one the connection a call brought it on came through), so destroying that budget forgets it, unbinds its prefixes and ends its hub connection, and a session’s piped, bound at every pipeline, leaves no entry and no handle behind (each once stayed for the session’s life). The hub’s session on the forgotten connection is ended as the budget is destroyed, so a request on it after (a stale file’s clunk at its term’s collection) is refused rather than sent: once the handle’s last term is collected its index may be the next launch’s send right, and a request sent on it then would reach that server, inside the session’s own table. Tested: a_destroyed_budgets_index_is_not_closed_when_its_term_is_collected, what_a_served_server_returned_goes_with_its_budget_and_is_not_closed_again, a_served_servers_binding_and_handle_go_with_its_budget and a_forgotten_connections_requests_reach_no_later_holder_of_its_index on the host, against a fake kernel that frees those indices as the kernel does, and pipe-eight and pipe-sixty-four on the machine. A request’s handles come with no budget: the kernel tells a receiver the sender’s badge, account and labels, never its budget, and a carved budget carries the session’s account and labels. So a receive right the VM serves, given in a launch‘s handles or namespace to a program in a budget the VM carved, goes to the program as a send right with a badge of that launch’s own (LAUNCH_BADGE on, the next for each launch on that endpoint), minted from the receive right and stamped with the budget, never the receive right itself. What the program sends through it is stamped with its budget at best, as its own mints are, so the handles a request with that badge brings die with the budget and go under its Life: a program that sends a handle it minted and is then destroyed leaves a term whose collection closes nothing at the index the kernel has given on, be it a sibling job’s budget or a connection of the session’s. A handle such a program forwarded rather than minted is kept, never closed, as a server’s is. The badge dies with the budget, so no request carries it after; one on its way to the VM as the budget went is known by its badge’s range, and its handles kept. A badge in that range the VM did not mint (a holder of the receive right outside the VM mints what it likes) is taken for such a launch’s, so what it brings is kept: a leak within the session’s table, bounded as a forwarded handle’s is, each such handle staying in the table until the VM ends, and never a close. Tested: what_a_launched_program_sends_to_a_served_endpoint_goes_with_its_budget_and_is_not_closed_again on the host and pipe-stage-handle on the machine, where a stage given the session’s served endpoint sends it the receive right of an endpoint it made, its budget is destroyed, and the budget carved next is still held once the term is collected. The residual: a handle revoked outside the VM, such as a grant whose stamp the steward destroys while the VM lives, is not known gone, and its index may be reused before its term is collected; collecting it then closes the handle that took the index, inside the session’s own table. A generation in the handle word is to guard that, in work of its own. A handle cannot reach another VM in a message: the term format writes a resource as a plain reference, with no state behind it, so a decoded copy grants nothing, and a handle crosses between processes only in a kernel call that names it. Delegation is always new_connection, a typed call on a connection, not a native (sessions). Tested: a_decoded_handle_grants_nothing, a_dropped_handle_is_closed_when_its_holder_is_collected, arguments_of_the_wrong_type_are_badarg_and_reach_no_platform_call, lists_past_their_caps_are_refused and a_handle_of_the_wrong_kind_is_refused_by_name on the host, and beamlet-natives-attack on the machine, where a VM offers every native a handle another VM wrote out, which it reads back as a plain reference, and asks the kernel for a child budget that adds a label (class_denied: a user-class caller may not); a VM in a labelled session runs nothing there, since the console carries no labels and its write-open of /dev/cons is refused (consoled). Over SSH a vault session’s channel carries its labels, and there steward-vault-launch carves with labels left out, launches a program the vault’s channel shows, and has fewer and more labels refused, in the steward’s own session (sshd). The cases’ tester carves each session as the steward does (steward): the principal’s account, a sub-budget per label set, the session from it; what it hands beyond a session’s own is said in each case.

Residuals:

• A request the Erlang side never answers is ended at its deadline with malformed (status 1): no status every protocol shares means a timeout, so its caller cannot tell the two apart.
• A typed call whose caller exits is not cancelled: it holds its pool thread until its reply or its timeout, and the reply, with any handles it brought, is dropped and closed.
• A forgotten connection’s waiter calls on the server’s handle by its index until the server’s death fails the call. Between two of its calls, the handle’s last term could be collected and the index given to the next launch’s send right, so that one completion call reaches that server, inside the session’s own table. Holding the handle until the waiter has returned would close that window; it stays, stated.
• The six waiter places are a hard bound: a seventh live server a session uses at once is refused emfile after FILE_WAIT_US (10 s), and a session that needs seven servers at once has no way but to let one go first. The wait is the operation’s, parked off the VM’s thread: an ended server whose waiter has not handed its end over yet holds its place only until it does, and the fresh request is served then (once, with a fixed second on the VM’s thread, one pipeline in sixty-four was refused on rv32 beside ten bench jobs).

The namespace and launching natives are the client library’s ns and launch, so policy stays in Elixir and encoding in Rust. The file server’s operations that name fids (Redoubt.File’s rename, copy_file and attributes) go through the library’s littlefsd beside the file natives, since the fids are the platform’s, not Elixir’s (native programs).

The Elixir modules over them are of two layers. A server’s typed calls are generated from its wire table, one function per message (wire), so the binding cannot drift from the server. Above the natives and the generated calls, a thin hand-written module gives what is idiomatic and adds no authority: Redoubt.Namespace, Redoubt.Budget and Redoubt.Process over the natives, Redoubt.Keys over keyd’s calls (userland/shell/lib/redoubt).

The Elixir modules live on the userland disk, one file per module, on a volume whose root the signed manifest pins (R75); only what the VM needs before it can read the disk is embedded in it: its own console server, code and kernel modules (beamlet_io, beamlet_code, beamlet_kernel, beamlet_port, beamlet_tcp) and its stand-ins for application, gen_tcp and ram_file (userland/otp/vm/src/vm.rs, EMBEDDED). The application stand-in runs no kernel application; when the first application with a callback module starts, it starts the kernel’s servers that other code calls, as BEAM starts them at boot: erl_signal_server, global_name_server, and kernel_safe_sup, which OTP’s group waits for before it serves a line. With no C library to ask a character’s width, prim_tty:wcwidth/1 answers {error, enotsup} and OTP measures with its own table, which agrees with BEAM’s libc on ASCII and wide East Asian characters and parts from it on a combining mark: libc gives it no column, the table one.

Screen natives

The screen buffer is beamlet-screen (userland/otp/screen), a crate of the VM’s with no unsafe and no dependency but the VM and the cells crate. Its natives, in the Erlang module redoubt_screen, are primitives over cells; widgets, layout and focus are Elixir (the shell).

A Style is {Fg, Bg, Modifiers}, each colour reset, {indexed, I} or {rgb, R, G, B} and the modifiers the cell protocol’s bits; a Rect is {X, Y, W, H}.

• A control character is refused, not drawn. put and fill raise badarg on one (the ASCII and 8-bit controls, DEL, the bidirectional embedding, override and isolate controls and the invisible format characters, a zero-width joiner inside an emoji sequence apart: the cells crate’s own rule, userland/native/cells/forbidden.txt) and write nothing of that call, so a caller that forgot to make text visible fails loudly, and nothing reaches the encoder that the cell protocol would not carry.
• Bounded. A buffer is at most 1024 cells on a side and 65,536 cells in all (larger is system_limit); put reads no more of its graphemes than the row has cells; no native does more than one pass over a buffer; frame reads a frame in place, allocating only its record of the positions given, a bit a cell of the buffer, made afresh for each frame, and charges by the bytes it read. So every call has a ceiling on its time and its allocation, and it is charged in reductions by the cells it touched.
• Counted. A process holds at most four buffers (a fifth is system_limit; one nothing holds any more is given back), counted in its process dictionary, and each buffer declares the bytes of its two grids and of frame’s record of positions, which count as its holder’s own memory, toward its own heap limit (limits), from new and from resize. So a loop of new/2 is ended by the limit that ends any runaway allocation. Code that erases the dictionary’s count starts a new one; the heap limit still holds.
• One writer. A buffer answers only the Erlang process that made it; any other gets badarg, but for draw, which writes no cell. It sits behind a lock because a resource may move between schedulers, and because the shell’s driver draws it: a screen registers its buffer with the driver when it opens, and asks the driver to draw it between whole frames. Holding a buffer’s term gives nothing but that draw, whose bytes go nowhere but to its caller; the driver writes only the bytes it drew itself from the buffer the screen in front registered.
• One segmentation and one width table, OTP’s. Text arrives split into graphemes by OTP’s own segmentation (String.graphemes/1), and a grapheme’s columns come from a table generated from OTP’s unicode_util:is_wide/1 (tools/gen-width.escript, which fails the build’s check when the table is not current): wide if a presentation selector follows its first code point or any of its code points is wide. The shell measures with the same is_wide/1, so what is measured is what the buffer lays out, at the pinned OTP’s Unicode version. The buffer does not segment text itself, so fill takes a single code point, and a caller that hands put two characters as one grapheme draws them in one cell, misplacing what follows, and nothing more; draw then places the next cell by the glyph’s width as columns measures it, where the shell’s Redoubt.Term.Frame.draw/1 sums its graphemes’, the one case where the two draw otherwise. frame puts a symbol of one code point, always one grapheme, itself, and hands back any other unput for the caller to segment.
• The diff speaks the cell protocol, and draw is the shell’s encoder for screens in Rust, over the buffer’s own cells: the bytes Redoubt.Term.Frame.draw/1 draws for the diff’s frame, byte for byte, which the shell’s suite holds it to over every code point below U+3000 that a symbol may be, the planes’ edges, hostile text made visible, every attribute and colour form, and frames after partial changes; and this crate’s tests hold it to draw.json, frames that Frame.draw/1 draws, written and kept current by the shell’s suite, with symbols holding control characters forced in, which draw makes visible again as Redoubt.Term.Text does (the shell). It writes only the style reset, CSI 2J, CSI row;col H and SGR from a cell’s numeric colours and attributes. It holds the buffer’s lock across the diff and the advance of what is shown, so a draw beside its owner’s writes on another scheduler sees each call whole and loses no change, and it is charged in reductions by the cells it examined. A new buffer, and one resized, is drawn whole at its first draw, as at its first diff.

Asynchronous underneath, synchronous on top

The VM is one trust domain on a few scheduler threads, so a blocking call would stop every Erlang process on that thread. So the VM’s I/O is asynchronous underneath: a request is submitted and the VM keeps running, its answer comes back later to whoever asked, and Platform::idle returns on a timer deadline or an answer. The kernel has no queued sends (a message occupies its sender’s thread until taken: IPC), so on Redoubt the asynchrony is the client library’s hub (native programs) over the servers’ multiplexed connections (the serving library), in beamlet-redoubt (userland/otp/redoubt/src/io.rs):

• One hub per VM, owned by the platform and run by whichever scheduler holds it. A request is sent from the scheduler’s own thread, waiting at most 1 ms for its server to take it, and its buffer is the hub’s until its answer.
• One waiter thread per connection, started the first time the VM uses the connection, blocked in its completion call; it hands each batch of answers to the VM’s own endpoint with one send.
• idle is a receive on that endpoint until the next timer deadline, and the platform hands each answer to whoever asked: the console’s to its reader, a file’s to the Erlang process whose call it was. An answer, request, input or the input’s end that arrived before the VM idled returns the idle at once: the receive is entered only with nothing to hand over, so nothing taken waits for the next wake-up to be looked at.

Above that, Elixir is ordinary synchronous code: File.read/1 blocks the calling Erlang process, not the scheduler. OTP’s prim_file runs unchanged: a file native whose operation the platform has begun answers it later (Files names the asking process), the process waits, and only that operation’s end (or an exit signal) wakes it, not a message; the native is then called again and takes the result. Several requests of one operation (a walk, an open, the reads up to a count) go out as each answer comes. A process that dies while it waits has its operation dropped with what it holds, and an operation that opened a file closes it. The VM’s own code loading reads files waiting in place. Concurrency is bounded by each server’s shares of requests and pages per connection (R77 (multiplexed requests)); over them, a request is answered busy and asked again RETRY_US (10 ms) later, a console write and a file operation’s request alike; the answers are counted, and said with the I/O report.

Threads. A process has at most 255 threads (processes). The VM’s are its schedulers (schedulers=N, two in a session) and its waiters, one per connection it uses, at most 6: a session’s bindings (bootfsd, the home volume, a labelled volume, ipd, the console, the system volume), a bind being one of them again. A server that ends takes its session with it: its waiter hands the end over and returns, closing the handle it woke the VM through, and the connection gives its place up when the next one needs it. A session’s piped is a new server at every pipeline, so a session runs any number of pipelines, each taking one place while it runs (servers_bound_and_ended_one_after_another_never_run_out_of_waiters; pipe-eight, eight in one session). An ended server’s waiter hands its end over only once its completion call has failed with the server’s death and the VM has received the hand-over. The VM waits for it when it forgets the connection, as the server’s budget is destroyed, up to HAND_US (1 s); an operation that finds every place held parks, off the VM’s thread, until an ended server’s waiter hands one over, or fails emfile at FILE_WAIT_US (10 s), the VM’s own bound and not a device’s error (a_fresh_connection_at_the_bound_waits_for_an_ended_waiters_hand_over: the connect returns Later at once and the operation completes when the hand-over comes; a_fresh_connection_past_the_bound_is_refused_emfile_and_the_live_ones_still_serve: every place held by a live server, the live ones still served). Once, before the VM waited for a hand-over at all, six ended pipeds whose waiters had not yet handed over refused a session’s seventh pipeline its first request as eio, and once a wait of 1 s on the VM’s thread was outlasted by a hand-over under load and refused a fourteenth (pipe-sixty-four on rv32 beside other work): the wait is now the operation’s, not the VM’s, and ten times longer. No call holds a thread while it waits: a console read, a file read and a read on a /net connection’s data file are each one request on the hub, so what a person or a peer takes to answer costs nothing but the request. Every 9P server in the image serves multiplexed sessions, so no call needs a thread of its own. What the scheduler’s thread still waits for itself: typed calls, which no hub carries (the console’s size and littlefsd’s rename), and the system volume’s module lookups, since loading code is synchronous in the VM; a server that stops answering one of them stops the VM. A scheduler away from its endpoint that long, past a server’s session bound, keeps its sessions: each waiter holds what arrives meanwhile and keeps calling (a busy caller). The serving natives and a job’s exit notice are launching’s (planned).

The forwards queue

A served endpoint’s thread never waits on the VM. Each call or send it takes is queued, with a copy of its lent bytes, in the VM’s forwards queue (serve.rs, Forwards), and the thread is back at its receive at once; the VM takes the whole queue at its next poll, the poll that takes the hub’s completions, each entry reaching its serving process as a request. A VM in its idle receive is woken by one send through the thread’s badge, carrying nothing: the VM says it is about to idle under the queue’s own lock, before its last look at the queue, so a thread that queues after that look sees the VM idle and wakes it, and one that queued before is found by the look; the wake waits at most WAKE_US (1 ms) at a time and only while the VM still says it idles, so a VM that woke on its own meanwhile is left to its poll. The queue is bounded: MAX_FORWARDS = MAX_SERVED (2) x (the calls one endpoint’s admission parks, 8 buckets of 4 = 32, each forwarded once, + SENDS = 32 one-way sends) = 128. The sends are what the steward can send a session before its VM polls: one lease_ended per live lease of the sponsor’s (sixteen under the latency manifest), one server_restarted per slot of its binding table, approval_waiting, with room over it (the steward). A thread that finds the queue full waits for the VM’s next poll before it takes its endpoint’s next message, as every forward once did (a_full_forwards_queue_holds_the_next_send_until_the_vms_poll): the residual, and the one way a sender’s bounded send to a served endpoint still expires with the serve thread alive. The queue’s worst hold is 128 entries of a lend’s MAX_BYTES each, 65,280 bytes, so 2,040 pages of the session VM’s heap (a cap of 11,220 pages holding the scanned peak twice, no forward in that scan), copied as the thread takes the message; a copy the heap refuses refuses the forward, as the lend’s own buffer once did, never a failed allocation in the VM.

Once the thread forwarded each message by a send to the VM’s endpoint that waited for the VM to take it, and a VM busy in Erlang took one per scheduler turn (its zero-wait poll each 2,000 reductions), so the thread could not take its endpoint’s next message until then: a burst of the steward’s lease_ended sends to one session was taken a turn each, 40 to 300 ms apart, the last of six returning 1.5 to 1.8 s after the deadline and of sixteen 2.1 to 2.8 s, and one send was lost at the steward’s 100 ms bound on rv64 at one hart. With the queue each send returns as soon as the thread takes it: when the queue landed, 37 to 68 ms after the previous at one hart on rv64, 42 to 86 on rv32 and 73 to 131 at two harts, the steward’s own work per exit between them, the last of six 1.1 to 1.3 s after its deadline and of sixteen 1.8 to 2.5 s, none lost over five boots; since the steward sends a burst’s notices before any exit’s effects the figures are its page’s (steward.md, “Failure and restart”) (steward-deadline-burst, steward-deadline-burst-16). On a block trace of the six-lease burst the thread’s one wake to the idle VM is taken in the thread’s own kernel entry and no send of the VM’s process blocks on its wake endpoint for the notices after it, where before each forward blocked there 107 ms median until the VM’s next turn. A reply to an idle VM is unchanged (a file read from an idle shell 1.0 to 1.3 s on the checked, traced kernel under icount, as before; beside a spinning process 1.4 to 1.6 s, the read’s own VM work at half the reductions). The VM’s poll cadence is unchanged: a busy VM still takes the queue once a turn, which under the checked kernel’s audits (about 60 % of a hart in the burst) is 100 to 140 ms at a session’s quarter share, a bench condition (a release kernel has no audits); a waiter’s hand-over of a completion to a busy VM still waits for that turn, one send a batch, 10 to 135 ms on the same trace. What the sponsor sees is the shell’s print of each notice, which its listener makes inline through the relay at 250 to 370 ms each, so the last [lease ended] line of six is drawn 2.3 to 2.6 s after the deadline and of sixteen 6.1 to 6.6 s: the shell’s, not the hub’s (the steward).

Residual: each connection costs its waiter thread, since a thread waits on one endpoint at a time; one thread waiting on several at once would be kernel work (IPC).

Why

An interpreter, not a JIT or a port of BEAM. BEAM is C and a JIT emits machine code, and Redoubt has neither C nor writable-and-executable pages anywhere (R11 (memory)). An interpreter in safe Rust is slower (up to about five times, measured on the host) and keeps the whole VM inside the language’s memory-safety argument. Speed is not a goal; auditability is.

One boundary. Every service the VM can reach is a method of Platform, with a default of nothing. A reader checks what a VM can do by reading one trait and one embedder, and an embedder that grants less gets a VM that can do less, with no code in between to trust.

One pinned compiler. Accepting exactly one OTP version lets the loader refuse everything else instead of carrying compatibility code for old formats, which is where parsers go wrong.

Natives are few, and of three kinds. A native is code every Erlang process in the session can reach, the ones handling hostile data among them, and it is not preempted. So what may be one is closed: OTP’s own natives, which the differential test holds to BEAM; the system’s, a fixed set with Elixir above it; and a primitive an interpreter cannot do fast enough, kept to the primitive. Nothing else is.

Its dependencies are userland’s. beamlet’s own crates hold no unsafe; the crates it links (RustCrypto, regex-automata, miniz_oxide, num-bigint) do, for speed and for the platform. That is the latitude an application has and privileged code does not (the tenets): the VM is per-principal code, and a bug in it reaches that principal’s own capabilities, which the kernel contains (trust tiers). What is shared with the servers is built from the same vendored bytes, and the rest is vendored too, one version of each (vendored dependencies); only what builds or tests on the host comes from crates.io, pinned by the lockfile.

One 9P client, not a method per service. Files, TCP and the console are all 9P on Redoubt, so one generic client in Rust covers them, and the framing (packet modes, line mode) stays in Erlang where OTP already has it. Adding a service means serving a tree, not growing the trait.