Files and binds

A file on Redoubt is something a server serves over 9P, reached through a connection in the session’s namespace. Elixir’s File, IO and Path work unchanged on top: beamlet’s file natives speak 9P to the file server, one instance per volume, which keeps the data in walfs on the SSD (walfsd), or in littlefs on a flash medium (littlefsd). There are no permission bits, no owners, no symlinks and no hard links. Access is by capability (holding the connection) and by label (the volume’s labels against the caller’s); a bind puts a connection the session already holds at another path.

Purpose

People and agents work with files more than with anything else, and the shape of the file system is where Redoubt differs most visibly from Unix. This page says what File does over 9P, which Unix habits carry over and which do not (chmod, ln -s, an atomic rename between two directories), how labels decide who may read and write a volume, and how a directory is shared with another principal.

How to use it

Ordinary Elixir:

File.write!("/home/alice/notes.txt", "buy milk\n")
File.read!("/home/alice/notes.txt")
File.ls!("/home/alice")
File.cp!("/home/alice/notes.txt", "/work/notes.txt")   # across volumes: a copy loop
File.rename!("/home/alice/a.txt", "/home/alice/b.txt")  # one volume: the server renames

The shell’s commands are the same operations with short names: cat, cp, mv, rm, mkdir, ls, stat (the shell). A bind names a connection the session holds under another prefix:

{home, _rest} = ns_lookup("/home/alice")   # the connection behind the prefix
bind("/h", home)
File.ls!("/h/projects")                     # the same files as /home/alice/projects

Redoubt-only operations are in Redoubt.File: copy_file, a copy the file server makes itself. File.rename within one volume is already the file server’s own rename, between two directories of it; per-file metadata through the server’s set_attr and get_attr is planned.

What it can and cannot do

Files over 9P

flowchart LR
    F["File, IO<br/>(Elixir)"] -.-> FM[":file, file_io_server<br/>(OTP, unchanged)"]
    FM -.-> PF["prim_file natives<br/>(beamlet)"]
    PF -.-> C["the 9P client<br/>(beamlet's Platform)"]
    C -.->|"9P over call and lend"| LFSD["walfsd, one per volume"]
    LFSD -.-> LFS["walfs"]
    LFS -.->|"typed calls"| B["blkd"]
    B -.-> D["virtio block device"]

Figure: the file I/O path from File to the block device. The links are built; they are drawn dashed until Elixir’s File runs over them in a session. The servers are the file server’s and blkd’s pages; littlefsd serves the same 9P and typed calls over littlefs.

File.read!/1 becomes OTP’s file module, whose prim_file calls beamlet implements over its 9P client (beamlet; userland/otp/redoubt/src/files.rs). The client resolves the path in the session’s namespace, walks the rest of it on that connection, and reads. A path above a binding, such as /, is a directory the namespace answers itself; a path neither inside nor above one is :enoent (sessions).

• A fid is the file descriptor, and a directory fid is a capability. The file position lives in the VM, because 9P reads and writes carry explicit offsets; nothing else is kept between calls. A closed file’s fid is clunked, and comes back when the server has let it go.
• Everything moves in pieces: a read asks at most what one answer carries (the 64 KiB msize less its header), a write at most a page, each one request on the VM’s hub (asynchronous underneath), so only the Erlang process that asked waits for it.
• Every hub request ends in exactly one answer to its caller, within its bound. The operation waits at most FILE_WAIT_US (10 s) for each request, busy retries counted, and for a connection’s place at MAX_WAITERS; past that it fails with the name of what it waited for (etimedout an answer, ebusy a share, emfile a place), the connection kept and a later answer dropped, after the caller has its own; the fid that request walked or held is clunked then, as a closed file’s is, or its number given back if the walk it asked for failed, so the server lets go of every fid the hub gave up on and the connection’s pool recovers (a give-up that kept the fid leaked one at the server and one in the pool per etimedout, and under sixteen agents’ timeouts drained the account’s working pool until the server refused the late starters’ first read). The bound is each request’s, so an operation’s is the sum over its requests: a write to append is a walk, a create or open, a stat, a write per page and a clunk, five or six requests, and one of them took 47 s to fail under sixteen agents’ load before it was answered. A fid the operation still holds at its end (a walk whose open the server refused, a request given up on while it waited to go again) is clunked as a closed file’s is, off the caller’s path: the answer is the caller’s at the bound, and the clunk is cleanup, the fid given back when the server answers it. The VM’s own calls, its code loading, wait in place with no bound of their own: the hub answers by construction, and a second clock on the same wait would hide a fault of the hub’s as a timeout of the VM’s. A server whose answer cannot be asked for again therefore parks a request for strictly less than the bound, by a stated margin: ipd’s data and ctl reads for DATA_WAIT_US and CTL_WAIT_US, 8 s, the 2 s margin the kernel’s delivery of the reply, so a read’s timeout reaches the hub before the hub gives up, and the bytes that arrive after it wait in the socket’s buffer for the read made next (pinned and abandoned calls). None of it waits on the VM’s thread. The file server’s rename is a typed call, which no hub carries: it is made on the VM’s thread.
• Plain 9P2000, with no Unix extensions: a stat has a name, a length, a modification time and a qid (the server’s identity and version for the file), and nothing else. Custom per-file metadata is the file server’s typed set_attr and get_attr, kept in the volume’s attributes.
• An error is a Redoubt error first. The file server refuses with its own reasons (not_found, not_permitted, exists, not_dir, removed for a fid whose file was removed, too_large for an attribute over its limit), and labels and budgets add theirs, each a name of the one table (wire). Everywhere but the File boundary a Redoubt error keeps its name; there, beamlet’s platform maps each name to the POSIX error OTP’s file expects, by the table’s last column.

Refuse visibly; report only real fields. The File API does not emulate POSIX: no program should rely on a permission bit that no server enforces, because authority on Redoubt is the capability, never a mode bit. :undefined is within OTP’s own file_info type for exactly these fields, so nothing is invented. This is the one statement of the rule; file transfer follows it (file transfer). Below File the refusal is visible: each native that would set a mode, an owner or a time answers enotsup for a file that is there. OTP’s own write_file_info takes that answer as done, as it does on a system without the field, so standard-library code that writes a file’s info back (the mode preservation in File.cp and File.cp_r, File.touch) succeeds and sets nothing, and is never fed a fake value; the M5 (self-hosted development) case that compiles a Mix project on the box catches any caller that breaks. The host tests above hold the undefined fields, the natives’ enotsups and a missing path’s enoent, and bench:beamlet-files sees mode undefined in a boot, a file’s info written back as ok and written to a missing file as enoent.

Residuals, each a departure from the table, until the file server serves what it needs:

• A request that times out retires its fid. An operation whose request went unanswered for FILE_WAIT_US keeps that fid out of use for the connection’s life, as a lost reply’s is, since the server may still hold it; a connection has MAX_FIDS (64) of them, so a server that never answers costs its connection at most 64 timed-out requests before every operation on it is emfile. The VM’s own waits (the console’s writes) consult the console’s retry, not a file operation’s deadline: a timed-out operation is noticed when the VM next idles or takes a completion, bounded by the console’s progress.
• No times are set and none is stored. The 9P skeleton refuses Twstat, so a time given to File.write_stat or File.touch is not kept, though the call is :ok, and a cut at a position (:file.truncate/1) other than an open’s is {:error, :enotsup}; and no file server has a clock to keep a modification time by, so it reads as 0 (1970).

Copying, moving, removing and binds

Two paths on different prefixes are usually on different servers, and one server cannot act on another’s files. So what an operation costs depends on where its two ends are:

Binds. bind(prefix, conn) adds an entry to the session’s own namespace: the connection conn, already held, appears at prefix. It changes nothing on any server and creates no authority, and no other process sees it. It replaces what Unix does with symbolic links, hard links and bind mounts: giving something a second name. A bind at a prefix already bound replaces what was there, and the table holds at most 64 entries, so a bind of a new prefix past them is {:error, :system_limit} (beamlet). A bind lasts as long as the session, or as long as the budget the connection came through when it came through one the session carved (the send right a launch with serve in that budget returned, or a handle a call on such a connection brought): destroying the budget unbinds every prefix naming the connection, so a path there is :enoent and not a dead server’s refusal, and the session’s handle to it is closed when its last copy goes (a pipeline’s piped, bound at /dev/pipe; a_served_servers_binding_and_handle_go_with_its_budget).

• Replacing is within the session’s own authority. A bind over /boot or /dev/cons changes what the session’s own exec reads its program from and mints its child’s console at, as any path does: a session can already run any code it writes, so a program read from elsewhere gains nothing (native programs). /boot in a session is the boot bundle only while the session leaves it so.
• A child gets only what its launcher writes into its startup block: the shell’s exec writes /dev/cons alone, so a session’s binds reach a child only if the session passes them on (native programs).
• One connection, one badge. A connection bound at two prefixes is one connection, with one badge and one label set, so every server’s check is the same through either name. A handle the VM holds that is no connection yet is attached once, over 9P; a server that does not speak it, or a budget, is not_a_connection.

Labels on files

Status: planned · M1 (sessions over SSH, kept apart)

Labels are per volume: each volume has its own file server instance and its own label set, fixed when the volume is set up (the file server). The file server checks every request against the caller’s label set, which the kernel stamps on the message (R14 (unforgeable sender)), by the servers’ label rule (labels):

• Read needs the caller’s labels to include the volume’s. A session with {tax} reads an unlabelled volume and the tax volume; an unlabelled session reads only unlabelled volumes.
• Write needs the caller’s labels to equal the volume’s. A vault session cannot write its principal’s unlabelled home, so it cannot copy labelled data down into it.
• Metadata follows the data. Names, sizes and listings of a labelled volume are refused to a caller that cannot read it, so an unlabelled caller can neither read labelled data nor learn that it exists.

Open: none.

Sharing a directory

Status: planned · M6 (persist, install, share)

Alice shares a directory with Bob through the steward. The only delegation primitive is new_connection(root, quota): it mints a connection rooted at a subdirectory, with its own byte quota, and returns it with a random id (the serving library).

1. Alice asks the steward to share /home/alice/shared with Bob.
2. The steward mints the share into a revocation scope: a budget with no pages, processes or weight, made under Alice’s budget only to be destroyed later. It passes the connection to Bob.
3. Bob can narrow it further: asking the file server for shared/sub gives a handle that keeps the share’s stamp (R9 (stamps)).
4. Alice un-shares: the steward destroys the scope, and every handle stamped with it dies, sub included, wherever the copies went (R10 (destruction)).

Labels still decide access: a share of a labelled volume reaches only a principal whose sessions carry the label. A project, a principal sponsored by several members, shares its own volume the same way, one revocation scope per member (the steward).

Open: none.

Why

Capabilities instead of permission bits. A mode and an owner are a question every server asks about every file (“may this user do that?”), and the answer depends on a global table of users. A connection is the answer given once: holding it is the permission, narrowed by the server when it was minted (a subdirectory, read-only). There is nothing to chmod because there is no table to consult.

No symlinks or hard links. A symbolic link is a path the server follows on the client’s behalf, which is how a program is tricked into writing where it should not (a link from /tmp into someone’s home). A hard link gives one file two parents, which breaks per-directory quotas and makes “remove” mean two things. A bind gives something a second name in one process’s namespace, and cannot point anywhere that process could not already reach.

Rename is not always atomic, and the docs say so. A rename between two servers cannot be atomic without a transaction protocol between them, and a transaction protocol between file servers would be the largest shared code in the system. Redoubt keeps each server alone and says which renames are copies.

Removing an open file succeeds. If a server refused to remove a file another client has open, the refusal would tell one client something about another: a channel. So a remove always does what it says.