Zomp/ZX File Definitions

Zomp and ZX make use of the same directory locations, configuration files, caches, bundles and locks. While it should not be necessary for users to go digging through any of these as they all have programmatic interfaces exposed via ZX, having clear file definitions and explanations can assist others in writing additional tools that augment Zomp and ZX. (It is also a handy reference for Zomp and ZX devs...)

Directory Layout

ZX installs itself by default to a location equivalent to $HOME/zomp (or %LOCALAPPDATA%\zomp in crazyland). An environment variable $ZOMP_DIR is defined by the ZX start script to refer to this location and can be set prior to calling the zx start script to indicate a custom install location.

The layout of the $ZOMP_DIR is as follows:

zx
zxh
john.locke  (if running)
etc/
    sys.conf
    version.txt
    [Realm Name]/
        realm.conf
        [username].user
        [Package Name]/
            [Version]/
var/
    [Realm Name]/
        history.log
        realm.cache
        realm.stash
        [Package Name]/
            [Version]/
tmp/
    [Realm Name]/
        [Package Name]/
            [Version]/
log/
    [Realm Name]/
        [Package Name]/
            [Version]/
key/
    [Realm Name]/
zsp/
    [Realm Name]/
lib/
    [Realm Name]/
        [Package Name]/
            [Version]/

The zx and zxh files are bash scripts that prepares the local environment for ZX to launch on a *nix type system. (A satirical equivalent called zx.cmd once existed, but the Windows launch strategy is changing considerably due to the weirdness inherent in that world.) The file john.locke is a lockfile used by ZX to determine whether another instance is running and how to contact it; ZX proxies file and network operations through the longest running instance to prevent conflicts.

The etc/, var/, tmp/ and log/ directories are provided as known locations for the use of ZX launched programs and take on their usual role within a system (they are free to ignore these, though). Inside of each a directory is created per realm, within which realm-level configuration and data is stored as well as a directory created per application/version that requires persistent configuration and data locations when run. An application can query zx for the absolute path to its config and data directories. The main setting file, sys.conf and the version indicator are the only files in the top-level of etc/.

The key/ directory contains a subdirectory per realm, and inside stores public and private keys in DER format as [Key ID].key.der and [Key ID].pub.der respectively. A key's global ID is the SHA512 hash of the public half of the key.

The zsp/ directory contains a subdirectory per realm and inside caches Zomp source packages as [Realm Name]-[Package Name]-[Version].zsp.

The lib/ directory contains a subdirectory per realm, inside which a directory per package is created, inside which a directory per version is created. Source packages are unpacked to their version directory first and then built and installed in place. Beam code for launched applications and specified dependencies is loaded directly by ZX from the installed locations in lib/.

Files

There are seven basic categories of files used by the system:

  1. Start Scripts
  2. The Lock File
  3. Configuration Files
  4. Bundles
  5. Caches
  6. Key Files
  7. Project Meta and Utilities

Start Scripts

The reason for having two different start scripts is to provide users (quite often developers) the option of running an application in "headless" (no erl shell attached) or "shell" mode.

The zx Start Script

Starts in headless mode. The common way for non-developers to run programs.

The zxh Start Script

Starts a program with an erl shell attached (most useful for developers).

john.locke

This is the environment-wide lock file for ZX instances. This file contains a port number listening on localhost that any new zx runtime can connect to and use as a proxy for operations that involve external resources:

This is how multiple ZX instances running concurrently from the same $ZOMP_DIR avoid clobbering each other.

Configuration Files

Zomp has a handful of configuration files it uses to keep track of the state of the system. Users should not have to manipulate them by hand (all have a command-line interface via ZX), though most are intended to be read with file:consult/1 and incidentally text-editor friendly. Compared to most systems Zomp/ZX files are simple to the point of triviality, and defaults can be restored to sanity with a simple command.

etc/sys.conf

The system-wide configuration file. This file is actually not necessary, and if it or attributes within it are missing the file will be fixed by ZX at start time and populated with defaults.

The default version of the file looks like this:

{timeout,     5}.
{retry,       3}.
{maxconn,     5}.
{managed,     []}.
{mirrors,     []}.
{status,      listen}.
{node_max,    16}.
{vamp_max,    16}.
{leaf_max,    256}.
{listen_port, 11311}.
{public_port, 11311}.

The timeout attribute is the base used to determine timeouts throughout the system. Most operations within the system have a timeout factor of 1000 milliseconds, meaning that most timeouts occur within 5 seconds as 5 * 1000ms = 5s.

The retry attribute tells the local node (whether ZX or Zomp) how many times to retry a remote action before giving up. This means that a ZX node will perform a connection attempt cycle to a needed realm this many times before giving up on a realm and reporting failure to the user (meaning, exhausting private mirrors, the prime node, and cached hosts X times). It also means that ZX or Zomp will attempt a fetch this many times before giving up, and because a fetch failure always results in a disconnect this means the fetch will be tried on this many different hosts before giving up. A high value has a higher chance of success overall, but also increases the wait time a user will experience if there are major network issues upstream.

The maxconn attribute controls how many parallel connection attempts can be made per configured realm. The default value is almost never reached in practice except on huge realms with scores of known cached hosts.

The managed attribute is a list of what realms this node serves as the prime node: its "managed" realms. As most nodes are not primes this always defaults to an empty list. The managed realms list can be updated with the zx takeover RealmName and zx abdicate RealmName commands.

The mirrors attribute is a list of Zomp mirrors that should be tried first. This attribute is usually set by network administrators in organizations to declare internal mirrors as high-speed options. The mirrors list can be viewed and edited with the zx add mirror [address [port]] command.

The node_max threshold makes it so that if new publicly accessible Zomp nodes are connecting and the threshold has not yet been reached, then the new connections are accepted regardless what realms they are requesting. If the threshold has been reached but some downstream nodes are not enrolled in realms for which this node is acting as prime and the connecting node lists the prime realm as an enrollment target then a downstream node that is not enrolled in a prime realm will be redirected to open a slot for the new connection. If all slots are full and the previous condition is not met then the new connection is immediately redirected.

The vamp_max threshold tells the node how many parasitic connections to accept. Most nodes are publicly visible and help spread the load out across the network which helps with redundancy and availability. A vampire connection is one that isn't publicly accessible for whatever reason, whether its public port is set to 0 (which indicates to upstream nodes that it is not reachable) or it simply can't be reached when the upstream node does its visibility check when the NODE session is established. The good news about data vampires is that they tend to be very low traffic clients which actually reduces external load across the entire public network.

The leaf_max threshold makes it so that new connections are accepted up to the threshold limit, and then redirected if downstream nodes exist for requested realms. If no downstream nodes exist for realms this node acts as prime, then the connection is accepted anyway.

The port settings are important to understand. Zomp listens on a port locally, but for that port to be visible to the outside world you almost always have to configure your network to forward traffic from a port on your public address to the port on the machine inside your network that is running Zomp. These might not be the same port numbers, so Zomp lets you designate two: the publicly visible one (that it will announced to other nodes when connections are made) and the local one actually being listened on.

etc/version.txt

This file contains a string representation of the current version of ZX on the system. This indicates to the ZX start scripts which version of ZX should be run and therefore how the path to it should be formed.

etc/[Realm Name]/realm.conf

Each realm has a configuration file that contains enough information to get a realm started from zero. This file defines the realm name, the prime host, the realm's root key (the first one the sysop created), and the user ID of the sysop who created the realm.

{realm,     zx:realm()}.
{prime,     zx:host()}.
{sysop,     zx:user_name()}.
{key,       zx:key_name()}.
{timestamp, calendar:datetime()}.

The realm.conf file is extracted and installed from a realm file when the command zx import realm RealmFile is run.

etc/[Realm Name]/[User Name].user

User data files represent the local user(s) information relative to the realm. Only users with a packager, manager, or sysop role in a realm require a user account (most of what ZX clients and all of what Zomp distribution nodes do is anonymous in the open source version). Creating multiple users within a single realm in a single ZX installation is possible, but unusual. In the case multiple users exist ZX will prompt the user to pick one before executing any commands that require authentication.

{realm,        zx:realm()}.
{username,     zx:user_name()}.
{realname,     string()}.
{contact_info, [zx:contact_info()]}.
{keys,         [zx:key_name()]}.

The data contained in a user file is basic, but combined with the realm name provides enough information to uniquely identify and therefore locate everything necessary to interact with a prime node on behalf of the user.

User files are created when the zx create user Realm command is run.

This file is rolled into an archive with relevant user keys when the zx export user command is run.

This file is added to a local installation with the zx import user UserFile command.

Bundles

Several files used by the system are archives. These bundles simplify tasks like transferring source files as signed packages and shipping around configuration data in a way that is relatively simple for users to deal with. All of these bundle files are the target of certain ZX commands.

Realm Files (.zrf files)

A "realm file" is a bundle that contains the etc/[Realm Name]/realm.conf file necessary to configure the local system and the realm's root key. The commands zx create realm and zx export realm RealmName both produce a realm file named "[RealmName].zrf" in the current working directory, and the command zx import realm RealmFile installs a realm from the realm file at the path provided to the command.

A realm file is simply an Erlang external binary containing:

The procedure to create the .zrf binary is:

{ok, RealmConf} = zx_lib:load_realm_conf(Realm)
{ok, KeyDer} = file:read_file(KeyPath)
Blob = term_to_binary({RealmConf, KeyDER})
ok = file:write_file(ZRF, Blob)

When a new realm is created the person who created it is the new sysop. The sysop either makes the .zrf file available publicly on a website or forum, or gets it into the hands of users some other way (email, configuration daemon, embedded in a VM or Docker image, etc.).

Public User Files (.zpuf files)

A public user file contains a binary representation of a tuple where user's data file is the first element and the user's and public key data (key names and the actual DER data) are the second. A .zpuf contains only public keys and is therefore the safe way to send a user profile to a sysop for addition to a realm.

This file can be created by using the command zx export user.

When someone wants to get a user account on a realm they do:

  1. Run zx create user on their own computer, making sure to select the realm to which they want to be added.
  2. Run zx export user, selecting the realm/user which was just created (if more than one exists).
  3. Send the resulting [realm]-[user].zpuf file to the sysop of the realm.

The sysop can then install it on the realm (using zx import user) without coming into contact with the user's private keys.

The internal structure of the file adheres to:

{ok, Bin} = file:read_file(Zpuf)
{UserData, PubKeyData} = binary_to_term(Bin)
UserData   :: [{Attribute :: atom(),  Value :: term()}]
PubKeyData :: [{KeyName :: zx:key_name(), DER :: binary()}]

The UserData element above is the result of file:consult/1 on the user's etc/[User Name].user file. The contained data gets written to the correct locations as part of the import/add procedure.

Dangerous User Files (.zduf files)

While the .zpuf files contain only public data, the .zduf files contain dangerous data. The difference is whether the user's private keys are present in the bundle.

These files can be created with the zx export user dangerous command and should only ever be handled by the user who owns the data. Never send it to sysop or store it on an unsecure device.

A user can use this file with zx import user to add his account (including private credentials) to another computer or ZX install location. This is very similar to moving SSH keys and GitLab credentials and settings around, but in this case the metadata moves with you and there is a little bit less to mess with than.

The internal structure of the file fits:

{ok, Bin} = file:read_file(Zduf)
{UserData, PubKeyData} = binary_to_term(Bin)
UserData :: [{Attribute :: atom(), Value :: term()}]
KeyData  :: [{KeyName :: zx:key_name(), KeyDER :: binary(), PubDER :: binary()}]

Source Packages (.zsp files)

The whole point of this circus.

These are binary strings with three major segments. It is easier to show than try to explain:

<<SigSize:24, Sig:SigSize/binary, Signed/binary>> = ZRP
<<MetaSize:16, MetaBin:MetaSize/binary, TarGz/binary>> = Signed
{PackageID, KeyName, Modules} = binary_to_term(MetaBin, [safe])

The Modules element in the tuple above is a manifest of all module names (as strings, not atoms) that are defined by the package.

The project itself is contained in the TarGz archive in the binary.

Package files are named according to their realm, package name and version number, then have the .zsp extension tacked on. So the package for the project "jumptext" from the repository "otpr" as of version 0.1.0 is: otpr-jumptext-0.1.0.zsp. Each realm has a package cache at zsp/[Realm Name]/ where source packages are cached once fetched and verified by signature.

Source packages submitted for review are the same as source packages for program installation from a realm, with the exception that submitted packages are signed by packagers, not sysops, and only available with the zx review PackageID command. That command extracts the package in place for inspection and use with the zx rundir command, not to the local system's lib/.

Caches

ZX and Zomp have to keep track of the current state of their configured realms to work efficiently. To find distribution nodes quickly ZX uses a caching strategy that keeps track of known redirect targets to which it attempts connections in parallel with attempts made to the prime node of each realm it is supposed to contact -- whichever succeeds first. Zomp must keep a complete record of the realm history for every realm it is configured so that it can provide missing bits of the realm's history to downstream requestors. Building the current index state out of a very long history is a bit cumbersome, so instead Zomp stashes a current snapshot of each realm's index data in the filesystem whenever it shuts down or has been idle for a minute and has a stale state stash.

All of these files can be replaced if corrupted or missing, the cost is some processing effort.

var/[Realm Name]/history.log

A realm's history log is a binary term version of the hash map that is maintained internally by the Zomp realm registry processes. The storage procedure is to write the output of term_to_binary/2 to this location, and reading the file is to read the contents of the file and pass it through binary_to_term/2. If anything about storing, loading, reading, etc. this file or the data contained inside is corrupted or unworkable the node will reset its serial to 0 and request a full sync of the realm's current history. The advantage of this approach is that the upstream node can simply send a copy of its current history.log instead of sending updates piece by piece.

var/[Realm Name]/realm.cache

Realm cache files are where ZX keeps track of a realm's current serial and known mirrors. If the realm cache is corrupted or lost it is discarded and another one will be generated in its place. If mirror connection attempts don't work out they are purged. If a prime node is connected and its serial is behind the currently tracked one the node assumes itself to be in error and resets the serial to whatever the prime claims it is.

The inside of a cache file contains the following:

{serial,  non_neg_integer()}.
{mirrors, [zx:host()]}.

Zomp realms are replicated in a degenerate, trickle-down sort of way. Every realm has a prime node (its prime server), any node may be prime for any number of realms, and any node may additionally mirror any number of other realms. Zomp nodes can also be used as organizational-internal private mirrors to speed things up in the case that a large number of users resides behind a single NAT interface or when a realm is entirely private to an organization. Because Zomp nodes come on and off the network at random, only the prime node and private mirrors are expected to be (somewhat) reliably available.

var/[Realm Name]/realm.stash

Zomp maintains the functional version of realm state as a set of interrelated index structures. Each realm has its own process that maintains this state. Building the state is a bit cumbersome on really large histories, so instead of rebuilding it every time something changes whenever the history.log is updated the indices are serialized and written to the realm's stash file. This speeds up start time considerably on large realms over rebuilding the indices from scratch using the history.log every time a node is started. If the realm stash is corrupted it is discarded and a new one is generated from the history.log.

The details of exactly how this works is subject to change, so for details on the current state of madness please refer to the zomp_realm.erl sources.

Key Files

Key files are DER-format files located in a path equivalent to key/[Realm Name]/[Key Name].[Type].der where the "Key Name" is usually "[User]-[Label]" and "Type" is "pub" for public keys and "key" for private keys. For example, the root public key for OTPR is key/otpr/zxq9-root.pub.key.der.

In the normal case a software user will only need the realm and package keys provided with the initial realm configuration bundle.

key/[Realm Name]/[Key Name].pub.der

The public half of an RSA keypair as used by the system. The only thing of special note is how the name is derived. A key ID looks like this: {Realm, KeyHash} where Realm is an ASCII string and the KeyHash is the binary SHA512 hash of the public half of the key (which can always be derived from either half). The path to the key can therefore be derived from its ID (the hash converted to base 36) with the extension ".pub.der" or ".key.der" added.

key/[Realm Name]/[Key Name].key.der

The private half of an RSA keypair as used by the system. The naming convention is the same as the public half, but the extension is ".key.der" in this case. These should never be exposed off of the user's system to whom they belong. Nothing in Zomp or ZX ever requires private keys to travel from a user's system -- even a realm's prime node is completely void of private keys unless it happens to be the personal computer of the sysop and he is hosting it directly (which isn't unheard of, but is a bit unusual outside of an organizational or short-term community setting like hosting a realm inside a small business or at a conference or hackathon).

Project Meta and Utilities

ZX provides a few utilities and conveniences for developers to manage projects just a little bit easier (or that's the intent, anyway). To make it easy for ZX to do its job without too much guessing a bit of meta about a project is kept within the project's root directory, the Erlang .app file is maintained in the course of this, and an optional utility script can be put in place by a developer to make things happen like initiating native code builds (like NIFs in Rust or C) or whatever else might need to happen at build time.

Not listed here but necessary is an Emakefile (a very simple version of which will be created if it doesn't already exist).

[Project Dev Dir]/zomp.meta

A zomp.meta file must exist in the root directory of any Zomp-compatible project. A zomp.meta file lets Zomp know how to handle the project, how to build it, where to put it, what version it is, and other information required to describe the project. A zomp.meta file is a list of Erlang terms readable by file:consult/1 (it is a properly list), though internally it is converted to an Erlang map.

[Project Dev Dir]/zmake

This file is optional and will be called via os:cmd/1 if present. This must be placed in the root directory of the project or it will not be found. Typically this is a Bash script, and the most common reason for it to exist is to initiate the building of NIF or Mercury code (a task ZX is not yet ready to integrate on its own just yet).

Code built using zmake should go in the usual location, but there are no contraints on what zmake might do. As a packager or developer this is your omni-hook and gives you complete freedom to make magical things work with ZX (or totally blow up the user's system -- so be careful about your user assumptions!).

[Project Dev Dir]/ebin/[App Name].app

Erlang .app files are part of how OTP makes sure its application management procedures are doing what they are supposed to do. ZX is compliant with OTP, though it doesn't deploy code as pre-built releases the way Erlang was traditionally used. Everything about Erlang releases is perfect for deploying code to phone switches or other embedded devices that require highly robust behavior, but it isn't quite as good a fit for programmers deploying web services, game servers, messaging infrastructure, and client-side utility and GUI code. Everything about Erlang/OTP is awesome for taking full advantage of modern ridiculously-multi-core hardware and inherently networked systems, though -- so Zomp/ZX makes an attempt to strike a balance between the two worlds. That means it must comply with OTP expectations, and that means keeping the .app file up to date.

The most important thing ZX does with an .app file is keeping the current version number up to date with the version in zomp.meta, and (probably more important) keeping the "modules" list accurate. ZX will not, however, modify anything else in the file aside from when it is initially written if the project was created with ZX.