Contents

  1. System Basics
    1. Public VS Private State
    2. Change History
    3. Package and Module Indices
    4. Multiple Instances on a Host
      1. Zomp Instances are Special
    5. Establishing Connections
    6. Protocol Modes
    7. Broadcasts and Subscriptions
  2. Message Structure
    1. Message Types
    2. Sequencing Rules
    3. Universal Yes/No Semantic
    4. Hashes and Keys
  3. AUTH Mode
    1. Request Types
      1. Unsigned, Unserialized
      2. Signed, Unserialized
      3. Signed, Serialized
    2. Response Messages
    3. File Transfers
    4. Requests
  4. NODE Mode
    1. Session Establishment
    2. Session Messages
      1. Enrollment
      2. Unenrollment
      3. Fetch Requests
      4. Pull Event
      5. Refresh History
      6. Pings
      7. Broadcasts
      8. Redirects
  5. LEAF Mode
    1. Session Establishment
    2. Session Messages
      1. Subscribe
      2. List Packages
      3. List Versions
      4. Latest Version
      5. Fetch
      6. Sync Keys
      7. Pings
      8. Package Update Notification
      9. Serial Update Notification

Zomp/ZX Protocol v1

The Zomp/ZX v1 protocol is how Zomp nodes communicate with each other and clients over the network.

This document provides a high-level explaination of how Zomp and ZX work, background for the reasons behind some design choices, an explanation of all valid sequences, and a definition of every message valid within the protocol. The Erlang type of deserialized terms are linked throughout the document.

In addition to defining the verson 1 protocol this document also acts as a fast-acting sleep aid for sufferers of insomnia, a great target for grammar Nazis, and even a way for people with useful criticism to contribute suggestions or corrections! (Seriously, send suggestions by email, the IRC #erlang channel on Freenode or OFTC, SO chat, open a ticket at GitLab, or even utilize the vortex of time management pain.)

System Basics

Zomp nodes serve source code packages to client systems. Source packages are pulled by clients at the moment they are needed and then unpacked, verified, built and run on the fly. Caching is used heavily by top-level repository nodes, distribution nodes (arranged in a tree; more on that below), and clients in such a way that popular packages are accessible almost instantly from anywhere and place no stress on core infrastructure, but to make that possible in a robust way requires a fair amount of communication, hence the need for a protocol that does a bit more than simply fetch binary blobs.

The highest level of package/project organization within Zomp is called a "realm". Every realm has a "prime" node that sits at the top of a tree of distribution nodes (mirroring nodes). Different operators can create and support their own Zomp realms by hosting a prime node if they desire. Code from different realms can be mixed and matched within projects. There is a canonical realm for open source code called "otpr", but that is not the only kind of realm. For example, hosting a realm internally to serve private code within an organization in combination with the otpr realm is a normal method of simplifying deployment (especially for server-side projects like web, game, or chat backends), and there are some methods that make serving key-restricted code for proprietary client-side projects possible as well.

Zomp service nodes can host one or more realms simultaneously, as prime for one or more realms and/or a distributor for one or more others. By default Zomp and ZX nodes are initially only configured to see the otpr realm, but realms can be easily added or removed.

Public VS Private State

Zomp realms have two kinds of state: public and private.

Public state is visible to all end-users and is mirrored by every distributor.

Public state includes:

Private state is the data necessary for a prime node to make user role assignments, accept new package update submissions, and push new submissions through the review/approval process.

Private state includes:

The package files themselves are "ephemerally public", in that they are accessible from anywhere, but only one canonical copy exists on the prime node for a realm. All other nodes and clients only cache packages as they are requested, making popular packages fast to obtain while making new distributor node startup as lightweight as possible.

Change History

Zomp nodes keep track of each realm's public state by referencing a single-stepped, continuous integer sequence called "the serial" (looks ominous in quotes). Each action that affects the public state of the realm receives a unique serial and is commited to a change history log. The serial is how a client can tell whether it is ahead or behind an upstream node to which it has just connected relative to a given realm's history. The change history log is how a node can "fast forward" to a specific point in time or rebuild its state if indices get corrupted. Each realm has its own discrete indices and history. Actions that do not affect the public state of a realm do not require a serial and are not recorded in the history log.

While packagers and maintainers are authorized to make changes within their project's review queue, the review process is not visible outside of the prime node and does not affect the public state of the realm. Only a sysop can make changes that affect the public state, and it is against the sysop's key(s) that the realm history and package authenticity is validated.

Each command sent by a sysop that will affect the public state carries with it the serial number of the change as well as the sysop's signature and a timestamp (the timestamp is mostly for human reference). These signed change messages themselves, composed in order, constitute the sum public state of the realm.

Package and Module Indices

Zomp/ZX is able to maintain its promise of perfect rollback and absolute version integrity because the package and module indices are write only. It is not possible to remove a package or version from a realm (not without a great deal of trouble, anyway). Such a feature may become necessary at some point in the future, but there are no "remove package" verbs or commands in the v1 protocol or the v1 Zomp/ZX system.

Multiple Instances on a Host

Multiple programs can be launched by ZX at the same time in separate runtime instances. In the normal configuration all ZX runtime instances executing in the context of a single user share a common $ZOMP_DIR and associated code cache. Because of the way caching works conflicts can arise if multiple ZX instances attempt to access filesystem resources at the same time. Odd network conflicts and unnecessary upstream burdens on Zomp service nodes can occur (thinning the service tree) if multiple ZX or Zomp instances are each making their own discrete connections to upstream nodes in parallel.

To avoid conflicts ZX checks for a lockfile called $ZOMP_DIR/john.locke on startup and if it does not exist (or if it stale), writes one. It then opens a port on ::1 to listen for other ZX instances that require proxy service and records the port number and some other data to the lockfile, thereby declaring itself the "zx proxy" for the local system. If another ZX instance starts before the first one terminates it reads the local proxy port from the lockfile and connects to it. The local zx proxy handles all file writes, package builds, and upstream network connections on behalf of whatever other ZX instances are running on the system. If the zx proxy shuts down, crashes or disconnects then the next-oldest ZX instance takes over and all other ZX instances connect to it.

Zomp Instances are Special

In the case where ZX is launching a Zomp instance, ZX makes its initial round of upstream connections as a ZX client (if required to get Zomp built/upgraded) and then launches Zomp like any other program. Once Zomp has started, however, it checks with the zx_daemon to see whether it is the zx proxy and if not checks the lockfile to see whether any other Zomp instances are running. If another Zomp instance is running it just shuts down (there can only be one!), but if not it tells the zx_daemon to shut down all the LEAF connections and take over as the zx proxy (if it isn't already), and note in the lockfile that it is a Zomp note, not just a ZX node.

If another ZX node is the zx proxy at the moment then the Zomp instance's zx_daemon signals to the acting zx proxy that it is going to take over, at which point the current zx proxy redirects all of its client connection to the new zx proxy and then opens a connection to it also. Once the Zomp instances's runtime has taken control of proxy tasks it starts making upstream connections and opens public Zomp service ports. The zx_daemon communicates at that point directly with the Zomp's processes to satisfy local ZX queries.

Establishing Connections

When a ZX node becomes the zx proxy it checks every $ZOMP_DIR/etc/[Realm Name]/realm.conf file to get a list of realm primes, and $ZOMP_DIR/sys.conf for a list of any private mirrors configured for the local network (common in organizational settings).

The local network mirrors are checked first, and if unavailable then the prime node is contacted. If the prime node is "full" (already handling a significant number of downstream nodes) then it will provide a list of second-tier redirect options to the connecting node. If a second-tier node is similarly busy then it will provide a further redirect list, and so on. The node trying to make connections will attempt to conduct a wide traversal of the service tree until eventually a service connection is accepted. If all options are exhausted it will attempt to connect again to the prime node.

Clients (LEAF nodes) connect in a similar manner, trying both the prime as well as a cache of previously known working nodes. Mirror and primary nodes will always accept connections from LEAF mode clients in the case that they have no downstream NODE mode mirrors themselves, so LEAF nodes tend to accumulate at the bottom of the tree in heavily used realms that lack a high ratio of mirrors.

Protocol Modes

The Zomp protocol has three modes: AUTH (authenticated), NODE (peer), and LEAF (client).

Relative to a given realm Zomp hosts connect together in a tree over NODE mode connections and LEAF clients (like ZX) connect to any available Zomp hosts in the tree but do not connect to other LEAF nodes. Each realm for which the local system is configured is treated as a service requirement and any combination of upstream connections that satisfies all required realms is acceptable. Zomp and ZX both reconnect automatically to any realms for which service has been lost.

Broadcasts and Subscriptions

Service nodes (Zomp instances) acting as realm mirrors must stay in sync with each realm's history of index changes but do not need to download any packages until requested by a downstream client. This makes starting a new mirror node lightweight compared to other repository systems, but also means that syncing realm index data is critical as mirror nodes cannot build or repair a realm index based on a scan of the filesystem. When a mirror connects to an upstream node it receives a list of realms the upstream provides along with the current serial for each realm. It must verify whether it is ahead or behind its upstream provider relative to whatever realms it needs serviced and catch up if it is out of date (or find another node if the upstream is out of date). After verification is complete it also must begin receiving ongoing update notifications so that it stays in sync. The downstream node asks the upstream to enroll it in any realms the upstream provides that the node currently lacks.

ZX and other LEAF mode clients have no need to keep track of a realm's state. When a client connects it receives a list of realms and serials provided by the upstream. If a node provides some but not all of the realms a client requires then the client will continue to connect to other nodes in parallel until all of its realm service needs are met. Once connections have been established whatever queued work is required to be performed with upstream begins (version queries, package fetches, etc.).

Whilc ZX itself does not have any need for subscriptions, programs launched by ZX can interface with the zx_daemon to subscribe to updates on a per-package basis. If desired, a programmer can trigger an automatic update of a program, a client-side notification of new version availability, or take whatever other action is needed based on package update messages. It is also one way that notification utilities can be built as discrete programs for use by system administrators and developers. All LEAF mode notification subscriptions are elective and must be explicitly requested.

Message Structure

All messages are prefixed with a 4-byte (32-bit) size header indicating the size of the message in bytes. Zomp/ZX is not anticipated to have any individual messages larger than 4GB in the forseeable future (chunking would be implemented long before that!). To visualize this in Erlang bit syntax:

<<Size:32, Message:Size/binary>>

For example, the process to form a complete message from the term {"foo", "bar"} is (in the Erlang shell):

1> Message = {"foo", "bar"}.
{"foo","bar"}
2> MessageBin = term_to_binary(Message).
<<131,104,2,107,0,3,102,111,111,107,0,3,98,97,114>>
3> Size = byte_size(MessageBin).
15
4> <<Size:32, MessageBin/binary>>.
<<0,0,0,15,131,104,2,107,0,3,102,111,111,107,0,3,98,97,114>>

The Erlang inet socket option {packet, 4} can be (and for the most part is) used to automatically insert this 4-byte size header on the sending side and automatically interpret it on the client side -- thereby breaking the TCP stream into correctly sized Erlang messages that can be dealt with like datagrams.

Erlang atoms are never sent over the wire, so all conversion calls are made as binary_to_term(Term, [safe]) and should never crash (if conversion calls crash it is probably for the best...).

Before file transfers all versions of the protocol go into "transfer mode" by performing a double-ack prior to the beginning of a transfer which guarantees that no async messages will be emitted during the transfer and gives both side fair warning to prepare their socket states. During transfer mode the protocol the {packet, 4} option is turned off to avoid timeouts on large transfers over slow connections, though a per-segment receive timeout is still in effect. This also allows human-facing utilities such as ZX to enter a progress reporting mode during large transfers which would otherwise be unavailable were the underlying runtime to magically handle large transfers. The detail regarding the socket option applies only to EVM-based programs, of course.

NOTE: Message definitions beyond this point omit the 32-bit length header in their description except where specified.

Message Types

Erlang binary external term format can be interpreted in non-EVM languages using BERT libraries.

Sequencing rules

Message reaction timeout is 5 seconds unless otherwise specified.

A single Zomp node may host any combination of realms and each realm maintains its own tree of downstream nodes. It is therefore possible that two nodes may wind up with two connections between them, each serving as the upstream of the other relative to different sets of realms. A case of mutual connections, though rare, is not an error and does not affect the operation of the protocol within the context of a given connection.

Messages are infrequent and tiny compared to the types of traffic for which most networks are currently optimized (namely web traffic/ads, streaming video and audio, and high-frequency updates in games) and resource requests may need to be passed along a chain of nodes via persistent connections. For this reason the the Nagle algorithm (accumulation of "tinygram" data) is a pessimization and it is normal to enable TCP_NODELAY.

Universal Yes/No (affirmative/negatory) Semantic

The universal affirmative response (or acknowledgement) is <<0:8>>. In other words, a single byte with a value of 0. This is often used where the Erlang atom 'ok' is used in internal protocols and function return values to indicate a procedural success or a "ready" status. In some cases it is prepended to a larger message, and sometimes is the entire message itself.

The universal negatory response is a single byte with any higher value than 0, most typically a simple 1, but all values higher than 0 are negatory. Negatory responses can carry an optional Erlang term attached (usually a string) that indicates a reason for failure or error, though the higher values themselves may encode a specific error condition depending on their context. Therefore <<1:8, Reason/binary>> may include an empty Reason. The form taken by error responses with a "reason" segment attached is specified by the sequence in which they occur.

Hashes and Keys

Hashes are always SHA-512.

Keys are always 16K RSA keypairs.

These selections are constant for the life of this protocol version. This will almost certainly change in the future.

AUTH Mode

The authenticated mode provides a set of administrative actions that can be performed by users with the required authority and a handful of administrative queries that are publicly accessible but require a connection directly with the realm prime.

A realm's package data is write-only. The canonical copy of the realm's public state (indices) is maintained by the prime node. The bulk of client servicing work is delegated to mirrors, but the prime node is the only place package submission (by a packager), review/rejection/approval (by a maintainer), and final rejection/acceptance (by a sysop) can occur. The prime node is also the only place users and keys can be added, updated or removed, and packager and maintainer permissions assigned.

Request Types

AUTH mode connections are only maintained for the duration of a single action (which may, however, involve an arbitrarily long sequence of messages). There are three forms an AUTH message may take:

The message categories above are abbreviated from this point on as "UU" for "unsigned, unserialized", "SU" for "signed, unserialized" and "SS" for "signed, serialized".

Message signatures occur at the beginning of a signed segment and start with a 3-byte (24-bit) length header followed by the signature itself. The segment to which the signature applies begins at the byte immediately following the signature and continues to the end of the message. Empty signatures are simply 3 zero bytes as the length of the omitted signature is 0.

UU

The form of a UU request is:

<<"AUTH 1:", 0:24, Command:8, TermBin/binary>>

As the name indicates, these messages lack a signature, hence the explicit match on the zero signature length and lack of a Sig segment. TermBin unpacks to a zx:package_id(), a zx:package() or a zx:realm() depending on the command.

SU

Similar to a UU request, an SU request carries the protocol header and the entire, signed request with it in a single message:

<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, Command:8, TermBin/binary>>

Sig applies to everything in the message that occurs after the Sig itself, including the 8-bit Command segment and the payload TermBin.

TermBin unpacks to an Erlang tuple of the basic form: {Signatory, KeyName, Data}

It is necessary to interpret TermBin before verifying the signature of SU and SS messages, as the zx:user_id() and zx:key_id() of the signature is required, both of which are packed into TermBin itself.

The nature of the Data element is dependent on the specific command being issued.

SS

Unlike UU and SU messages, SS messages require a serial. An SS sender cannot know what serial number to use for the next message without checking with the prime node and having it temporarily lock the change history to prevent races with other concurrent SS messages. For this reason the initial message in an SS request sequence is a "null request": a signed request with 0 for its command code and no payload other than a tuple containing the name of the realm for which a lock is being requested and a timestamp:

<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 0:8, TermBin/binary>>

TermBin unpacks to a {Realm, Timestamp, Signatory, KeyName}. The timestamp must be within five seconds of the prime node's clock or the request will fail with an error code of 1. The 5-second window is consistent with the 5-second response timeout, limiting the effectiveness of a DoS attack based on a replay of the original message to spam locks (whether the remote and local clocks are within 5 seconds of one another can be a limiting factor, of course).

Upon receipt of a valid null request the node will temporarily lock its acceptance of AUTH requests for the given realm (returning error code 9 with an explanation string "Busy. Retry in 10 seconds.") and then issue an affirmative code with a tuple appended that contains the next valid serial number and the node's current timestamp:

<<0:8, Message/binary>>

The Message segment unpacks to a tuple of the form {Serial, Timestamp}, the elements of which the client then uses to pack its complete request message:

<<0:8, SigSize:24, Sig:SigSize/binary, Command:8, TermBin/binary>>

TermBin is a binary encoded tuple of the form: {Serial, Timestamp, Signatory, KeyName, Data}.

The nature of the Data element is dependent on the command being issued.

All successfully executed SS actions prompt the prime node to broadcast an update message to all enrolled downstream distributors and subscribed clients.

Response Messages

Basic response messages are indicated below. The affirmative response and the basic error messages defined here apply to all response message sequences. In some message sequences an affirmative response is followed immediately by a message or data segment. Specific sequences are defined below.

Response Meaning
<<0:8>> Success; affirmative; yes; ok; nominal; ready, etc.
<<1:8, Reason/binary>> Failure; negative; no; error; invalid message, etc.
Reason is an optional UTF-8 string explaining the error.
<<2:8, Version:16/integer>> Unavailable protocol version.
Version is an optional 16-bit integer indicating an available protocol version.
Extended protocol negotiation is not supported or planned.
<<3:8>> Requestor unknown.
<<4:8>> Unknown signature key.
<<5:8>> Bad signature.
<<6:8>> Permission denied.
<<7:8>> Invalid target.
<<8:8, Prime/binary>> Not the prime node for the indicated realm.
Prime is an optional zx:host() term indicating the correct prime host.
The node can only return Prime if it is configured for the target realm.
<<9:8, Reason/utf8>> The upstream node is unable to service the request.
Reason is an optional UTF-8 string explaining why.
(overload, read/write failure on the host, low disk space, etc.)
<<10:8, Reason/utf8>> Command-specific error.
Reason is an optional UTF-8 string explaining why.

File Transfers

Source file data (.zsp file contents) are transferred outside of normal message payloads. Each .zsp carries its own signature.

If a transfer is indicated then the sending node issues an affirmative <<0:8>> message, the receiver indicates it is ready to receive by sending a <<0:8>> of its own, and then the sender begins the transfer as a binary stream prefixed by a length header. The reason for this double-ack is that the receiver of the file almost certainly wants to turn off the {packet, 4} socket option and handle the large file receive manually. Once the transfer is complete the receiver sends an acknowledgement (usually indicating the file's signature checked out).

There is no support for partial transfers or transfer resumption. If a transfer fails for whatever reason (timeout, etc.) any buffered data or tempfiles on the receiving side should be purged. Inter-transfer receive timeout is 5 seconds; that is, if the delay between TCP segments is greater than 5 seconds the transfer will time out (this will almost certainly change as we get some real-world use data).

Requests

The available requests are:

All request sequences are explained below. Each explanation is prefaced by a psuedo-code example of a successful exchange sequence. Each command's section name is a link to that command's usage page.

Submit (SU)

%% CLIENT: Sends request
<<"AUTH 1:", SSize:24, Sig:SSize/binary, 1:8, TermBin:TSize/binary>> = Message
{Realm, Signatory, KeyName, {PackageName, Version}} = binary_to_term(TermBin, [safe])

%% SERVER: OK
<<0:8>>

%% CLIENT: Sends package data
<<PackageData/binary>>

%% SERVER: OK
<<0:8>>

Submission is the first step in adding a new version of a package to its realm. Once submitted it is available to be reviewed by a maintainer or sysop. Packagers, maintainers, and sysops are permitted to submit packages.

The Realm element of Signatory, KeyName and PackageID must all match, of course.

After verifying the request the prime node will return a response, affirmative if it is ready to receive the source package or a relevant error otherwise. If the client waits longer than 5 seconds to begin transferring the file then the prime node will terminate the connection. Once the transfer is complete the prime node will send another response, affirmative if the package checks out and a relevant error otherwise.

If the package has already been submitted for review (this submission is a duplicate) then error code 10 is returned with the message "Duplicate submission." in text following it. If the package version is already accepted and being served by the realm then error code 10 is returned with the message "Invalid version. That version number is already used." in text following it. If the package is not served by the current realm at all then error code 7 is returned.

List Packagers (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 2:8, TermBin:TSize/binary>> = Message
Realm = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of user data
<<0:8, ListBin/binary>>
[{UserID, RealName, ContactInfo}] = binary_to_term(TermBin, [safe])

Lists the packagers for a given package, returning their UserID, real name (an arbitrary string) and contact info.

List Packagers (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 3:8, TermBin:TSize/binary>> = Message
Package = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of user data
<<0:8, ListBin/binary>>
[{UserID, RealName, ContactInfo}] = binary_to_term(TermBin, [safe])

Lists the packagers for a given package, returning their UserID, real name (an arbitrary string) and contact info.

List Maintainers (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 4:8, TermBin:TSize/binary>> = Message
Package = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of user data
<<0:8, ListBin/binary>>
[{UserID, RealName, ContactInfo}] = binary_to_term(TermBin, [safe])

Similar to the "list packagers" command, it lists the maintainers for a given package, returning their UserID, real name (an arbitrary string) and contact info.

List Sysops (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 5:8, TermBin:TSize/binary>> = Message
Realm = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of user data
<<0:8, ListBin/binary>>
[{UserID, RealName, ContactInfo}] = binary_to_term(TermBin, [safe])

Similar to other list user type commands, it lists the sysops for the given realm, returning their UserID, real name (an arbitrary string) and contact info.

List Pending (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 6:8, TermBin:TSize/binary>> = Message
Package = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of package IDs
<<0:8, ListBin/binary>>
[PackageID] = binary_to_term(TermBin, [safe])

This command is used by reviewers (and curious people) to check what packages are in the review queue. Package maintainers check this list to see what packages have been submitted (or whether they have been submitted). It is assumed, of course, that packagers and maintainers are communicating outside of Zomp.

List Approved (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 7:8, TermBin:TSize/binary>> = Message
Realm = binary_to_term(TermBin, [safe])

%% SERVER: OK + list of package IDs
<<0:8, ListBin/binary>>
[PackageID] = binary_to_term(TermBin, [safe])

Similar to the "list pending" command, it lists what packages within the realm have been approved by package maintainers but are still awaiting a re-signature from a sysop before being permanently accepted into the realm. This is the most common query sysops typically run on large realms.

Review (UU)

%% CLIENT: Sends unsigned request
<<"AUTH 1:", 0:24, 8:8, TermBin:TSize/binary>> = Message
PackageID = binary_to_term(TermBin, [safe])

%% SERVER: Found
<<0:8>>

%% CLIENT: Ready
<<0:8>>

%% SERVER: Sends package data
<<PackageData/binary>>

%% CLIENT: OK
<<0:8>>

A review request initiates a download of a package that is staged for review but has not yet been approved. The package is still signed using the packager's key.

If the package version requested is pending review on the prime node then an affirmative response is returned (indicated the file was found), then the client responds with an acknowledgment (indicating it is ready to receive), and then the prime node sends the file data. The client then must verify the signature on the package before unpacking it fully. If the KeyID used for the package signature is not present on the client then the client will need to perform a get key request separately (ZX does this automatically).

Approve (SU)

%% CLIENT: Sends request
<<"AUTH 1:", SSize:24, Sig:SSize/binary, 9:8, TermBin:TSize/binary>> = Message
{Realm, Signatory, KeyName, {PackageName, Version}} = binary_to_term(TermBin, [safe])

%% SERVER: OK
<<0:8>>

An approve command is issued by a maintainer or a sysop to move a package forward from the "pending" queue to the "accept" queue. A package in the accept queue can still be downloaded via a "review" request.

An affirmative response means the package has been moved to the accept queue and removed from the pending queue. If the package is not in the pending queue when the "approve" request is received error code 7 is returned.

Reject (SU)

%% CLIENT: Sends request
<<"AUTH 1:", SSize:24, Sig:SSize/binary, 10:8, TermBin:TSize/binary>> = Message
{Realm, Signatory, KeyName, {PackageName, Version}} = binary_to_term(TermBin, [safe])

%% SERVER: OK
<<0:8>>

A reject command is issued by a maintainer or a sysop to remove a package from the approval process completely. A maintainer can remove a package from the pending queue, a sysop can remove a package from the pending or the accept queue. Once a package version is removed from the approval process the same version is free to be re-submitted. It is expected, obviously, that close coordination and communication is occurring among the packagers, maintainers and sysops outside of Zomp.

Accept (SS)

%% CLIENT: Null request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 0:8, TermBin/binary>> = NullRequest
{Realm, Timestamp, Signatory, KeyName} = binary_to_term(TermBin, [safe])

%% SERVER: OK + Status response
<<0:8, Status/binary>> = StatusResponse
{Serial, Timestamp} = binary_to_term(Status, [safe])

%% CLIENT: OK + Action request
<<0:8, SigSize:24, Sig:SigSize/binary, 11:8, TermBin/binary>> = ActionRequest
{Serial, Timestamp, Signatory, KeyName, PackageID} = binary_to_term(TermBin, [safe])

%% SERVER: Found
<<0:8>>

%% CLIENT: Ready
<<0:8>>

%% SERVER: Sends submitted package data
<<OriginalPackage/binary>>

%% CLIENT: OK
<<0:8>>
%% CLIENT: Sends re-signed package data
<<ResignedPackage/binary>>

%% SERVER: OK
<<0:8>>

In the exchange above each response going either direction may be an error instead of an affirmation. Errors coming from the client side adhere to the same semantics as errors coming from the server.

Initiate an accept procedure. The accept procedure is as follows:

  1. A null request is sent from a client (must be a sysop).
  2. The prime node returns a status message containing the next serial and a timestamp.
  3. The client sends a signed accept request.
  4. The prime node sends an acknowledgement if the package is found
  5. The client indicates it is ready to receive the package
  6. The prime node sends the original package data
  7. The client verifies the package with the packager's public key.
  8. The client resigns the package with the sysop's private key
  9. The client sends an affirmative response.
  10. The client sends the re-signed package to the prime node.
  11. The prime node verifies the signature.
  12. The prime node removes the package completely from the approval process.
  13. The prime node sends a final response and closes the connection.

If any error responses are encountered the connection is immediately terminated by either side and the package is not removed from the accept queue. If two sysops attempt to accept at the same time the first one to remove the package from the approval process "wins" and the other one will receive error code 7 as its final result (or a "busy" lock response if one request is received while the other is already in progress).

Add Package (SS)

%% CLIENT: Null request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 0:8, TermBin/binary>> = NullRequest
{Realm, Timestamp, Signatory, KeyName} = binary_to_term(TermBin, [safe])

%% SERVER: OK + Status response
<<0:8, Status/binary>> = StatusResponse
{Serial, Timestamp} = binary_to_term(Status, [safe])

%% CLIENT: OK + Action request
<<0:8, SigSize:24, Sig:SigSize/binary, 12:8, TermBin/binary>> = ActionRequest
{Serial, Timestamp, Signatory, KeyName, PackageID} = binary_to_term(TermBin, [safe])

%% SERVER: OK
<<0:8>>

Create a new package entry in the realm. This action is available only to sysops. Once a package's entry exists in the realm the sysop must go on to assign packagers and maintainers from the realm's userbase unless the sysop will be managing everything about the realm directly by himself (often the case for small, niche realms or realms internal to an organization). If users do not yet exist, of course, they will also need to be created before being assigned a role.

Note that this command does not upload any .zsp files to the realm, it only creates the index record for the package.

Add User (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 13:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, NewUserData} = binary_to_term(TermBin, [safe])
{NewUserName, RealName, ContactInfo, KeyData} = NewUserData

%% SERVER: OK
<<0:8>>

Adds a new user to a realm. Available only to sysops.

This command will fail with error code 7 if the user already exists.

Rem User (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 14:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, TargetUserName} = binary_to_term(TermBin, [safe])

%% SERVER: OK
<<0:8>>

Delete a user, remove them from all assigned roles, and purge their keys. Available only to sysops.

Attempting to remove the sysop if there is only one will return error code 7 (invalid target).

Add Packager (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 15:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, PackagerData} = binary_to_term(TermBin, [safe])
{Package, TargetUserName} = PackagerData

%% SERVER: OK
<<0:8>>

Grant a user the authority to package and submit new versions of a project. If the user or the package does not yet exist error code 7 is returned and either or both must be first created. Available only to sysops.

Rem Packager (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 16:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, PackagerData} = binary_to_term(TermBin, [safe])
{Package, TargetUserName} = PackagerData

%% SERVER: OK
<<0:8>>

Revoke a user's packaging authority. If the packager does not exist or is not assigned, or if the package does not exist then error code 7 is returned. Note that this only revokes authority for the indicated package, not all packages for which the user may have packaging authority. Available only to sysops.

Add Maintainer (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 17:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, MaintainerData} = binary_to_term(TermBin, [safe])
{Package, TargetUserName} = MaintainerData

%% SERVER: OK
<<0:8>>

Grant a user maintainer authority over a package. If the user or the package does not yet exist error code 7 is returned and either or both must first be created. Available only to sysops.

Rem Maintainer (SU)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 18:8, TermBin/binary>> = ActionRequest
{Realm, Signatory, KeyName, MaintainerData} = binary_to_term(TermBin, [safe])
{Package, TargetUserName} = MaintainerData

%% SERVER: OK
<<0:8>>

Revoke a user's maintainer authority over a package. If the maintainer does not exist or is not assigned, or if the package does not exist then error code 7 is returned. Note that this only revokes authority for the indicated package, not all packages for which the user may have maintainer authority. Available only to sysops.

Add Key (SS)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 19:8, TermBin/binary>> = ActionRequest
{Serial, Timestamp, Signatory, KeyName, NewKeyData} = binary_to_term(TermBin, [safe])
{TargetUserID, KeyData} = NewKeyData

%% SERVER: OK
<<0:8>>

Add a new public key to the TargetUserID's account. Only the target user or a sysop can use this command. It is not possible to add a sysop key in this manner (yet).

Rem Key (SS)

%% CLIENT: Request
<<"AUTH 1:", SigSize:24, Sig:SigSize/binary, 20:8, TermBin/binary>> = ActionRequest
{Serial, Timestamp, Signatory, KeyName, OldKeyData} = binary_to_term(TermBin, [safe])
{Owner, OldKeyID} = OldKeyData

%% SERVER: OK
<<0:8>>

Purge a key from the system. This command will fail with error code 6 if the key being removed is the final key available to the realm or a sysop. This command will fail with error code 7 if a packager or maintainer is trying to remove a key that has been used to sign a package that is in the process of review or approval.

This command will currently fail with error code 10 and the string "Cannot remove keys used for realm signatures." in the case that a sysop is trying to remove a key that has been used to sign a package or SS messages that are part of the realm history or package repo. (In the future this will change to a confirmation process, where the sysop can choose to re-sign everything in the realm with another key -- with the warning that this may be a long, complex operation on a very large realm and will prompt a cascade of update broadcasts to downstream nodes.)

NODE Mode

This mode connects two Zomp service nodes together in a directional manner. The node initiating the connection is downstream, the one receiving the connection is upstream. When one service node connects to another it announces the desired mode and version and sends up the list of public realms for which it is configured. The upstream node then responds with an affirmation and a list of provided realms and their serials, a redirect, or an error code. The connecting node checks the realms and serials provided against its own list of configured realms and their serials, and if it needs any of the realms being provided sends up an enrollment request for those realms. Once enrolled it ensures it is synced with the upstream state.

Session Establishment

NODE connections are requested by having the downstream (connecting) side send an ASCII string indicating the desired mode and version of the Zomp protocol terminated by a colon, followed by a binary term. The mode and version number are separated by a single space and the mode/version segment of the message is terminated by a colon. The binary term contains a pair with the publicly visible port number to redirect overload clients toward (or a 0 if the node is not publicly visible) as the first element, and a list of realm names as the second:

<<"NODE 1:", Term/binary>>

If the upstream node does not receive a protocol initiation message within 5 seconds of establishing a TCP connection it will disconnect. If it receives any message other than a valid protocol initiation message it will disconnect.

Upon receipt of a valid protocol request the upstream node will attempt to make a connection back to the connecting node to see whether the public port number is valid (unless it is set to 0, in which case this step is skipped). The upstream node responds with one of the following messages:

Response Meaning
<<0:8, Realms/binary>> Connection accepted (yay!).
Realms is a list of the realms provided by this node and their current serials.
The form of the list is: Realms :: [{zx:realm(), zx:serial()}]
<<1:8, Hosts/binary>> Redirect.
Hosts is a list of hosts and realms provided by each.
The form of the list is: Hosts :: [{zx:host, [zx:realm()]}]
<<2:8, Version:16/integer>> Unavailable protocol version.
Version is an optional 16-bit integer indicating an available protocol version.
Extended protocol negotiation is not supported or planned.
<<3:8, Reason/utf8>> The upstream node is unable to service the request.
Reason is an optional UTF-8 string explaining why.
(overload, failures on the host, resource exhaustion, etc.)
The node will always try to redirect before returning a failure.

After this list of available realms is received the downstream node has 5 seconds to make an enrollment request to a realm.

Session Messages

Like other modes, verbs in NODE mode are single bytes, but because NODE mode mixes downstream initiated and upstream initiated messages over the same channel the first bit is used to differentiate between the types: message sequences initiated from downstream having a 0 prefix, and message sequences initiated from upstream having a 1 prefix. This is specified in all message descriptions below using Erlang binary notation, for example a ping message is written: <<1:1, 0:7>> instead of <<128:8>>. Synchronous messages partially block: the requestor must still be able to handle messages while waiting for a response (except in file transfer mode), but will only make a single request at a time.

Downstream initiates:

Upstream initiates:

Enrollment

%% DOWNSTREAM: Sends enrollment request
<<0:1, 1:7, RealmBin/binary>> = Request
Realm = binary_to_term(RealmBin, [safe])

%% UPSTREAM: OK
<<0:1, 0:7>>

Enroll in the specified realm. Possible errors are:

All errors result in session termination. It is acceptable for the downstream side to reconnect immediately to the same upstream node.

Unenrollment

%% DOWNSTREAM: Sends enrollment request
<<0:1, 2:7 RealmBin/binary>> = Request
Realm = binary_to_term(RealmBin, [safe])

%% UPSTREAM: OK
<<0:1, 0:7>>

A downstream node that is not enrolled in any realms will be disconnected at the next ping.

Unenroll in the specified realm. Possible errors are:

All errors result in session termination. It is acceptable for the downstream side to reconnect immediately to the same upstream node.

Fetch

%% CLIENT: Sends fetch request
<<0:1, 6:7, Reference/binary>> = Request
PackageID = binary_to_term(Reference, [safe])

%% SERVER: Acknowledges
<<0:1, 0:7>>

%% SERVER: Found + Number of hops
%% This message loops, Distance decrementing each time the package gets one hop closer.
<<1:1, 3:7, Distance:8, Reference/binary>>
PackageID = binary_to_term(Reference, [safe])

%% SERVER: Ready to send; 0 Hops
%% Other messages are queued from receipt of this message until the end of the sequence.
<<1:1, 3:7, 0:8, Reference/binary>>
PackageID = binary_to_term(Reference, [safe])

%% DOWNSTREAM: Ready
<<0:1, 0:7>>

%% UPSTREAM: Sends the data
<<Blob/binary>>

%% DOWNSTREAM: OK
<<0:1, 0:7>>

Fetch requests are required when a downstream node receives a fetch request from one of its own downstream clients (Zomp or ZX) and does not have the requested package cached locally already. In this case it must forward the fetch request upstream. This process of forwarding fetch requests for uncached packages continues until the request reaches a node that has it cached or reaches the prime node of the realm (which maintains the canonical copy of all packages).

When a node receives a request for a package that it has stored locally it responds with <<0:1, 0:7, 0:8, Reference/binary>> where the second byte (0 in this case) indicate that the package is zero hops away (in other words, it was found locally). 0 hops indicates that the client needs to acknowledge its readiness to receive the package. The node that receives the "0 hops" message increments the message, resulting in a "1 hop" message, and forwards that to its downstream requestor. Subsequent downstream requestors receive these forwarded messages, all incremented by 1 each hop. After forwarding the "0 hops" message the receiving node indicates to the upstream node that it is ready to receive the file data by sending a "Ready" signal. Packages received in this manner are cached on the local node, and then forwarded downstream as requested.

The resulting effect is that requestors (whether Zomp nodes or ZX clients) receive a countdown of hop distances to the desired package, until they finally receive the package itself. Nodes between the prime and the bottom-tier clients cache popular packages quite quickly, but rarely have to deal with legacy or unpopular source packages, and can serve most package requests immediately from their local cache.

Transfers timeout in 30 seconds.

Possible errors to a package fetch request are:

A node should never be able to submit a request for a package that doesn't exist in the realm. If this happens it is very likely that the downstream node's package index is corrupted and needs to be rebuilt or that a ZX client sent a mistaken request. A timeout causes the downstream request chain to be aborted -- it is up to the original requestor to determine what to do.

Possible errors returned by a client in its final message (after receiving the package) are:

Pull Event

%% DOWNSTREAM: Request event by serial
<<0:1, 4:7, TermBin/binary>> = Request
{Realm, Serial} = binary_to_term(TermBin, [safe])

%% UPSTREAM: OK + Event data
<<0:1, 0:7, SigSize:24, Sig:SigSize/binary, Command:8, TermBin/binary>>
{Serial, Timestamp, Signatory, KeyName, Data} = binary_to_term(TermBin, [safe])

When a node finds it is missing an event or span of events, it requests them from the upstream by serial number. It is up to the requestor to choose which serials to request and assemble them. There is no semantic for requesting a span. If the requestor has a problem with the received data or receives corrupted data it can choose to re-try or disconnect and find another node.

Refer to the SS command definitions in the AUTH mode section of this protocol to learn the possible types the Data element of the message may take.

Refresh History

%% DOWNSTREAM: Request history
<<0:1, 5:7, Realm/utf8>> = Request

%% UPSTREAM: OK
%% Other messages are suspended from this point until the transfer is complete
<<0:1, 0:7>>

%% DOWNSTREAM: Ready
<<0:1, 0:7>>

%% UPSTREAM: Send data
<<HashBin/binary>>
History = binary_to_term(HashBin, [safe])

If a node has a blank history, a very large span of missing history, corrupted history, etc. it can request a complete copy of the current realm history to be forwarded. This results in a large binary transfer similar to how packages are transferred, so a double-ack and suspension of other messages until completion applies.

Pings

%% UPSTREAM: Ping
<<1:1, 0:7>>

%% DOWNSTREAM: Pong
<<1:1, 0:7>>

Response checks (keepalive pings) originate from upstream. One ping is made every 60 seconds of inactivity. If a pong is not received by the pinger within 5 seconds the connection is considered stale and is terminated by the upstream node. If the client is actually still alive it will initiate its connection routine again to reacquire whatever realms were lost.

Keepalive messages are suspended completely during data transfers (not queued), but transfer chunks/frames must be received within 5 seconds of one another or the receiver will consider the connection stale and disconnect.

Broadcasts

%% UPSTREAM: Sends broadcast to all enrolled downstream nodes
<<1:1, 1:7, SigSize:24, Sig:SigSize/binary, Command:8, TermBin/binary>> = Broadcast
{Serial, Timestamp, Signatory, KeyName, Data} = binary_to_term(TermBin, [safe])

Updates are broadcasted downstream from a realm's prime node to all NODE connections enrolled in that realm. The payload of each update message is the complete signed copy of the AUTH SS message that caused the update, so there is not anything new to define other than the broadcast prefix (see the AUTH section for a complete definition of SS messages).

Recieving an SS event broadcast initiates the same history update procedure as a "pull event" request. These messages are purely asynchronous. If a downstream node receives a bad broadcast it can choose to disconnect or request a re-send (via "pull event"), but it must not forward the bad message to its downstream nodes.

Clients in LEAF mode subscribe to specific package "accept" updates. In the case that the received SS message is an "accept" then the Zomp node checks its list of subscribers and forwards the message to any subscribers that are interested in the message.

Redirects

%% UPSTREAM: Sends a message to a specific downstream node.
<<1:1 2:7, Hosts/binary>> = RedirectMessage
[{zx:host(), [zx:realm()]}] = binary_to_term(Hosts, [safe])

%% DOWNSTREAM: Replies with OK
<<1:1, 0:1>> = OK

The upstream host may, at any moment, need to redirect its subordinate nodes to balance or open its downstream tree. This is usually a consequence of the upstream node acting as prime for a realm to which downstream nodes are enrolled. If the downstream node fails to reply with the reply timeout (usually 5 seconds) the upstream will terminate the connection.

LEAF Mode

This mode connects ZX and other clients to Zomp service nodes. When a client connects to a service node it announces its mode and version and waits for a list of provided realms and their current serials. If any configured but unserviced realms are provided by the Zomp node it will maintain the connection, if not it will disconnect. In the case of a system where only a single ZX instance is running and short tasks are being performed (very common on developer systems) connections may be of very short duration and occur frequently. In a situation where a high number of redirects are occurring (generally happens only to developers) a common remedy is to start a Zomp node locally to push queries down to the local system, even if no public service is being provided.

Session Establishment

LEAF connection requests are initiated by sending an ASCII string identifying the protocol mode and version number separated by a space, and a final colon. Unlike AUTH and NODE connection requests, nothing follows the colon.

<<"LEAF 1:">>

If this string is not received within 5 seconds of establishing a TCP connection the service node will terminate the connection.

Upon receipt of a valid initiation string the service node will respond with one of:

Response Meaning
<<0:8, Realms/binary>> Connection accepted (yay!).
Realms is a list of the realms provided by this node and their current serials.
The form of the list is: Realms :: [{zx:realm(), zx:serial()}]
<<1:8, Hosts/binary>> Redirect.
Hosts is a list of hosts and realms provided by each.
The form of the list is: Hosts :: [{zx:host, [zx:realm()]}]
<<2:8, Version:16/integer>> Unavailable protocol version.
Version is an optional 16-bit integer indicating an available protocol version.
Extended protocol negotiation is not supported or planned.
<<3:8, Reason/utf8>> The upstream node is unable to service the request.
Reason is an optional UTF-8 string explaining why.
(overload, failures on the host, resource exhaustion, etc.)
The host will always try to redirect before returning a refusal.

Unlike NODE connections, LEAF connections are permitted to remain connected and idle.

Session Messages

Like other modes, verbs in LEAF mode are single bytes, but because LEAF mode mixes client initiated and server initiated messages over the same channel the first bit is used to differentiate between the types: message sequences initiated from clients have a 0 prefix, and message sequences initiated from servers have a 1 prefix. This is specified in all message descriptions below using Erlang binary notation, for example a ping message is written: <<1:1, 0:7>> instead of <<128:8>>. Synchronous messages partially block: the requestor must still be able to handle messages while waiting for a response (except in file transfer mode), but will only make a single request at a time.

Clients initiate:

Servers initiate:

Subscribe

%% CLIENT: Send request
<<0:1, 1:7, Reference/binary>> = Request
PackageID = binary_to_term(Request, [safe])

%% SERVER: OK
<<0:1, 0:7>>

Once subscribed to a package any AUTH SS "accept" messages will be forwarded to the client as update alerts. It is not an error to subscribe to the same package twice, though clients should try to avoid sending duplicate subscriptions.

Possible error messages:

Unsubscribe

%% CLIENT: Send request
<<0:1, 2:7, Reference/binary>> = Request
PackageID = binary_to_term(Request, [safe])

%% SERVER: OK
<<0:1, 0:7>>

Unsubscribes from a package. Sending an unsubscribe request when not subscribed is not an error, though clients should try to avoid sending duplicate unsubscribe requests.

Possible error messages:

List Packages

%% CLIENT: Send request
<<0:1, 3:7, Reference/binary>>
Realm = binary_to_term(Reference, [safe])

%% SERVER: OK + list
<<0:1, 0:7, Result/binary>>
[Package] = binary_to_term(Result, [safe])

Request a list of packages provided by Realm.

Possible error messages:

List Versions

%% CLIENT: Send request
<<0:1, 4:7, Reference/binary>>
PackageID = binary_to_term(Reference, [safe])

%% SERVER: OK + list
<<0:1, 0:7, Result/binary>>
[Version] = binary_to_term(Result, [safe])

Request a list of versions available for the PackageID provided. The PackageID can have a complete, partial or omitted version segment. The response will be a list of versions constrained by the scope of the version segment provided if present.

Possible error messages:

Latest Version

%% CLIENT: Send request
<<0:1, 5:7, Reference/binary>>
PackageID = binary_to_term(Reference, [safe])

%% SERVER: OK + list
<<0:1, 0:7, Result/binary>>
Version = binary_to_term(Result, [safe])

Request the latest version available for the PackageID provided. The PackageID can have a complete, partial or omitted version segment. The response will be the latest version available for the package requested, constrained by the version segment provided.

Possible error messages:

List Sysops

%% CLIENT: Send request
<<0:1, 6:7, Reference/binary>>
Realm = binary_to_term(Reference, [safe])

%% SERVER: OK + list
<<0:1, 0:7, Result/binary>>
[Sysop] = binary_to_term(Result, [safe])

Request userdata for all sysops of the given Realm. Used mostly by users to find contact data for administrative reasons.

Possible error messages:

Fetch

%% CLIENT: Sends fetch request
<<0:1, 6:7, Reference/binary>> = Request
PackageID = binary_to_term(Reference, [safe])

%% SERVER: Found + Number of hops
%% This message loops, Distance decrementing each time the package gets one hop closer.
<<0:1, 0:7, Distance:8>>

%% SERVER: Ready to send; 0 Hops
%% Other messages are queued from receipt of this message until the end of the sequence.
<<0:1, 0:7, 0:8>>

%% CLIENT: Ready
<<0:1, 0:7>>

%% SERVER: Sends the data
<<Blob/binary>>

%% CLIENT: OK
<<0:1, 0:7>>

When a client needs to open a source package it first checks its local cache. If the package is not available in the local cache it makes a request to a service node. The service node will locate the package within the realm's node tree and tell the client how many hops away the package is found. As the package makes its way to the client the service node will report its progress, decrementing the number of hops as the package gets nearer. Once the number of hops is 0 the transfer will begin. All other messages are suspended once the "0 hops" message is sent by the service node until the end of the transmission.

Possible error messages to a fetch request:

(As above the Reference here converts to the PackageID of the target.)

Once the package is received by the client it will be opened and its contents and signature checked.

Sync Keys

%% CLIENT: Sends key sync request
<<0:1, 7:7, Reference/binary>> = Request
KeyID = binary_to_Term(Reference, [safe])

%% SERVER: OK + Keydata
<<0:1, 0:7, KeyBin/binary>>
<<SigSize:24, Sig:SigSize/binary, 19:8, TermBin/binary>> = binary_to_term(KeysBin, [safe])
{Serial, Timestamp, SignatoryID, SignatoryKeyID, Key} = binary_to_term(TermBin, [safe])
{KeyOwnerID, KeyData} = Key

When a client receives a package or SS message signed with a key that it doesn't have, it must request the key from upstream. Because the client may be an arbitrary historical distance from the current realm state the key data it receives may also be out of date. In that case the client must request the missing signatory key also, and may need to iterate this operation until it works its way back in key history all the way to the root key that started the realm. Once it has received the complete chain of keys it will proceed to validate them up the chain, and then perform whatever validation steps that were pending.

Pings

%% SERVER: Ping
<<1:1, 0:7>>

%% CLIENT: Pong
<<1:1, 0:7>>

Response checks (keepalive pings) originate from upstream. One ping is made every 60 seconds of inactivity. If a pong is not received by the pinger within 5 seconds the connection is considered stale and is terminated by the upstream node. If the client is actually still alive it will initiate its connection routine again to reacquire whatever realms were lost.

Keepalive messages are suspended completely during data transfers (not queued), but transfer chunks/frames must be received within 5 seconds of one another or the receiver will consider the connection stale and disconnect.

Package Update Notification

%% SERVER: Sends broadcast to all subscribed downstream nodes
<<1:1, 1:7, TermBin/binary>>
PackageID = binary_to_term(TermBin, [safe])

Clients that have subscribed to update messages for a package are forwarded each new available version of a package ID as it is accepted into its realm. Clients can use this as a trigger for forced update events, user notifications, or whatever else they might dream up.

Serial Update Notification

%% SERVER: Sends broadcast to all connected downstream nodes
<<1:1, 2:7, TermBin/binary>>
{Realm, Serial} = binary_to_term(TermBin, [safe])

When a realm experiences an update event downstream clients are informed via a serial update notification. This allows them to update their realm cache data and stay in step with the realm history.