The Intellectual Wilderness There is nothing more useless than doing efficiently that which should not be done at all.

2020.02.3 15:19

X-Y Problems

Filed under: Computing,Science & Tech,Society — Tags: , , , , , — zxq9 @ 15:19

People obsess about their X-Y problems to the point of ignoring accepted wisdom, plugging their ears to the deafening silence of the solution’s instructive whisper, picking themselves up as hard as they can by their own knees and wondering why they can’t fly.

They then run off and formalize their wrong solution as a PR into a core project.

If core maintainers aren’t mindful they’ll incorporate these disturbances into a previously still space, and if they are indelicate they will piss off the misguided (but industrious) boob who made the PR who is already by this point fanatically dedicated to his wrong solution and the idea that nobody “gets it” but him.

Ah, another day at the Bazaar.

2020.01.9 00:26

Packaging and Distributing/Deploying Erlang GUI apps with ZX

Filed under: Computing — Tags: , , , , , , — zxq9 @ 00:26

In the last two posts I wrote walkthroughs for how to create new CLI and GUI apps in Erlang from scratch using a tool called ZX. Today I want to show how to package apps and publish them using Zomp (the package distribution system) and get them into the hands of your users with minimal fuss.

To understand how packages are distributed to users (the “why does anything do anything?” part of grokking ZX/Zomp), one must first understand a bit about how Zomp views the world.

Packages in the system are organized by “realms”. A code realm is like a package repository for a Linux distribution. The most important thing about realm/repository configuration is that you know by each package’s signature whether it really came from the it claims as its origin. Just like Linux repositories, anyone can create or host a Zomp realm, and realms can be mirrored.

(As you will see in a future tutorial, though, administration and and mirroring with Zomp is way easier and flexible than traditional Linux repositories. As you will see below, packaging with ZX is just a single command — ZX already knows everything it needs from the zomp.meta file.)

In this example I am going to put the example CLI project, Termifier GUI (a toy example app that converts JSON to Erlang terms) into the default FOSS realm, “otpr”. Because I am the sysop I have packaging and maintenance permissions for every package in the realm, as well as the sole authority to add projects and “accept” a package into the write-only indexes (packagers have “submit” authority, maintainers have “review”, “reject” and “approve” authorities).

[Note: The indexes are write only because dependencies in ZX are statically defined (no invisible updates) and the indexes are the only complete structure that must be mirrored by every mirroring node. Packages are not copied to new mirrors, they are cached the first time they are requested, with mirror nodes connected in a tree instead of a single hub pushing to all mirrors at once. This makes starting a new mirror very light weight, even for large realms, as no packages need to be copied to start (only the realm’s update history, from which the index is constructed), and packages in high demand experience “trickle down” replication, allowing mirrors to be sparse instead of complete. Only the “prime node” for a given realm must have a complete copy of everything in that particular realm. Nodes can mirror an arbitrary number of realms, and a node that is prime for one or more realms may mirror any number of others at the same time, making hosting of private project code mixed with mirrored public FOSS code a very efficient arrangement for organizations and devops.]

In the original Termifier GUI tutorial I simply created it and launched it from the command line using ZX’s zx rundir [path] and zx runlocal commands. The package was implicitly defined as being in the otpr realm because I never defined any other, but otpr itself was never told about this, so it merely remained a locally created project that could use packages hosted by Zomp as dependencies, but was not actually available through Zomp. Let’s change that:

ceverett@okonomiyaki:~/vcs$ zx add package otpr-termifierg

Done. That’s all there is to it. I’m the sysop, so this command told ZX to send a signed instruction (signed with my sysop key) to the prime node of otpr to create an entry for that package in the realm’s index.

Next we want to package the project. Last time we messed with it it was located at ~/vcs/termifierg/, so that’s where I’ll point ZX:

ceverett@okonomiyaki:~/vcs$ zx package termifierg/
Packaging termifierg/
Writing app file: ebin/termifierg.app
Wrote archive otpr-termifierg-0.1.0.zsp

Next I need to submit the package:

ceverett@okonomiyaki:~/vcs$ zx submit otpr-termifierg-0.1.0.zsp

The idea behind submission is that normally there are two cases:

  1. A realm is a one-man show.
  2. A realm has a lot of people involved in it and there is a formal preview/approval, review/acceptance process before publication (remember, the index is write-only!).

In the case where a single person is in charge rushing through the acceptance process only involves three commands (no problem). In the case where more than one person is involved the acceptance of a package should be a staged process where everyone has a chance to see each stage of the acceptance process.

Once a package has been submitted it can be checked by anyone with permissions on that project:

ceverett@okonomiyaki:~/vcs$ zx list pending otpr-termifierg
0.1.0
ceverett@okonomiyaki:~/vcs$ zx review otpr-termifierg-0.1.0
ceverett@okonomiyaki:~/vcs$ cd otpr-termifierg-0.1.0
ceverett@okonomiyaki:~/vcs/otpr-termifierg-0.1.0$ 

What the zx review [package_id] command does is download the package, verify the signature belongs to the actual submitter, and unpacks it in a directory so you can inspect it (or more likely) run it with zx rundir [unpacked directory].

After a package is reviewed (or if you’re flying solo and already know about the project because you wrote it) then you can “approve” it:

ceverett@okonomiyaki:~/vcs$ zx approve otpr-termifierg-0.1.0

The if the sysop is someone different than the packager then the review command is actually necessary, because the next step is re-signing the package with the sysop’s key as a part of acceptance into the realm. That is, the sysop runs zx review [package_id], actually reviews the code, and then once satisfied runs zx package [unpacked_dir] which results in a .zsp file signed by the sysop. If the sysop is the original packager, though, the .zsp file that was created in the packaging step above is already signed with the sysop’s key.

The sysop is the final word on inclusion of a package. If the green light is given, the sysop must “accept” the package:

ceverett@okonomiyaki:~/vcs$ zx accept otpr-termifierg-0.1.0.zsp

Done! So now let’s see if we can search the index for it, maybe by checking for the “json” tag since we know it is a JSON project:

ceverett@okonomiyaki:~/vcs/termifierg$ zx search json
otpr-termifierg-0.1.0
otpr-zj-1.0.5
ceverett@okonomiyaki:~/vcs/termifierg$ zx describe otpr-termifierg-0.1.0
Package : otpr-termifierg-0.1.0
Name    : Termifier GUI
Type    : gui
Desc    : Create, edit and convert JSON to Erlang terms.
Author  : Craig Everett zxq9@zxq9.com
Web     : 
Repo    : https://gitlab.com/zxq9/termifierg
Tags    : ["json","eterms"]

Yay! So we can now already do zx run otpr-termifierg and it will build itself and execute from anywhere, as long as the system has ZX installed.

I notice above that the “Web” URL is missing. The original blog post is as good a reference as this project is going to get, so I would like to add it. I do that by running the “update meta” command in the project directory:

ceverett@okonomiyaki:~/vcs/termifierg$ zx update meta

DESCRIPTION DATA
[ 1] Project Name             : Termifier GUI
[ 2] Author                   : Craig Everett
[ 3] Author's Email           : zxq9@zxq9.com
[ 4] Copyright Holder         : Craig Everett
[ 5] Copyright Holder's Email : zxq9@zxq9.com
[ 6] Repo URL                 : https://gitlab.com/zxq9/termifierg
[ 7] Website URL              : 
[ 8] Description              : Create, edit and convert JSON to Erlang terms.
[ 9] Search Tags              : ["json","eterms"]
[10] File associations        : [".json"]
Press a number to select something to change, or [ENTER] to continue.
(or "QUIT"): 7
... [snip] ...

The “update meta” command is interactive so I’ll spare you the full output, but if you followed the previous two tutorials you already know how this works.

After I’ve done that I need to increase the “patch” version number (the “Z” part of the “X.Y.Z” semver scheme). I can do this with the “verup” command, also run in the project’s base directory:

ceverett@okonomiyaki:~/vcs/termifierg$ zx verup patch
Version changed from 0.1.0 to 0.1.1.

And now time to re-package and put it into the realm. Again, since I’m the sysop this is super fast for me working alone:

ceverett@okonomiyaki:~/vcs$ zx submit otpr-termifierg-0.1.1.zsp 
ceverett@okonomiyaki:~/vcs$ zx approve otpr-termifierg-0.1.1
ceverett@okonomiyaki:~/vcs$ zx accept otpr-termifierg-0.1.1.zsp

And that’s that. It can immediately be run by anyone anywhere as long as they have ZX installed.

BONUS LEVEL!

“Neat, but what about the screenshot of it running?”

Up until now we’ve been launching code using ZX from the command line. Since Termifier GUI is a GUI program and usually the target audience for GUI programs is not programmers, yesterday I started on a new graphical front end for ZX intended for ordinary users (you know, people expert at things other than programming!). This tool is called “Vapor” and is still an ugly duckling in beta, but workable enough to demonstrate its usefulness. It allows people to graphically browse projects from their desktop, and launch by clicking if the project is actually launchable.

Vapor is like low-pressure Steam, but with a strong DIY element to it, as anyone can become a developer and host their own code.

I haven’t written the window manager/desktop registration bits yet, so I will start Vapor from the command line with ZX:

You’ll notice a few things here:

  • Termifier GUI’s latest version is already selected for us, but if we click that button it will become a version selector and we can pick a specific version.
  • Observer is listed, but only as a “virtual package” because it is part of OTP, not actually a real otpr package. For this reason it lacks a version selector. (More on this below.)
  • Vapor lacks a “run” button of its own because it is already running (ZX is similarly special-cased)

When I click Termifier’s “run” button Vapor’s window goes away and we see that the termifierg-0.1.1 package is fetched from Zomp (along with deps, if they aren’t already present on the system), built and executed. If we run it a second time it will run immediately from the local cache since it and all deps are already built.

When Termifier terminates Vapor lets ZX know it is OK to shutdown the runtime.

A special note on Observer and “Virtual Packages”

[UPDATE 2020-01-12: The concept of virtual packages is going away, observer will have a different launch method soon, and a rather large interface change is coming to Vapor soon. The general principles and function the system remain the same, but the GUI will look significantly different in the future — the above is the day-2 functioning prototype.]

When other programs are run by Vapor the main Vapor window is closed. Remember, each execution environment is constructed at runtime for the specific application being run, so if we run two programs that have conflicting dependencies there will be confusion about the order to search for modules that are being called! To prevent contamination Vapor only allows a single application to be run at once from a single instance of Vapor (you can run several Vapor instances at once, though, as each invocation of ZX creates an independent Erlang runtime with its own context and environment — the various zx_daemons coordinate locally to pick a leader, though, so resource contention is avoided by proxying through the leader). If you want several inter-related apps to run at once within the same Erlang runtime, create a meta-package that has the sole function of launching them all together with commonly defined dependencies.

Because Observer is part of OTP it does not suffer from dependency or environmental conflict issues, so running Observer is safe and the “run” button does just that: it runs Observer. Vapor will stay open while Observer is running, allowing you to pick another application to run, and you can watch what it is up to using Observer as a monitoring tool, which can be quite handy (and interesting!).

If you want to run an Erlang network service type application using Vapor while using Observer (like a chat server, or even a Zomp node) you should start Vapor using the zxh command (not just plain zx), because that provides an Erlang shell on the command line so you can still interact with the program from there. You can also run anything using plain old zx run, and when the target application terminates that instance of the runtime will shut down (this is why ZX application templates define applications as “permanent“).

Cool story, bro. What Comes Next?

The next step for this little bundle of projects is to create an all-encompassing Windows installer for Erlang, ZX and Vapor (so it can all be a one-shot install for users), and add a desktop registration feature to Vapor so that Erlang applications can be “installed” on the local system, registered with desktop icons, menu entries and file associations in FreeDesktop and Windows conformant systems (I’ll have to learn how to do it on OSX, but that would be great, too!). Then users could run applications without really knowing about Vapor, because authors could write installation scripts that invoke Vapor’s registration routines directly.

If I have my way (and I always get my way eventually) Erlang will go from being the hardest and most annoying language to deploy client-side to being one of the easiest to deploy client-side across all supported platforms. BWAHAHAHA! (I admit, maybe this isn’t a world-changing goal, but for me it would be a world-changing thing…)

2019.12.28 23:56

Starting a simple GUI project in Erlang with ZX

A few days ago I wrote a tutorial about how to create a CLI program in Erlang using a new code utility called ZX that makes launching Erlang a little bit more familiar for people accustomed to modern dynamic language tooling.

Today I want to do another one in the same spirit, but with a simple GUI program as the example.

In the previous tutorial I did a CLI utility that converts files containing JSON to files containing Erlang terms. It accepts two arguments: the input file path and the desired output file path.

Today we’ll do a slightly more interesting version of this: a GUI utility that allows hand creation/editing of both the input before conversion and the output before writing. The program will have a simple interface with just three buttons at the top: “Read File”, “Convert” and “Save File”; and two text editing frames as the main body of the window: one on the left with a text control for JSON input, and one on the right a text control for Erlang terms after conversion.

First things, first: We have to choose a name and create the project. Since we did “Termifier” with a package and module name “termifier” before, today we’ll make it called “Termifier GUI” with a package and appmod “termifierg” and a project prefix “tg_”. I’ve clipped out the creation prompt output for brevity like before, but it can be found here: zx_gui_creation.txt.

ceverett@okonomiyaki:~/vcs$ zx create project

### --snip snip--
### Prompts for project meta
### --snip snip--

Writing app file: ebin/termifierg.app
Project otpr-termifierg-0.1.0 initialized.
ceverett@okonomiyaki:~/vcs$

If we run this using ZX’s zx rundir command we see a GUI window appear and some stuff happen in the terminal (assuming you’re using a GUI desktop and WX is built into the Erlang runtime you’re using).

The default templated window we see is a GUI version of a “Hello, World!”:

If we try the same again with some command line arguments we will see the change in the text frame:

The output occurring in the terminal is a mix of ZX writing to stdout about what it is building and WX’s GTK bindings warning that it can’t find an optional style module (which isn’t a big deal and happens on various systems).

So we start out with a window that contains a single multi-line text field and accepts the “close window” event from the window manager. A modest, but promising start.

What we want to do from here is make two editable text fields side by side, which will probably require increasing the main window’s size for comfort, and adding a single sizer with our three utility buttons at the top for ease of use (and our main frame, of course, being derived from the wxEvtHandler, will need to connect to the button click events to make them useful!). The text fields themselves we probably want to make have fixed-width fonts since the user will be staring at indented lines of declarative code, and it might even be handy to have a more natural “code editor” feel to the text field interface, so we can’t do any better than to use the Scintilla-type text control widget for the text controls.

Now that we know basically what we want to do, we need to figure out where to do it! To see where to make these changes we need to take a little tour of the program. It is four modules, which means it is a far different beast than our previous single-module CLI program was.

Like any project, the best way to figure out what is going on is to establish two things:

  1. How is the code structured (or is there even a clear structure)?
  2. What is called to kick things off? (“Why does anything do anything?”)

When I go into termifierg/src/ I see some very different things than before, but there is a clear pattern occurring (though it is somewhat different than the common Erlang server-side “service -> worker” pattern):

ceverett@okonomiyaki:~/vcs$ cd termifierg/src/
ceverett@okonomiyaki:~/vcs/termifierg/src$ ls -l
合計 16
-rw-rw-r-- 1 ceverett ceverett 1817 12月 27 12:50 termifierg.erl
-rw-rw-r-- 1 ceverett ceverett 3166 12月 27 12:50 tg_con.erl
-rw-rw-r-- 1 ceverett ceverett 3708 12月 27 12:50 tg_gui.erl
-rw-rw-r-- 1 ceverett ceverett 1298 12月 27 12:50 tg_sup.erl

We have the main application module termifierg.erl, the name of which we chose during the creation process, and then we also have three more modules that use the tg_ prefix we chose during creation: tg_con, tg_gui and tg_sup. As any erlanger knows, anything named *_sup.erl is going to be a supervisor, so it is logical to assume that tg_sup.erl is the top (in this case the only) supervisor for the application. It looks like there are only two “living” modules, the *_con one, which seems short for a “control” module, and the *_gui one, which seems likely to be just the code or process that controls the actual window itself.

We know that we picked termifierg as the appmod for the project, so it should be the place to find the typical OTP AppMod:start/2 function… and sure enough, there it is: termifierg:start/2 is simply call to start things by calling tg_sup:start_link/0. So next we should see what tg_sup does. Being a supervisor its entire definition should be a very boring declaration of what children the supervisor has, how they depend on one another (order), and what restart strategy is being employed by that supervisor.

(Protip: magical supervision is evil; boring, declarative supervision is good.)

init([]) ->
     RestartStrategy = {one_for_one, 0, 60},
     Clients   = {tg_con,
                  {tg_con, start_link, []},
                  permanent,
                  5000,
                  worker,
                  [tg_con]},
     Children  = [Clients],
     {ok, {RestartStrategy, Children}}.

Here we see only one thing is defined: the “control” module called tg_con. Easy enough. Knowing that we have a GUI module as well, we should expect that the tg_con module probably links to the GUI process instead of monitoring it, though it is possible that it might monitor it or maybe even use the GUI code module as a library of callback functions that the control process itself uses to render a GUI on its own.

[NOTE: Any of these approaches is valid, but which one makes the most sense depends entirely on the situation and type of program that is being written. Is the GUI a single, simple interface to a vast and complex system underneath? Does each core control component of the system have its own window or special widget or tab to render its data? Are there lots of rendered “views” on the program data, or a single view on lots of different program data? Is it OK for updates to the GUI to block on data retrieval or processing? etc.]

Here we see a program that is split between interface code and operation code. Hey! That sounds a lot like the “View” and “Control” from the classic “MVC” paradigm! And, of course, this is exactly the case. The “Model” part in this particular program being the data we are handling which is defined by the Erlang syntax on the one hand and JSON’s definition on the other (and so are implicit, not explicit, in our program).

The tg_con process is the operation code that does things, and it is spawn_linking the interface that is defined by tg_gui. If either one crashes it will take the other one down with it — easy cleanup! For most simple programs this is a good model to start with, so we’ll leave things as they are.

The tg_gui process is the one that interfaces with the back-end. In this simple of a program we could easily glom them all together without getting confused, but if we add even just a few interesting features we would bury our core logic under the enormous amounts of verbose, somewhat complex code inherent in GUI applications — and that becomes extremely frustrating to separate out later (so most people don’t do it and their single-module-per-thingy WX code becomes a collection of balls of mud that are difficult to refactor later, and that stinks!).

Since we already know what we want to do with the program and we already proved the core code necessary to accomplish it in the previous tutorial, we can move on to building an interface for it.

This is what the final program looks like, using the same example.json from the CLI example:

At this point we are going to leave the blog and I will direct you instead to the repository code, and in particular, the commit that shows the diff between the original template code generated by ZX and the modified commit that implements the program described at the beginning of this tutorial. The commit has been heavily commented to explain every part in detail — if you are curious about how this Wx program works I strongly recommend reading the code commit and comments!

2019.12.25 23:04

Starting a simple CLI project in Erlang with ZX

Filed under: Computing,Science & Tech — Tags: , , , , , — zxq9 @ 23:04

Yesterday I wrote a post about a new tooling suite for developers and users that makes dealing with Erlang more familiar to people from other languages. Using the tool for packaging and deployment/launch makes writing and deploying end-user programs in Erlang non-mysterious as well, which is a great benefit as Erlang provides a wonderful paradigm for making use of modern overwhelmingly multi-core client systems.

It is still in beta, but works well for my projects, so I’ll leave a quick tutorial here that shows the basic flow of writing a simple CLI utility in Erlang using ZX.

In this example we’ll make a program that accepts two arguments: a path to a file with JSON in it and a path to a file where the data should be written back after being converted to Erlang terms.

To start a project we do zx create project and follow the prompts.
(The snippet below excludes the full output for brevity, but you can view the entire creation prompt log here: zx_cli_creation.txt.)

ceverett@okonomiyaki:~/vcs$ zx create project

### --snip snip--
### Prompts for project meta
### --snip snip--

Writing app file: ebin/termifier.app
Project otpr-termifier-0.1.0 initialized.
ceverett@okonomiyaki:~/vcs$ 

After the project is created we see a new directory in front of us called “termifier” (or whatever the project is named). We can execute this now just to make sure everything is going as expected:

ceverett@okonomiyaki:~/vcs$ ls
termifier
ceverett@okonomiyaki:~/vcs$ zx rundir termifier
Recompile: src/termifier
Hello, World! Args: []
ceverett@okonomiyaki:~/vcs$ zx rundir termifier foo bar baz
Hello, World! Args: ["foo","bar","baz"]

Ah! So we already have something that builds and runs, very similar to how an escript works, except that using ZX we can easily add dependencies from Zomp package realms and package and execute this program on any system in the world that has ZX on it via Zomp ourselves.

…not that we have any reason to deploy a “Hello, World!” program to the wider public, of course.

Notice here that the first time we run it we see a message Recompile: src/termifier. That means the module termifier is being compiled and cached. On subsequent runs this step is not necessary unless the source file has changed (the compiler detects this on its own).

Next lets search Zomp for the tag “json” to see if there are any packages that list it as a tag, and if there are any let’s get a description so maybe we can find a website or docs for it:

ceverett@okonomiyaki:~/vcs$ zx search json
otpr-zj-1.0.5
ceverett@okonomiyaki:~/vcs$ zx describe otpr-zj
Package : otpr-zj-1.0.5
Name    : zj
Type    : lib
Desc    : A tiny JSON encoder/decoder in pure Erlang.
Author  : Craig Everett zxq9@zxq9.com
Web     : https://zxq9.com/projects/zj/docs/
Repo    : https://gitlab.com/zxq9/zj
Tags    : ["json","json encoder","json decoder"]

Ah. Checking the website it is clear we can use this to decode JSON by simply calling zx:decode(JSON). Easy. So let’s add it to the project as a dependency and invoke it in src/termifier.erl:

ceverett@okonomiyaki:~/vcs$ cd termifier
ceverett@okonomiyaki:~/vcs/termifier$ zx set dep otpr-zj-1.0.5
Adding dep otpr-zj-1.0.5
ceverett@okonomiyaki:~/vcs/termifier$ cd src
ceverett@okonomiyaki:~/vcs/termifier/src$ vim termifier.erl

Inside termifier.erl we can see the templated code for start/1:

start(ArgV) ->
    ok = io:format("Hello, World! Args: ~tp~n", [ArgV]),
    zx:silent_stop().

Lets change it so it does what we want (note here I’m going a bit further and writing a function write_terms/2 based on an older post of mine — this performs the inverse procedure of file:consult/1):

start([InPath, OutPath]) ->
    {ok, Bin} = file:read_file(InPath),
    {ok, Terms} = zj:decode(Bin),
    ok = write_terms(OutPath, Terms),
    zx:silent_stop();
start(_) ->
    ok = io:format("ERROR: Two arguments are required."),
    zx:silent_stop().


write_terms(Path, Terms) when is_list(Terms) ->
    Format = fun(Term) -> io_lib:format("~tp.~n", [Term]) end,
    Text = lists:map(Format, Terms),
    file:write_file(Path, Text);
write_terms(Path, Term) ->
    write_terms(Path, [Term]).

Note that we are calling zj:decode/1 in the body of start/1 above, knowing that ZX will find it for us and configure the environment at execution time. And now let’s give it a go!

ceverett@okonomiyaki:~/vcs$ zx rundir termifier example.json example.eterms
Recompile: src/zj
Recompile: src/termifier
ceverett@okonomiyaki:~/vcs$ cat example.json 
{
    "fruit": "Apple",
    "size": "Large",
    "color": "Red"
}
ceverett@okonomiyaki:~/vcs$ cat example.eterms 
{"color" => "Red","fruit" => "Apple","size" => "Large"}.

From here I could run zx package termifier, submit it to a realm (either the default public realm, or a new one I can create and host myself by doing zx create realm and then zx run zomp), and anyone could then run the command zx run termifier [in path] [out path] and ZX will take care of finding and building the necessary packages and launching the program.

That’s all there is to it. ZX’s template for CLI applications is quite minimal (as you can see) and is more similar to an escript than a more traditional OTP-style, supervised Erlang application. ZX has templates, however, for full-blown OTP applications, GUI code (structured also in the OTP paradigm), minimalist CLI programs like we see above, pure library code, and escripts (sometimes escript is exactly the tool you need!).

Happy coding!

2018.07.11 17:38

Erlang: Getting Started Without Melting

There are two things that might be meant when someone references “Erlang”: the language, and the environment (the EVM/BEAM and OTP). The first one, the language part, is actually super simple and quick to learn. The much larger, deeper part is learning what the BEAM does and how OTP makes your programs better.

It is clear that without an understanding of Erlang we’re not going to get very far in terms of understanding OTP and won’t be skilled enough to reliably interact with the runtime through a shell. So let’s forget about the runtime and OTP for a bit and just aim at the lowest, most common beginners’ task in coding: writing a script that tells me “Hello, World!” and shows whatever arguments I pass to it from the command line:

#! /usr/bin/env escript

% Example of an escript
-mode(compile).

main(Args) ->
    ok = io:setopts([{encoding, unicode}]),
    ok = io:format("Hello, world!~n"),
    io:format("I received the args: ~tp~n", [Args]).

Let’s save that in a file called e_script, run the command chmod +x e_script to make it executable, and take a look at how this works:

ceverett@takoyaki:~$ ./e_script foo bar
Hello, world!
I received the args: ["foo","bar"]
ceverett@takoyaki:~$

Cool! So it actually works. I can see a few things already:

  1. I need to know how to call some things from the standard library to make stuff work, like io:format/2
  2. io:setopts([{encoding, unicode}]) seems to makes it OK to print UTF-8 characters to the terminal in a script
  3. An escript starts execution with a traditional main/1 function call

Some questions I might have include how or why we use the = for both assignment and assertion in Erlang, what the mantra “crash fast” really means, what keywords are reserved, and other issues which are covered in the Reference Manual (which is surprisingly small and quick to read and reference).

An issue some newcomers encounter is that navigating an unfamiliar set of documentation can be hard. Here are the most important links you will need to know to get familiar and do useful things with the sequential language:

This is a short list, but it is the most common links you’ll want to know how to find. It is also easy to pull up any given module for doing a search for “erlang [module name]” on any search engine. (Really, any of them.)

In the rare case that erlang.org is having a hard time I maintain a mirror of the docs for various Erlang release versions here as well: http://zxq9.com/erlang/

Start messing with sequential Erlang. Don’t worry about being fancy and massively concurrent or maximizing parallelization or whatever — just mess around at first and get a feel for the language using escript. It is a lot of fun and makes getting into the more fully encompassing instructional material much more comfortable.

2018.06.27 00:06

Erlang: ZJ docs

Filed under: Computing — Tags: , , , , , — zxq9 @ 00:06

Docs for the ZJ Erlang JSON encoder/decoder are now available here:

http://zxq9.com/projects/zj/docs/

The binary_encode/1 function will probably be live tomorrow, along with a proper v1.0 release.

2018.06.26 14:52

Your tests don’t tell you what you think they do

Filed under: Computing,Science & Tech — Tags: , , , , , , , , — zxq9 @ 14:52

Yesterday I wrote a tiny JSON encoder/decoder in Erlang. While the Erlang community wasn’t in dire need of yet another JSON parser, the ones I saw around do things just a tiny bit differently than I want them to and writing a module against RFC-8259 isn’t particularly hard or time consuming.

Someone commented on (gasp!) the lack of tests in that module. They were right. I just needed the module to do two things, the code is boring, and I didn’t write tests. I’m such a rebel! Or a villain! Or… perhaps I’m just someone who values my time.

Maybe you’re thinking I’m one of those coding cowboys who goes hog wild on unsafe code! No. I’m not. Nothing could be further from the truth. What I have learned over the last 30 years of fiddling about with software is that hand-written tests are mostly a waste of time.

Here’s what happens:

  1. You write a new thingy.
  2. You throw all the common cases at it in the shell. It seems to work. Great!
  3. Being a prudent coder you basically translate the things you thought to throw at it in the shell into tests.
  4. You hook it up to an actual project you’re using somewhere — and it breaks!
  5. You fix the broken bits, and maybe add a test for whatever you fixed.
  6. Then other people start using it in their projects and stuff breaks quite a lot more ZOMG AHHH!

Where in here did your hand-written tests help out? If you write tests to define the bounds of the problem before you actually wrote your functions then tests might help out quite a lot because they deepen your understanding of the problem before you really tackle it head-on. Writing tests before code isn’t particularly helpful if you already thoroughly understand the problem and just need something to work, though.

When I wrote ZJ yesterday I needed it to work in the cases that I care about — and it did, right away. So I was happy. This morning, however, someone else decided to drop ZJ into their project and give it a go — and immediately ran into a problem! ZJ v0.1.0 returns an error if it finds trailing commas in JSON arrays or objects! Oh noes!

Wait… trailing commas aren’t legal in JSON. So what’s the deal? Would tests have discovered this problem? Of course not, because hand-written tests would have been bounded by the limits of my imagination and my imagination was hijacked by an RFC all day yesterday. But the real world isn’t an RFC, and if you’ve ever dealt with JSON in the wild that you’re not generating you’ll know that all sorts of heinous and malformed crap is clogging the intertubes, and most of it sports trailing commas.

My point here isn’t that testing is bad or always a waste of time, my point is that hand-written tests are themselves prone to the exact same problems the code being tested is: you wrote them so they carry flaws of implementation, design and scope, just like the rest of your project.

“So when is testing good?” you might ask. As mentioned earlier, those cases where you are trying to model the problem in your mind for the first time, before you’ve written any handling code, is a great time to write tests for no other reason than they help you understand the problem. But that’s about as far as I go with hand-writing tests.

The three types of testing I like are:

  • type checks
  • machine generated (property testing)
  • real-world (user testing)

A good type checker like Dialyzer (or especially ghc’s type system, but that’s Haskell) can tell you a lot about your code in very short order. It isn’t unusual at all to have sections of code that are written to do things that are literally impossible, but you wouldn’t know about until much later because, due simply to lack of imagination, quite often hand-written tests would never have executed the code, or not in a way that would reveal the structural error.
Typespecs: USE THEM

Good property testing systems like PropEr and QuickCheck generate and run as many tests as you give them time to (really, it is just constrained by time and computing resources), and once they discover breakages can actually fuzz the problem out to pinpoint the exact failing cases and very often indicate the root cause pretty quickly. It is amazing. If you ever experience this you’ll never want to hand write tests again.
Property Testing: USE IT

What about user testing? It is simply necessary. You’ll never dream up the insane stuff to try that users will, and neither will a property-based test generation system. Your test and development environment will often bear little resemblance to your users’ environments (a few weirdos out there still use Windows!), the things you might think to store in your system will rarely look anything like the sort of stuff they will wind up storing in it (you were thinking text, they were thinking video), and the frequency of operation that you assumed might look realistic will almost never been anywhere close to the mark (your one-off utility program that you assumed would run in isolation initiated by a user command in ~/bin/ may become the core part of a massively parallelized service script executed every minute by a cron job running as root).
Your Users: COMMUNICATE WITH THEM

Ultimately, hand-written tests tend to reveal a lot more about the author of the tests than the status of the software being tested.

2018.06.25 21:03

Tiny strings-as-strings JSON in portable Erlang

There are several JSON libs for Erlang at this point, and as there is no correct mapping between JSON types and Erlang types, all make different tradeoffs that either work or don’t for your project. Beyond that, various interface and implementation differences exist due to the tradeoffs inherent in manipulating elements of the Black Tongue known as lolscript:

  • Accept values to encode as magic tagged tuples so you can specify exactly what you want VS being ambiguous
  • Never allow “naked” values (everything must be in a list/array or a map or a [whatever]) VS “hanging” values
  • Treat all strings ever as binaries because “strings are big” VS treating all strings (and binaries) as strings because strings are easy to manipulate (io_lists…)
  • Decode JSON “objects” as proplists VS decode JSON objects to dicts or maps VS add an “options” argument to the decode function
  • Encode and decode values various ways based on optional switches VS “sane defaults” (aka “works for me”)
  • Achieve lolspeed via NIFs and only work on *nix VS maintain portability via pure Erlang
  • etc.

No combination is correct for every situation, hence the proliferation of libraries. In addition to proliferation, something as simple as what is described by RFC-8259 shouldn’t require a 20k LoC dependency to manage, at least not in Erlang of all languages.

The general strings-as-strings + portability tradeoffs were made by mochiweb years ago, with mochijson2 being the go-to JSON parser for lots of projects. Now that “tuple calls” have finally been retired after years of obsolescence and deprecation, mochijson2 is finally giving up the ghost as well (as it was based on tuple calls). As a replacement that makes mostly the same tradeoffs but is arguably simpler, I wrote a single-module JSON encoder/decoder lib. It treats all strings as strings, is in pure Erlang, and is utterly boring in how simple the code is. Nothing magical to see. At all. So don’t get excited.

If you need to read things in and read things out, in JSON, and don’t really care about lolspeed but want to understand what is happening, then ZJ is for you: ZJ project @ gitlab

Note that if you have roughly the same requirements but you want to make the strings-as-binaries tradeoff then JSX is the lib for you.

 

2018.02.1 23:40

Confounding Beginner Question: What is an Erlang Atom and Why is it Useful?

Like other Erlangers, I tend to take the atom data type for granted. Coming from another language, however, you might be puzzled at why we have all these little strings that aren’t really strings.

The common definition you’ll hear most frequently is something like:

An atom is a label. Its only meaning is itself.

Well, that’s true, but that also sounds a bit useless to someone coming from Python or R or JavaScript or whatever. So let’s break that down: what is a “label” useful for in programs?

  • Variable names are labels.
  • Function names are labels.
  • Module names are labels.
  • The strings you use as keys in a key/value data structure are labels.
  • The enums and label macros you might use in C for semantically significant internal values are almost exactly like atoms

OK, so we use labels all the time, why don’t any of those other languages have atoms, though? Let’s examine those last two reasons for a moment for a hint why.

In Python strings are objects and while building them is expensive, hashing them can be done ahead of time as a cached operation. This means comparing two strings of arbitrary length for equality is extremely cheap, because it is reduced to a large integer comparison for equality. This is not true in, say, C or Erlang or Lisp unless you build your own data structure to carry around the pre-hashed data. In Python it is simple enough to say:

if 'foo' in some_dict:
  # stuff
else:
  # other stuff

In C, however, string comparison is a bit of a hassle and dealing with string data in a cross-platform environment at all can be super annoying depending the age of the systems you might be interacting with or running/building your code on. In Erlang the syntax of string comparison is super simple, but the overhead is not pre-paid like in Python. So what is the alternative?

We use integer values to represent keys that are semantically meaningful to the program at the time it is written. But integers are hard to remember, so instead of having magic numbers floating all around the place we typically have semantically significant integer values aliased from a text label as a macro. This is helpful so that I don’t have to remember the meaning of code like

if (condition == 42) launch_missiles();
if (condition == 86) eat_kittens();

Instead I can write code like:

#define UNDER_ATTACK    42
#define VILE_UNDERBEAST 86

if (condition == UNDER_ATTACK)    launch_missiles();
if (condition == VILE_UNDERBEAST) eat_kittens();

It is extremely common in programs to have variables or arguments like condition in the above example. It doesn’t matter whether your language has matching (like Erlang, Rust, logic languages, etc.) or uses explicit conditionals like the fake C example above — there will always be a huge number of micro datatypes that carry great semantic significance within your program and only within your program and it is as useful to be able to label these enumerated values in a way that the human coders can understand and remember as it is useful for the computer to be able to compare them as simple integers instead of going to the trouble of string comparison every time your code needs to make a decision (because string comparison entails an arbitrarily long sequence of integer comparisons every single time you compare two strings).

In C we use those macros like above (well, not always; C actually does have super convenient enums that work a lot like atoms, but didn’t when I started using it as a kid in the stone age). In Erlang we just use an atom right there in place. You don’t need a declaration or definition anywhere, the runtime just keeps track of these things for you.

Underneath the hood Erlang maintains a running table of atom label values and translates them to integer values on the way into the system and on the way out of the system. The integer each atom actual resolves to is totally unimportant to you, so Erlang abstracts that detail away, but leaves the machine comparing integer values instead of doing full-string comparisons all over the place.

“But Erlang maps don’t do string comparisons on keys!” you might say.

And indeed, you would be right. Because map keys might be any arbitrary value each key is hashed on the way in, and every time keys are compared the comparing term is hashed the same way, so the end comparison is super fast, but we have to hash the input value first for it to mean anything. With atoms, though, we have a shortcut, because we already know they are both unambiguous integer values throughout the system, and this is a slight win over having to hash first before comparing keys.

In other situations where the comparison values cannot be hashed ahead of time, like function-head matching, however, atoms are a huge win over string comparisons:

-module(atoms).
-export([foo/1, bar/1]).

foo("Some string value that I don't really recall") ->
    {ok, 1};
foo("Some string value that I don't really care about") ->
    {ok, 2};
foo("Where is my cheeseburger?") ->
    {ok, 3};
foo(_) ->
    {error, wonky_input}.

bar(dont_recall) ->
    {ok, 1};
bar(dont_care) ->
    {ok, 2};
bar(cheeseburger) ->
    {ok, 3};
bar(_) ->
    {error, wonky_input}.

I’ve slowed the clockspeed of the system so that we can notice any difference here in microseconds.

1> timer:tc(fun() -> atoms:foo("Some string value that I don't really care about.") end).
{16,{error,wonky_input}}
2> timer:tc(fun() -> atoms:foo("Where is my cheeseburger?") end).
{13,{ok,3}}
3> timer:tc(fun() -> atoms:foo("arglebargle") end).
{12,{error,wonky_input}}
4> timer:tc(fun() -> atoms:bar(dont_care) end).
{9,{ok,2}}
5> timer:tc(fun() -> atoms:bar(cheeseburger) end).                                      
{10,{ok,3}}
6> timer:tc(fun() -> atoms:bar(arglebargle) end).                                        
{10,{error,wonky_input}}

See what happened? The long string that varies only at the tail end from two options in the function head takes 16 microsecond to compare and return a value. The string that differs at the head is evaluated as a bad match for the first two options the moment the very first character is compared. The total mismatch is our fastest return because that string never need be traversed even a single time to know that it doesn’t match any of the available definitions of foo/1. With atoms, however, we see a pretty constant speed of comparison. That speed would not change at all even if the atoms were a hundred characters long in text, because underneath they are all just integer values.

Now take a look back and consider the return values defined for foo/1 and bar/1. They don’t return naked values, they return pairs where the first member is an atom. This is a pretty common technique in Erlang when writing either a library intended for 3rd party use or when defining functions that have side-effecty operations that might fail (here we have pure functions, but I’m just using this as an example). Remember, the equal sign in Erlang is both an assignment operator and an assertion operator, when calling a function that nests its return values you have the freedom to decide whether to crash the current process on an unexpected value or to handle the “error” (in which case for your program it becomes an expected condition and not an exception).

blah(Condition) ->
    {ok, Value} = foo(Condition),
    do_stuff(Value).

The code above will crash if the tuple {error, wonky_input} is returned, because the expected atom 'ok' does not match the actually returned atom ‘error’.

bleh(Condition) ->
   case foo(Condition) of
       {ok, Value}          -> do_stuff(Value);
       {error, wonky_input} -> get_new_condition()
   end.

The code above now does not crash on that error return value and instead moves on to get another condition to try out, because the error tuple matches one of the case conditions that is defined as a return value. All this can happen really fast because atoms comparisons are really integer comparisons, and that means we save a ton of processor time (and space) by avoiding string/list or binary comparisons all over the place.

In addition to atoms being a much nicer and dramatically more flexible version of global enumerated types that let us write code in a more natural style that uses normal-language labels for program semantics, it turns out that function and module names are also atoms. This is a really nice feature in itself, because it allows us to write highly dynamic code with a lot less confusion about what types both sides of a call needs to be as well as making the code easier to read. I can even implement my own version of apply/3:

my_apply(Module, Function, Args) ->
    Module:Function(Args).

Of course, there is a whole pile of reasons why you will never want to actually write a function like this in a real program, but that’s the sort of power we have without doing any type casting magic, introspection, or on-the-fly modification of our program, references or memory space.

Once you get used to using atoms and matching you’ll really start to miss them in other languages and wonder how you ever got along without them. Now run off and start writing some code to practice thinking with atoms. They will become natural to you before the day is out.

2018.01.31 18:27

James Mickens: Life as a Developer

Below is a somewhat whimsical talk given by James Mickens last year titled Life as a Developer: My Code Does Not Work Because I am a Victim of Complex Societal Factors That are Beyond My Control.

(If you recall, Mickens is the guy who did the insane/awesome Monitorama talk about the cloud a few years ago.)

As usual for dear begoateed James it is full of silly digressions, but the theme is quite an important one today: unnecessary complexity is increasing and that trend will conspire with our own limited capacity for comprehension, information retention, research of the new and obscure, and interpersonal communication to ensure that if we keep going as we are the whole world will eventually succumb to entropy and leave us in a big ball of digital hellfire.

Life As A Developer: My Code Does Not Work Because I Am A Victim Of Complex Societal Factors That Are Beyond My Control…. from NDC Conferences on Vimeo.

Older Posts »

Powered by WordPress