The Intellectual Wilderness There is nothing more useless than doing efficiently that which should not be done at all.

2020.11.25 14:12

Erlang: FizzBuzz in Python vs Erlang — a discussion about conditionals

Filed under: Computing — Tags: , , , , , — zxq9 @ 14:12

I’ve had a few discussions with beginners over the last few months that often come to center around not really understanding the conditional branching constructs in Erlang, so I decided to do a video on it using “FizzBuzz” as the central example.

The traditional if/else if/else boolean paradigm is quite different from Erlang’s concept of matching and guards in case and function head matches and an if construct that is really more appropriate for range checks than general boolean branching. Hopefully I was able to explain at least a little bit about why Erlang’s matching and guards are a much more flexible method for writing conditionals once you grow accustomed to thinking in these terms.

2020.11.22 19:20

Erlang: Building a Telnet Chat Server from Scratch Using ZX

Filed under: Computing — Tags: , , , , , , , — zxq9 @ 19:20

A few weeks ago I made a two-part video discussion about building a telnet chat server from scratch using ZX and forgot to post any prose reference to it. (Most people are following my blog RSS, not my (extremely tiny) video channels.)

The resulting project is called “Trash Talk” and it has a repo on GitLab here. The license is MIT and is quite simple to hack around on, so have fun.

Part 1

The first video is a bit slower paced than the second. I cover:

  • What ZX templates when you do zx create project and select the “Traditional Erlang Application” project option
  • How everything fits together and works
  • Discussion about why things are structured the way they are in OTP applications
  • Demonstrate a little bit of command implementation in the telnet server (mostly to show where in the code you would do that).

This video is a lot less exciting than the second one because there aren’t a lot of jazzy features demonstrated, but it is probably the one you want to watch once now if you’re new to Erlang and a second time a few weeks later once you’ve written a few projects and stubbed your toes a few times (a second viewing brings different insights, not because the video changes but because you change through your experiences).

Part 2

The second video is the more fun one because the initial explanation that covers the critical question of “Why does anything do anything?” has already been covered in the first one, and while you might find the first video interesting, it isn’t as exciting as this second one where you get to see features that do stuff that might be more relevant to problems you have in your real-world projects get implemented. In this one I cover:

  • Checking the state of the running system using observer
  • The difference between zx and zxh when running your code in development
  • The “Service ⇒ Worker Pattern” (aka “SWP”) and how ZX can help you drop one in via zx template swp
  • One way to implement text commands in a structured way, with categories of commands indicated by prefixes (this seems elementary and very IRC-ish at first, but in command protocols a direct parallel of this often happens at the byte level — so think on that)
  • How to separate the “clients” concept as a service from the “channels” concept as a service, and discuss how that idea extends to much more complex systems
  • A bit more useful version of a “service manager” process in the form of the chan_man
  • And much much more! (advertising people from the 1980’s tell me I should always say that instead of “etc.”)

These aren’t super jazzy, but they get the job done. Hopefully they give some new Erlanger out there a leg-up on the problem of going from a “Hello, World!” escript to a proper OTP application.

2020.09.25 10:48

Excerpts from Chat: Discovering Structure and Grokking the True Nature of a Problem

This is an excerpt from a private chat with someone about code and I think it expresses something universal about programming, mathematics and engineering or any discipline where iterative development is necessary to manage complexity and discover simplicity.

…about a bit of code where I haven’t quite grasped what needs to happen, or the core essence of what problem the computer is actually solving for me (because this is sometimes distinct from the real world or abstract world where the problem really exists) — and so my code is convoluted and annoying and I can’t figure out how to untangle it without lots of extra complexity.

And then one day I make a change to the way the data is represented (it usually manifests with a data representation change, not always, but usually) and I refactor the code to accommodate that and… VOILA! It just sorts itself out and I find all these ways to simplify and it all makes so much sense.

That kind of code, especially if it is functional, is sort of annoying for people to read sometimes. They look at the code and see this elegant solution that tightly represents the program in a novel way and think “Ah, this is great. I should learn FP!” and then when they themselves tackle a difficult problem they don’t understand the nature of and start writing their own ball of mud they go back and look at that elegant, ideal example that inspired their current task and suddenly feel like a child looking up at the roof of the Sistine Chapel wondering how the hell Michelangelo managed to paint all that with a 12-meter-long paint brush.

They’ve never seen or imagined all the scaffolding that was once in place.

— Me, just now

I am quoting myself above, but it doesn’t really matter who wrote it. This expression came out well and I thought it was worth saving somewhere the world can see it (still following Joe Armstrong’s admonishment that I should publish and share too much and let search engines and individuals sort things out).

2020.09.23 17:57

Erlang: [video] The GUI experience on Windows with ZX and Vapor

Filed under: Computing — Tags: , , , , , , , — zxq9 @ 17:57

I’ve written and spoken a bit about how ZX makes it easy to create, distribute and launch templated GUI projects in Erlang. I also showed how the (still ugly) GUI program launcher Vapor can make it easy for non-technical desktop users to use those programs.

So far I’ve been doing my examples on Linux where everything works in a familiar way and the terrain is fairly well known to us as developers, so in this video I want to show a bit about how things feel to Windows users who run client-side Erlang via Vapor using the standard Erlang runtime installation provided by Erlang Solutions and point out a few things that I find significant or can be improved in the experience.

If you have any insights into the issues highlighted in the video or have ideas about cross platform client development in general please let me know!

2020.09.22 14:23

Erlang: [Video redo!] Creating and running GUI apps with ZX

Filed under: Computing — Tags: , , , , , — zxq9 @ 14:23

I had a little bit of time to re-make a video on how to use ZX to create GUI applications. Hopefully the video demonstrates the basic use case well enough to make the purpose of the tool itself obvious.

2020.09.20 21:01

Erlang: [Video] Creating and running GUI apps with ZX

Filed under: Computing — Tags: , , , , , , , , — zxq9 @ 21:01

I had a little bit of time to make a video on how to use ZX to create GUI applications, but not enough time to do any post processing. Hopefully the video demonstrates the basic use case well enough to make the purpose of the tool itself obvious. (The audio isn’t great — hopefully I’ll have to either go back and dress that up a bit or make a better version of this video.)

2020.09.11 16:05

Hardware Development VS Software Development Budget Allocation

Filed under: Computing — Tags: , , , , , , — zxq9 @ 16:05

It is funny to me that hardware engineers are able to test the bejeezus out of essentially solved problems like how many keypresses a keyboard can take before mean operations to failure and document every minute aspect of their development, but software developers are pretty much prohibited from being given the time to document code.

In fact, it strikes me that hardware engineers’ actual job is to produce a specification from which hardware can be built, nearly always by people other than the engineers developing the product. That is to say, the product of the engineers’ labor is documentation.

The software developers, on the other hand, have the ability to document their product in coordination with its development, but are almost always forbidden from taking the time to do so, despite that documentation effort costing far less than the process hardware manufacturers have to go through to tool up a production line for a physical product.

This is a pretty insane state of affairs. If I could only show the public some of the chewed-to-crap, undocumented, wazoo code I’ve seen in production on closed source projects…

2020.09.9 12:51

Erlang: Barnsley’s Fern

Filed under: Computing — Tags: , , , , — zxq9 @ 12:51

A mathematician friend of mine asked me the other day whether we used many techniques from fractal theory in game development. I told her that I didn’t think so, at least not formally. She asked me if I had ever implemented “Barnsley’s Fern” (Wikipedia) and of course I never had. So she asked me to implement it and tell her what I thought.

Her plan seems to have been to get me to recognize that we do use techniques derived from fractal theory all over the place by implementing a famous fractal by hand myself. The plan worked: it was immediately obvious to me that Barnsley’s Fern makes use of a technique that is central to the way random map generators work in game development, but I had never realized this was actually from “fractal theory”, having stumbled on the technique myself because it was a useful shortcut to making game maps that were interesting and felt natural(ish).

Here is the interesting part of the code:

The interesting part about that is the fact that the plotting of points is actually a random function, not a concretely defined rotation of an existing pattern. The constants involved in the fern1-4 functions are found here:

My Barnsley’s Fern implementation is available on gitlab and can be run using either ZX or Vapor if you have ZX on your system. The most recent version as of this post, 0.1.2, uses OpenGL to render the image and seems to work much more reliably across platforms than the previous implementation using a WX graphics context (some versions of WX don’t like the way I drew the points). In Vapor you can select the version with the version drop box if you want to see the WX implementation:

Or you can run it directly from ZX using the command line with:
zx run barnsley_fern

Here is what the OpenGL version looks like at 100001 iterations:

The OpenGL interface allows you to rotate and move the image around a bit, though in v0.1.2 the center of rotation is a bit off center. Also, if you have more than a few hundred thousand points it becomes cumbersome to render repeatedly in animation because it is actually re-plotting each frame (I didn’t go to the trouble to plot the points to a buffer or texture and simply rotate that instead).

The previous version looks like this at the same number of iterations:

The coordinate systems are different for the two implementations, hence the difference in the direction of the curve.

Hanging around mathematicians lately has made me realize that there is a tremendous amount of higher math involved in a lot of what we do in programming, but that the mathematicians rarely talk to the computer science people, and computer science people are living on their own little planet with little connection to what actual developers are doing in industry (all of us little people just “trying to make it go”). Further, the semantic map of what words are used to mean what in which context is an absolute mess, so it takes some patience and explanation to understand what the other person is saying half the time if you are talking outside your tribe.

Keep the patience! Explain exhaustively! Listen carefully! It is so much more interesting when you have a chance to confer with people outside your tribe!

2020.02.3 15:19

X-Y Problems

Filed under: Computing,Science & Tech,Society — Tags: , , , , , — zxq9 @ 15:19

People obsess about their X-Y problems to the point of ignoring accepted wisdom, plugging their ears to the deafening silence of the solution’s instructive whisper, picking themselves up as hard as they can by their own knees and wondering why they can’t fly.

They then run off and formalize their wrong solution as a PR into a core project.

If core maintainers aren’t mindful they’ll incorporate these disturbances into a previously still space, and if they are indelicate they will piss off the misguided (but industrious) boob who made the PR who is already by this point fanatically dedicated to his wrong solution and the idea that nobody “gets it” but him.

Ah, another day at the Bazaar.

2020.01.9 00:26

Packaging and Distributing/Deploying Erlang GUI apps with ZX

Filed under: Computing — Tags: , , , , , , — zxq9 @ 00:26

In the last two posts I wrote walkthroughs for how to create new CLI and GUI apps in Erlang from scratch using a tool called ZX. Today I want to show how to package apps and publish them using Zomp (the package distribution system) and get them into the hands of your users with minimal fuss.

To understand how packages are distributed to users (the “why does anything do anything?” part of grokking ZX/Zomp), one must first understand a bit about how Zomp views the world.

Packages in the system are organized by “realms”. A code realm is like a package repository for a Linux distribution. The most important thing about realm/repository configuration is that you know by each package’s signature whether it really came from the it claims as its origin. Just like Linux repositories, anyone can create or host a Zomp realm, and realms can be mirrored.

(As you will see in a future tutorial, though, administration and and mirroring with Zomp is way easier and flexible than traditional Linux repositories. As you will see below, packaging with ZX is just a single command — ZX already knows everything it needs from the zomp.meta file.)

In this example I am going to put the example CLI project, Termifier GUI (a toy example app that converts JSON to Erlang terms) into the default FOSS realm, “otpr”. Because I am the sysop I have packaging and maintenance permissions for every package in the realm, as well as the sole authority to add projects and “accept” a package into the write-only indexes (packagers have “submit” authority, maintainers have “review”, “reject” and “approve” authorities).

[Note: The indexes are write only because dependencies in ZX are statically defined (no invisible updates) and the indexes are the only complete structure that must be mirrored by every mirroring node. Packages are not copied to new mirrors, they are cached the first time they are requested, with mirror nodes connected in a tree instead of a single hub pushing to all mirrors at once. This makes starting a new mirror very light weight, even for large realms, as no packages need to be copied to start (only the realm’s update history, from which the index is constructed), and packages in high demand experience “trickle down” replication, allowing mirrors to be sparse instead of complete. Only the “prime node” for a given realm must have a complete copy of everything in that particular realm. Nodes can mirror an arbitrary number of realms, and a node that is prime for one or more realms may mirror any number of others at the same time, making hosting of private project code mixed with mirrored public FOSS code a very efficient arrangement for organizations and devops.]

In the original Termifier GUI tutorial I simply created it and launched it from the command line using ZX’s zx rundir [path] and zx runlocal commands. The package was implicitly defined as being in the otpr realm because I never defined any other, but otpr itself was never told about this, so it merely remained a locally created project that could use packages hosted by Zomp as dependencies, but was not actually available through Zomp. Let’s change that:

ceverett@okonomiyaki:~/vcs$ zx add package otpr-termifierg

Done. That’s all there is to it. I’m the sysop, so this command told ZX to send a signed instruction (signed with my sysop key) to the prime node of otpr to create an entry for that package in the realm’s index.

Next we want to package the project. Last time we messed with it it was located at ~/vcs/termifierg/, so that’s where I’ll point ZX:

ceverett@okonomiyaki:~/vcs$ zx package termifierg/
Packaging termifierg/
Writing app file: ebin/termifierg.app
Wrote archive otpr-termifierg-0.1.0.zsp

Next I need to submit the package:

ceverett@okonomiyaki:~/vcs$ zx submit otpr-termifierg-0.1.0.zsp

The idea behind submission is that normally there are two cases:

  1. A realm is a one-man show.
  2. A realm has a lot of people involved in it and there is a formal preview/approval, review/acceptance process before publication (remember, the index is write-only!).

In the case where a single person is in charge rushing through the acceptance process only involves three commands (no problem). In the case where more than one person is involved the acceptance of a package should be a staged process where everyone has a chance to see each stage of the acceptance process.

Once a package has been submitted it can be checked by anyone with permissions on that project:

ceverett@okonomiyaki:~/vcs$ zx list pending otpr-termifierg
0.1.0
ceverett@okonomiyaki:~/vcs$ zx review otpr-termifierg-0.1.0
ceverett@okonomiyaki:~/vcs$ cd otpr-termifierg-0.1.0
ceverett@okonomiyaki:~/vcs/otpr-termifierg-0.1.0$ 

What the zx review [package_id] command does is download the package, verify the signature belongs to the actual submitter, and unpacks it in a directory so you can inspect it (or more likely) run it with zx rundir [unpacked directory].

After a package is reviewed (or if you’re flying solo and already know about the project because you wrote it) then you can “approve” it:

ceverett@okonomiyaki:~/vcs$ zx approve otpr-termifierg-0.1.0

The if the sysop is someone different than the packager then the review command is actually necessary, because the next step is re-signing the package with the sysop’s key as a part of acceptance into the realm. That is, the sysop runs zx review [package_id], actually reviews the code, and then once satisfied runs zx package [unpacked_dir] which results in a .zsp file signed by the sysop. If the sysop is the original packager, though, the .zsp file that was created in the packaging step above is already signed with the sysop’s key.

The sysop is the final word on inclusion of a package. If the green light is given, the sysop must “accept” the package:

ceverett@okonomiyaki:~/vcs$ zx accept otpr-termifierg-0.1.0.zsp

Done! So now let’s see if we can search the index for it, maybe by checking for the “json” tag since we know it is a JSON project:

ceverett@okonomiyaki:~/vcs/termifierg$ zx search json
otpr-termifierg-0.1.0
otpr-zj-1.0.5
ceverett@okonomiyaki:~/vcs/termifierg$ zx describe otpr-termifierg-0.1.0
Package : otpr-termifierg-0.1.0
Name    : Termifier GUI
Type    : gui
Desc    : Create, edit and convert JSON to Erlang terms.
Author  : Craig Everett zxq9@zxq9.com
Web     : 
Repo    : https://gitlab.com/zxq9/termifierg
Tags    : ["json","eterms"]

Yay! So we can now already do zx run otpr-termifierg and it will build itself and execute from anywhere, as long as the system has ZX installed.

I notice above that the “Web” URL is missing. The original blog post is as good a reference as this project is going to get, so I would like to add it. I do that by running the “update meta” command in the project directory:

ceverett@okonomiyaki:~/vcs/termifierg$ zx update meta

DESCRIPTION DATA
[ 1] Project Name             : Termifier GUI
[ 2] Author                   : Craig Everett
[ 3] Author's Email           : zxq9@zxq9.com
[ 4] Copyright Holder         : Craig Everett
[ 5] Copyright Holder's Email : zxq9@zxq9.com
[ 6] Repo URL                 : https://gitlab.com/zxq9/termifierg
[ 7] Website URL              : 
[ 8] Description              : Create, edit and convert JSON to Erlang terms.
[ 9] Search Tags              : ["json","eterms"]
[10] File associations        : [".json"]
Press a number to select something to change, or [ENTER] to continue.
(or "QUIT"): 7
... [snip] ...

The “update meta” command is interactive so I’ll spare you the full output, but if you followed the previous two tutorials you already know how this works.

After I’ve done that I need to increase the “patch” version number (the “Z” part of the “X.Y.Z” semver scheme). I can do this with the “verup” command, also run in the project’s base directory:

ceverett@okonomiyaki:~/vcs/termifierg$ zx verup patch
Version changed from 0.1.0 to 0.1.1.

And now time to re-package and put it into the realm. Again, since I’m the sysop this is super fast for me working alone:

ceverett@okonomiyaki:~/vcs$ zx submit otpr-termifierg-0.1.1.zsp 
ceverett@okonomiyaki:~/vcs$ zx approve otpr-termifierg-0.1.1
ceverett@okonomiyaki:~/vcs$ zx accept otpr-termifierg-0.1.1.zsp

And that’s that. It can immediately be run by anyone anywhere as long as they have ZX installed.

BONUS LEVEL!

“Neat, but what about the screenshot of it running?”

Up until now we’ve been launching code using ZX from the command line. Since Termifier GUI is a GUI program and usually the target audience for GUI programs is not programmers, yesterday I started on a new graphical front end for ZX intended for ordinary users (you know, people expert at things other than programming!). This tool is called “Vapor” and is still an ugly duckling in beta, but workable enough to demonstrate its usefulness. It allows people to graphically browse projects from their desktop, and launch by clicking if the project is actually launchable.

Vapor is like low-pressure Steam, but with a strong DIY element to it, as anyone can become a developer and host their own code.

I haven’t written the window manager/desktop registration bits yet, so I will start Vapor from the command line with ZX:

You’ll notice a few things here:

  • Termifier GUI’s latest version is already selected for us, but if we click that button it will become a version selector and we can pick a specific version.
  • Observer is listed, but only as a “virtual package” because it is part of OTP, not actually a real otpr package. For this reason it lacks a version selector. (More on this below.)
  • Vapor lacks a “run” button of its own because it is already running (ZX is similarly special-cased)

When I click Termifier’s “run” button Vapor’s window goes away and we see that the termifierg-0.1.1 package is fetched from Zomp (along with deps, if they aren’t already present on the system), built and executed. If we run it a second time it will run immediately from the local cache since it and all deps are already built.

When Termifier terminates Vapor lets ZX know it is OK to shutdown the runtime.

A special note on Observer and “Virtual Packages”

[UPDATE 2020-01-12: The concept of virtual packages is going away, observer will have a different launch method soon, and a rather large interface change is coming to Vapor soon. The general principles and function the system remain the same, but the GUI will look significantly different in the future — the above is the day-2 functioning prototype.]

When other programs are run by Vapor the main Vapor window is closed. Remember, each execution environment is constructed at runtime for the specific application being run, so if we run two programs that have conflicting dependencies there will be confusion about the order to search for modules that are being called! To prevent contamination Vapor only allows a single application to be run at once from a single instance of Vapor (you can run several Vapor instances at once, though, as each invocation of ZX creates an independent Erlang runtime with its own context and environment — the various zx_daemons coordinate locally to pick a leader, though, so resource contention is avoided by proxying through the leader). If you want several inter-related apps to run at once within the same Erlang runtime, create a meta-package that has the sole function of launching them all together with commonly defined dependencies.

Because Observer is part of OTP it does not suffer from dependency or environmental conflict issues, so running Observer is safe and the “run” button does just that: it runs Observer. Vapor will stay open while Observer is running, allowing you to pick another application to run, and you can watch what it is up to using Observer as a monitoring tool, which can be quite handy (and interesting!).

If you want to run an Erlang network service type application using Vapor while using Observer (like a chat server, or even a Zomp node) you should start Vapor using the zxh command (not just plain zx), because that provides an Erlang shell on the command line so you can still interact with the program from there. You can also run anything using plain old zx run, and when the target application terminates that instance of the runtime will shut down (this is why ZX application templates define applications as “permanent“).

Cool story, bro. What Comes Next?

The next step for this little bundle of projects is to create an all-encompassing Windows installer for Erlang, ZX and Vapor (so it can all be a one-shot install for users), and add a desktop registration feature to Vapor so that Erlang applications can be “installed” on the local system, registered with desktop icons, menu entries and file associations in FreeDesktop and Windows conformant systems (I’ll have to learn how to do it on OSX, but that would be great, too!). Then users could run applications without really knowing about Vapor, because authors could write installation scripts that invoke Vapor’s registration routines directly.

If I have my way (and I always get my way eventually) Erlang will go from being the hardest and most annoying language to deploy client-side to being one of the easiest to deploy client-side across all supported platforms. BWAHAHAHA! (I admit, maybe this isn’t a world-changing goal, but for me it would be a world-changing thing…)

Older Posts »

Powered by WordPress