Category Archives: Science & Tech

Everything from lampooning Popular Science to picking on Microsoft and Oracle belong here.

Zomp/zx: Yet Another Repository System

I’ve been working on a from-source repo system for Erlang on and off for the last few months, contributing time to it pretty much whenever real-life is not interfering. I’m getting close to making a release. Now that my main data bits are worked out, the rest isn’t all that hard. I need to figure out what I want to say in an announcement.

The problem is that I’m really horrible at announcements and this system does things in a pretty different way to other repository systems out there, so I’m not sure what things are going to be important about it to users (worth putting into an announcement) and what things are going to be important to only me because I’m the one who wrote it (and am therefore obsessed with its externally inconsequential internals). What is internally interesting about a project is almost never what is externally interesting about it. Marketing; QED. So I need to sort that out, and writing sometimes helps me sort that kind of thing out.

I’m making this deliberately half-baked, disorganized, over-long post public because Joe Armstrong gave me some food for thought the other day. I had written him my thoughts on a subject posted to a mailing list but sent the message in private. I made my message to him off-list for two reasons: first, I wasn’t comfortable with my way of expressing the idea just yet; and second, I am busy with real-life stuff and side projects, including the repo system, and don’t want to get sucked into online chatter that might amount to nothing more than bikeshedding. (I’m a world-class bikeshedder!) Joe wrote me back asking why I made the reply private, I told him my reasons, and he made me change my mind. He hopes that more people will publish their ideas all the time, good or bad, fully baked or still soggy — because that’s the only way we can ever find any other interesting ideas these days is by searching for them, usually in text, on the net somewhere. It isn’t like we can’t go back and revise, but whether or not we do go back and clean up our literary messes, the availability of core ideas and exposure of thought processes are more important than polish. He’s been on a big drive to make sure that he posts most of his thoughts to public mailing lists or blogs so that his ideas get at least indexed and archived. On reflection I agree with him.

So here I am, trying to publicly organize my thoughts on my repository system.

I should start with the goals of the system.

This system is intended to smooth over a few points of pain experienced when trying to get a new Erlang project off the ground, and in particular avert the path of pain peculiar to Erlang newcomers when they encounter the “how to set up a project” problem. Erlang’s tooling is great but a bit crufty (deeply featured, but confusing to interface with) and not at all what the kool kids expect these days. And anyway I’m really just trying to scratch my own itch here.

At the moment we have two de facto standards for publishing Erlang systems: erlang.mk and Rebar. I like both of these, especially erlang.mk, but they do one thing that annoys me and never seems to quite fit my need: they build Erlang releases.

Erlang releases are great. They cut all the cruft of a release out and pack everything needed to actually run a system into a single blob of digits that you can move, in a single shot, to a new target system — including the Erlang runtime itself. Awesome! Self-contained deployment and it never misses. This has been an Erlang feature since before people even realized that they needed repeatable deployment infrastructure outside of the classic “let’s build a monolithic, static binary executable” approach. (Erlang is perpetually ahead of its time, even by today’s standards. I look at the poor kids stubbing their toes with Docker and language du jour and just shake my head — though part of that is because many shops are using Docker to solve concurrency issues that they haven’t even become cognizant of, thinking that they are experiencing “scaling” problems but missing the point entirely.)

Erlang releases are awesome when the deployment target is an embedded system, but not so awesome if the target is a full-blown operating system, VM, container, or virtual environment fully stocked with gobs of memory and storage and flush with system utilities and resources. Erlang releases sort of kitchen-sink the deployment itself. What if you want to run several different Erlang programs, all delivered as releases, all depending on the same library? You’ve got tons of copies of that library. Which is OK, but still sort of weird, because you also have tons of copies of the runtime (among other things). Each release is self-contained and lean, but in aggregate this is a bit odd.

Erlang releases make sense when you’re deploying to a phone switch or a sensor device in the middle of nowhere and the runtime is basically acting as its own operating system. Erlang releases are, in that context, analogous to putting a Gentoo stage 3 binary image on a system to leapfrog most of the toolchain process. Very cool when you’re in that situation, but a bit tinker-tacky when you’re just trying to run, say, a client program written in Erlang or test a web front-end for something that uses YAWS or Cowboy.

So that’s the siloed-kitchen-sink issue. The other issue is that newcomers are perpetually confused about releases. This makes teaching elementary Erlang hard. In my view we should really focus on escript for beginner code — just let the new guy run something out of a single file the way he is used to doing when learning a new language instead of showing him pages of really slick code, then some interpreter stuff, and then leaping straight from that to a complex and advanced packaging setup necessarily tailored for conducting embedded deployments to slim hardware devices. Seriously. WTF. Escripts give beginners all the power of Erlang necessary for exploring the more interesting bits of code and refactoring needed to learn sequential Erlang with the major advantage of being able to interface with the system the same way programmers from other environments are used to dealing with langauge runtimes like Bash, AWK, Python, Ruby, Perl, etc.

But what about that gap between scripts and full-blown production deployments for embedded hardware?

Erlang has… nothing.

That’s right! There is no agreed-upon way to deploy or even run Erlang code in the same manner a Python coder would expect to execute a python program. There is no virtualenv type system, there is no standard answer to the question “if I’m in the project directory and type ./do_thingy it will just work, right?” The answer is always “Well, it depends…” and what actually winds up happening is that people either roll a whole release just to crank a trivial amount of code up or (quite often) implement an ad hoc way to get the same effect in a lighter-weight way. (erlang.mk shines here, actually.)

Erlang does provide a number of ways to make a system run locally from source of .beam files — and has actually quite reasonable built-in resources for this — but nothing has been built around these tools that also deals with external dependencies, argument passing in a standard way, or any of the other little things you really need if you want to claim a complete solution. Hence all the ad hoc solutions that “work on my machine” but certainly aren’t something you expect your users to use (not with broad success, anyway).

This wouldn’t be such a big problem if it weren’t for the fact that not having any standard way to “just run a program” also means that there really isn’t any standard way to deal with client side code in Erlang. This is a big annoyance for me because much of what I do is client-side code. In Erlang.

In fact, it totally boggles my mind that client-side Erlang isn’t more common, especially considering that AMD is already fielding zillion-core processors for desktops, yet most languages are fundamentally single-threaded. That doesn’t mean you can’t do concurrency and parallelism in other languages, but most problems are not parallel in nature to begin with (parallel problems are easy to write solutions to in any language) while most real-world problems are concurrent. But concurrent systems are hard to write in almost every language. Concurrent problems are the bulk of the interesting problems we’re still not very good at solving with computers. AMD is moving to make the tools available to make much more interesting concurrent processing tools available on the client side (which means Intel will soon start pouring it gajillions worth of blood diamond money into a similar effort), but most languages and environments have no good way to make use of that on the client side. (Do you see why I hear Lady Fortune knocking?)

Browsers? Oh yeah. That’s a great plan. Have you noticed that most sites slowly move toward the “Single Page App” design over time (read as: the web sucks, so now we write full-but-crippled client-programs and deliver them over the web), invest heavily in do-sneaky-things-without-telling-you JavaScript and try to hog every core your system has if you allow it the slightest permission to do so? No. In the age of bitcoin miners embedded in nearly every ad this is not the direction I think we should be envisioning things going.

I want to take better advantage of the cores users have available, and that doesn’t necessarily mean make more efficient use of every cycle as much as it means to make scheduling across processes more efficient to reduce latency throughout the system overall. That’s something users care about quite a lot. This is the problem Erlang has already solved in a way no other runtime out there has. So I want to capitalize on it.

And yet, there is still not standardish way of dealing with code from source, running it locally, declaring or resolving dependencies, or even launching a client-side program at all.

So… how am I approaching it?

I have a project called “zomp” which is a repository system. It is a distributed repository system, so not everything has to be held in one place. Code in the zomp universe is held in little semantic silos called “realms”. Each realm can have whatever packages the owner (sysop) wants it to have. Each realm must have one server node somewhere that is its “prime” — the node in charge of that realm. That node is where system operator tasks for that realm take place, packagers and maintainers submit code for inclusion, where the package index is built, where the canonical copy of everything is stored. Other nodes configured to see that realm connect to the prime node and receive a copy of the current indexes and are tested for availability and published as available resources for querying indexes or downloading packages.

When too many subordinate nodes connect to a prime the prime will redirect a new node to a subordinate, when a subordinate gets “full” of subordinates itself, it picks a subordinate for new redirects itself, etc. so each realm winds up forming a resource tree of mirror nodes that connect back to the realm prime by a single path. A single node might be prime for several realms, or other nodes may act as prime for different realms — and any node can be configured to become a part of any number of realm trees.

That’s the high-level code division.

The zomp constellation is interfaced with via the “zx” program (short for “zomp explorer”, or “zomp exchanger”, or “Zomp eXtreem!”, or homage to the Sinclair ZX-81, or whatever else might lend itself to the letters “zx” that you might want to make up — I actually forget what it originally stood for, but it is remarkably convenient to type so it’s staying that way)

zx is configured to have visibility on zomp realms the same way a zomp node is (in fact, they use the same configuration files and it isn’t weird to temporarily host a zomp node on your desktop the same way you might host a torrent node for a while — the only extra effort is that you do have to open a port, zomp doesn’t (yet) do hole punching magic).

You can tell zx to run a program using the highly counter-intuitive command:

zx run Realm-ProgramName[-Version]

It breaks the program name down into:

  • Realm (optional, defaulting to the main realm of public FOSS packages called “otpr”)
  • Name (necessary — sort of the whole point)
  • Version (which is optional and can also be partial: “1.0.3” vs just “1.0” or “1”, defaulting to the latest in a series or latest overall)

With those components it then contacts any zomp node it knows provides the needed realm, resolves the latest version number of the requested program, downloads and unpacks it, checks and downloads any missing dependencies, builds the program, and launches it. (And if it doesn’t know any active mirrors it asks the prime node and is seeded with known mirror nodes in addition to getting its query answered.)

The packages are kept in a local cache stored at the user level, not the system level (sort of like how browsers keep their JS and page caches) — though if you want to daemonize zomp and run it as a permanent service (if you run a realm prime, for example) then you would want to create an unprivileged system user specifically for the purpose. If you specify a fully-qualified “realm-name-version” for execution and the packages already exist and are built, zx just launches the code directly (which is the majority case, so no delay there — fast startup).

All zomp nodes carry a complete index of their configured realms and can answer queries with very little overhead, but only the prime node has a copy of all the packages for that realm

 

Zomp realms are write-only. There is no facility for removing a package from a realm entirely, only for upgrading the versions of packages whenever necessary. (Removal is, of course, possible, but requires manual intervention by the sysop.)

When a zx client or zomp node asks an upstream node for a package and the upstream node does not have a copy it will query its upstream until the request reaches a node that does have a copy. Once found a “found” notice goes back down to the client telling it how many hops away the package is, and new “hops away” notices are sent as the package is passed downstream toward the original requestor (avoiding timeouts and allowing the user to get some feedback about what is going on). The package is cached at each node along the way, so subsequent requests for that same package will be handled immediately without any more relay downloading.

Because the tree of nodes is expected to be relatively ephemeral and in a constant state of flux, the tendency is for package stores on mirror nodes to be populated by only the latest, most popular packages. This prevents the annoying problem with old realms having gobs of packages that nobody uses but mirror hosts being burdened with maintaining them all anyway.

But why not just keep the latest of everything and ditch old packages?

Ever heard of “version shear”? Yeah. Me too. It sucks. That’s why.

There are no “up to” or “greater than” or “abstract version 3” type dependency declarations in zomp package metadata. As a package maintainer you must explicitly declare the complete version of each dependency in your system. In the case of diamond-shaped dependencies (where two packages in your system depend on slightly different versions of the same package) the burden is on the packagers to declare a version that works for a given release of that package. There are no dependency trees for this reason. If your package depends on X, and X depends on Y and Z then your package must be defined as depending on X, Y and Z — and fully specify the versions involved.

Semver is strictly enforced, by the way. That is, all release numbers are “Major.Minor.Patch”. And that’s it. No more, no less. This is one of the primary criteria for inclusion into a public realm and central to the way both zx and zomp interpret package semantics. If an upstream project has some other numbering scheme the packager will need to create a semver standard of his own. And actually, this turns out to not be very hard in practice. There is one weird side-effect of full, static dependency version declarations and semver: updating dependencies results in incrementing your package’s patch number, so even if you don’t change anything in a program for a long time, a program with many dependencies under heavy development may wind up on version 2.3.257 without much change other than the {deps, PackageIDs}. line in the package meta file.

zx helps make you aware of these situations, so solving them has not been particularly difficult in practice.

Why do things this way?

The “static dependencies forever and ever, amen” decision is a tradeoff between the important feature of fully repeatable builds Erlang releases are famous for (to the point of bug-compatibility between deployment sites — which is critical in production) and the flexibility users and developers have come to expect from source repository systems like pip, pypi, CPAN, etc. Because each realm is write-only there is no danger that a package will be superceded and disappear. The way trickle-down caching works for mirror zomp nodes does not unduly burden the subordinate realm mirrors, and the local caching behavior of zx itself at launch time tends to make all of this mostly delay-free for zx clients and still gives them the option to always run “latest available version” if they want.

And on the note of “latest version”…

Client-side programs are not expected to be run too terribly long at a time. People shut desktop programs down, restart computers, update their kernels, etc. So even if a client program runs a long time (on the order of web, email, IRC, certain games, crypto wallets/miners, torrent nodes, Freenode, Tor, etc) it will still have a chance to restart every few days or weeks to check for a new version (if invoked in a way that omits the version number so that it always queries the latest version).

But what about for long-running server-side type programs? When zx starts a script checks the initial environment and then starts the erlang runtime with zx as its target application, passing it the package ID of the desired program to run and its arguments as arguments. That last sentence was odd. An example is helpful:

zx run foo-bar arg1 arg2 arg3

zx invokes the launching script (a Bash script on Linux, BSD and OSX, a batch file on Windows — so actually the command is zx.bash or zx.cmd)  with the arguments run foo-bar arg1 arg2 arg3. zx receives the instruction “run” and then breaks “foo-bar” into {Realm, Name} = {"foo", "bar"}. Everything after that is passed in as strings which wind up being the input arguments to the program being run: “foo-bar”.

zx registers a process called zx_daemon which remains resident in the runtime and waits for a subscription request or zomp query. Any Erlang program written with the intention of being used with zx can send a message to zx_daemon and ask it to maintain a connection to the program’s parent realm and enroll for update notifications. If the target program itself is the subject of a realm index update then it will get a message letting it know what has changed. The program can respond any way the author wants to such a notification.

In this way it is possible to write a client-side or server-side application that can enroll to become aware of updates to itself without any extra infrastructure and a minimal amount of code. In some programs I’ve used this to cause a pop up notification to appear to desktop users so they know that a new version has become available and they should restart the program (the way Firefox does on Windows). It could also be used to initiate a restart on its own, or whatever else you might come up with.

There are several benefits to developers of using this system as well.

As a developer I can start a new project by doing zx init app [Realm-Name] or zx init lib [Realm-Name] in an existing project root directory and a zomp.meta file will be generated for it, or a new project template directory will be created (populated with a functioning sample skeleton project). I can do zx dailyze and zx will make sure a generally relevant PLT exists or is built (if not up to date) and used to check the typespecs of the project and its dependencies. zx create package [Path] will create a zomp package, sign it, and populate the metadata for it. zomp keygen will generate the kind of keys necessary to interact with a zomp server. zomp submit PackageFilePath will submit a package for review.

And so on.. It is a lot easier to do most things now, and that’s the main point.

(There are commands for reviewing, approving, or rejecting package submissions, adding packagers and maintainers to package projects, adding dependencies to projects, X.Y.Z version incrementing, etc. as well.)

This is about 90% of the way I want it to be, but that means about 90% of the effort remains (pessimistically assuming the 90/10 rule, because life sucks and nobody cares). Most of that is probably going to be finagling some network lunacy, but a lot of the effort is going to be in putting polish to it.

Zomp/zx is based on a similar project I wrote for use within Tsuriai a few years ago that has much sparser features but does basically the same thing: eases packaging and repeatable deployment from source to client systems. I would never release that version publicly because it has a lot of “works for me!” level functionality, but very little polish and requires manually diddling quite a few settings files in error-prone ways (which is fine because it was just us diddling them).

My intention here is to Cadillac this out a bit so that newcomers can slide into the new language and just focus on that language after learning a minimum of tooling commands or environmental details. I think zx init app foo-bar and zx runlocal are a low enough bar for entry.

Web Designers: Stop making SPAs for inherently web 1.0 style sites

It is 2017. What’s with the drive to make everything an SPA whether it needs to be or not? This is getting a little ridiculous. I’m going to ramble on below a bit because I’ve got a hankering to do so — pay this no mind.

All around the web I see sites that are best represented as a collection of inter-linked documents, and all around the web I see many of those being changed into single-page application (SPAs). Even more stupid is when the SPA in question was built by some naive dope who included a little bit of almost every JS framework in existence — including a random selection from the thousands of obsolete and dead ones.

What is the goal? What’s the deal? Do web authors today not know how the web was actually intended to work originally? That document publication is actually its reason for existence in the first place and that “web applications” are a new thing that is a backhack to an incomplete standard that only sorta-kinda-works?

Granted, the reason it only sorta-kinda-works is due mostly to the problems inherent in the fact that only a single language is allowed in scripts… which is ridiculous. Was nobody paying attention to the Guile2 approach all those years? The only lesson learned from the Java applet and Flash experience seems to have been that “it sucks to force users to install runtimes as plugins”. Ugh.

Anyway, back to web applications…

I get it. For the moment we don’t have a solid distinction between “a document browser” and “an application browser” so we are stuck with this insufficient worst-of-both-worlds nether region of “applications that masquerade as documents”. And that drives anyone nuts who has given this much thought.

Not that a lot of people have considered the difference deeply. I imagine that is probably because very few new coders today have ever written more than a line or two of code intended to run natively on a user’s local system. Nearly everyone has written thousands of lines of code intended to run natively on server-side systems, but even that is getting wonky because many youngsters today don’t know how to deploy without using Docker yet lack the faintest inkling as to what problems Docker actually is intended to solve and wind up bypassing better solutions when they exist.

Tools shine when they are used in a focused way, performing they job for which they were intended. The web is the same way. Yes, it is a big jumble of crap. So let’s just leave that there. Networks are a big jumble of crap, too, and so are human societies — so we’ve adopted dirty ways of dealing with the dirt. The jumbly pile of shit that is the web is one of our ways of dealing with that. Everything times out. Everything is sent in text. Protocols are bloated and redundant. There isn’t even a proper definition of what “valid” HTML and XML and JSON and whatever else is in most cases. Its all racing toward a singularity where everything is uniformly stupid. But… whatever, it sort of kind of still works — and humans just barely work themselves, so that’s par for the course.

The original web was designed to function as an insecure document publication system where documents could be interlinked. We realized that we could include more interesting stuff by expanding the definition of “document” to include more than just text, and quite recently with HTML5 the way in which documents can be written is only a few orders of magnitude behind, say, LaTeX, in its ability to arrange things on the screen (that’s feature lag is not entirely the fault of the HTML5 authors).

This gives a lot of freedom to website authors — perhaps too much.

If a website is a set of news articles or academic papers (or even tweets) then you really don’t need a SPA, you need a more traditional sort of “web site”. It can be dressed up all pretty with shiny things sprinkled around, of course, but we don’t want a SPA that mysteriously changes state in ways that users cannot bookmark things, can’t easily send links to one another to specific resources (something Twitter got right despite some initial confusion over how to frame their content), etc.

If a website is actually just a delivery front end for a graphical RPG, well, obviously the game part of the site is probably best designed as a SPA, but the rest of the site — the forums, armory, character pages, beastiary, fan wiki, manual, guild rankings, lore pages, etc. — are absolutely best presented outside of that SPA as an actual website.

See the difference?

The game example is actually quite useful to contemplate for a variety of reasons. I’ll probably come back and cut this post down to just that part. Either that or eventually come back and rewrite the first bits to more accurately convey the humor with which I, as a graybeard resident in cyberspace for about 30 years now, view the state of the web today.

Whatever you do, dear reader, have fun coding, and remember: Don’t outsmart yourself!

The most basic Erlang service ⇒ worker pattern

There has been some talk about identifying “Erlang design patterns” or “functional design patterns”. The reason this sort of talk rarely gets very far is because generally speaking “design patterns” is a phrase that means “things you have to do all the time that your language provides both no primitives to represent, and no easy way to write a library function behind which to hide an abstract implementation”. OOP itself, being an entire paradigm built around a special syntax for writing dispatching closures, tends to lack a lot of primitives we want to represent today and has a litany of design patterns.

NOTE: This is a discussion of a very basic Erlang implementation pattern, and being very basic it also points out a few places new Erlangers get hung up, like the context in which a specific call is — because that’s just not obvious if you’re not already familiar with concurrency at the level Erlang does it. If you’re already a wizard, this article probably isn’t for you.

But what about Erlang? Why have so few design patterns (almost none?) emerged here?

The main reason is that what would have been design patterns in Erlang have mostly become either functional abstractions or OTP (here “OTP” refers to the framework shipped with Erlang). This is about as far as the need for patterns has needed to go in the most general case. (Please note that it very often is possible to write a framework that implements a pattern, though it is very difficult to make such frameworks completely generic.)

But there is one thing the ole’ Outlaw Techno Psychobitch doesn’t do for us that quite a few of us do have a common need for but we have to discover for ourselves: how to create a very basic arrangement of service processes, supervisors, and workers that spawn workers according to some ongoing global state or node configuration. (Figuring this out is almost like a rite of passage for Erlangers — and often even experienced Erlangers have never distilled this down to a pattern, even if many projects do eventually evolve into something structured similarly.)

The case I will describe below involves two things:

  • There is some service you want to create that is represented by a named process that manages it and acts as its sole interface. Higher-level code in the system doesn’t want to call low-level code to get things done, the service should know how to manage itself.
  • There is some configurable state that is relevant to the service as a whole, should be remembered, and you should not be forced to pass in as arguments every time you call for this work to be done.

For example, let’s say we have an artificial world written in Erlang. Let’s say its a game world. Let’s say mob management is abstracted behind a single mob manager service interface. You want to spawn a bunch of monster mobs according to rules such as blahlblahblah… (Who cares? The game system should know the details, right?) So that’s our task: spawning mobs. We need to spawn a bunch of monster mob controller processes, and they (of course) need to be supervised, but we shouldn’t have to know all the details to be able to tell the system to create a mob.

The bestiary is really basic config data that shouldn’t have to be passed in every time you call for a new monster to be spawned. Maybe you want to back up further and not even want to have to specify the type of monster — perhaps the game system itself should know what the correct spawn/live percentages are for different types of mobs. Maybe it also knows the best way to deal with positioning to create a playable density, deal with position conflicts, zone conflicts, leveling or phasing influences, and other things. Like I said already: “Who cares?”

Wait, what am I really talking about here? I’m talking about sane defaults, really. Sane defaults that should rule the default case, and in Erlang that generally means some sane options that are comfortably curried away in the lowest-arity calls to whatever the service functions are.  But from whence come these sane defaults? The service state, of course.

So now that we have our scenario in mind, how does this sort of thing tend to work out? As three logical components:

  • The service interface and state keeper, let’s call it a “manager” (typically shortened to “man”)
  • The spawning supervisor (typically shortened to “sup”)
  • The spawned thingies (not shortened at all because it is what it is)

How does that typically look in Erlang? Like three modules in this imaginary-but-typical case:

  • game_mob_man.erl
  • game_mob_sup.erl
  • game_mob.erl

The game_mob_man module represents the Erlang version of a singleton, or at least something very similar in nature: a registered process. So we have a definite point of contact for all requests to create mobs: calling game_mob_man:spawn_mob/0,1,... which is defined as

spawn_mob() ->
    spawn_mob(sane_default()).

spawn_mob(Options) ->
    gen_server:cast(?MODULE, {beget_mob, Options}).

 

Internally there is the detail of the typical

handle_cast({beget_mob, Options}, State) ->
    ok = beget_mob(Options, State),
    {noreply, State};
%...

and of course, since you should never be putting a bunch of logic or side-effecty stuff in directly in your handle_* function clauses beget_mob/2 is where the work actually occurs. Of course, since we are talking about common patterns, I should point out that there are not always good linguistic parallels like “spawn” ⇒ “beget” so a very common thing to see is some_verb/N becomes a message {verb_name, Data} becomes a call to an implementation do_some_verb(Data, State):

spawn_mob(Options) ->
    gen_server:cast(?MODULE, {spawn_mob, Options}).

%...

handle_cast({spawn_mob, Options}, State) ->
    ok = do_spawn_mob(Options, State),
    {noreply, State};

% ...

do_spawn_mob(Options, State = #s{stuff = Stuff}) ->
    % Actually do work in the `do_*` functions down here

The important thing to note above is that this is the kind of registered module that is registered under its own name, which is why the call to gen_server:cast/2 is using ?MODULE as the address (and not self(), because remember, interface functions are executed in the context of the caller, not the process defined by the module).

Also, are the some_verb/N{some_verb, Data}do_some_verb/N names sort of redundant? Yes, indeed they are. But they are totally unambiguous, inherently easy to grep -n and most importantly, give us breaks in the chain of function calls necessary to implement abstractions like managed messaging and supervision that underlies OTP magic like the gen_server itself. So don’t begrudge the names, its just a convention. Learn the convention so that you write less annoyingly mysterious code; your future self will thank you.

So what does that have to do with spawning workers and all that? Inside do_spawn_mob/N we are going to call another registered process, game_mob_sup. Why not just call game_mob_sup directly? For two reasons:

  1. Defining spawn_mob/N within the supervisor still requires acquisition of world configuration and current game state, and supervisors do not hold that kind of state, so you don’t want data retrieval tasks or evaluation logic to be defined there. Any calls to a supervisor’s public functions are being called in the context of the caller, not the supervisor itself anyway. Don’t forget this. Calling the manger first gives the manager a chance to wrap its call to the supervisor in state and pass the message along — quite natural.
  2. game_mob_sup is just a supervisor, it is not the mob service itself. It can’t be. OTP already dictates what it is, and its role is limited to being a supervisor (and in this particular case of dynamic workers, a simple_one_for_one supervisor at that).

So how does game_mob_sup look inside? Something very close to this:

-module(game_mob_sup).
-behavior(supervisor).

%%% Interface
spawn_mob(Conf) ->
    supervisor:start_child(?MODULE, [Conf]).

%%% Startup
start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    RestartStrategy = {simple_one_for_one, 5, 60},
    Mob = {game_mob,
           {game_mob, start_link, []},
           temporary,
           brutal_kill,
           worker,
           [game_mob]},
    Children = [Mob],
    {ok, {RestartStrategy, Children}}.

(Is it really necessary to define these things as variables in init/1? No. Is it really necessary to break the tuple assigned to Mob vertically into lines and align everything all pretty like that? No. Of course not. But it is pretty darn common and therefore very easy to catch all the pieces with your eyes when you first glance at the module. Its about readability, not being uber l33t and reducing a line count nobody is even aware of that isn’t even relevant to the compiled code.)

See what’s going on in there? Almost nothing. That’s what. The interesting part to note is that very little config data is going into the supervisor at all, with the exception of how supervision is set to work. These are mobs: if they crash they shouldn’t come back to life, better to leave them dead and signal whatever keeps account of them so it can decide what to do (the game_mob_man, for example, which would probably be monitoring these). Setting them as permanent workers can easily (and hilariously) result in a phenomenon called “highly available mini bosses” — where a crash in the “at death cleanup” routine or the mistake of having the mob’s process retire with an exit status other than 'normal' causes it to just keep coming back to life right there, in its initial configuration (i.e. full health, full weapons, full mana, etc.).

But what stands above this? Who supervises the supervisor?

Generally speaking, a component like mob monsters would be a part of a larger concept of world objects, so whatever the world object “service” concept is would sit above mobs, and mobs would be one component of world entities in general.

To sum up, here is a craptastic diagram:

Yes, my games involve wildlife and blonde nurses.

Yes, my games involve wildlife and blonde nurses.

The diagram above shows solid lines for spawn_link, and dashed lines to indicate the direction of requests for things like spawn_link. The diagram does not show anything else. So monitors, messages, etc. are all just not there. Imagine them. Or don’t. That’s not the point of this post.

“But wait, I see what you did there… you made a bigger diagram and cut a bunch of stuff out!”

Yep. I did that. I made an even huger, much crappier, more inaccurate diagram because I wasn’t sure at first where I wanted to fit this into my imaginary game system.

And then I got carried away and diagrammed a lot more of the supervision tree.

And then I though “Meh, screw it, I’ll just push this up to a rough imagining of what it might look like pushed all the way back to the SuperSup”.

Here is the result of that digression:

It wouldn't look exactly like this, so use your imagination.

It wouldn’t look exactly like this, so use your imagination.

ALL. THAT. SUPERVISION.

Yep. All that. Right there. That’s why its called a “supervision tree” instead of a “supervision list”. Any place in there you don’t have a dependency between parts, a thing can crash all by itself and not bring down the system. Consider this: the entire game can fail and chat will still work, users will still be logged in, etc. Not nearly as big a deal to restart just that one part. But what about ItemReg? Well, if that fails, we should probably squash the entire item system (I’ve got guns, but no bullets! or whatever) because game items are critical data. Are they really critical data? No. But they become critical because gamers are much more willing to accept a server interruption than they are losing items and having bad item data stored.

And with that, I’m out! Hopefully I was able to express a tiny little bit about one way supervision can be coupled with workers in the context of an ongoing, configured service that lives within a larger Erlang system and requires on-the-fly spawning of supervised workers.

(Before any of you smarties that have been around a while and point out how I glossed over a few things, or how spawning a million items as processes might not be the best idea… I know. That’s not the point of this post, and the “right approach” is entirely context dependent anyway. But constructive criticism is, as always, most welcome.)

How the Internet of Things Will Change the World: Not by Much

Are you ready for the enormous, revolutionary, ground-shattering changes coming with the IoT?

If you said “yes” and by “yes” you meant you were prepared for breathtaking changes, you are a naive child wading in a murky pool of lampreys, your will putty in the hands of the same charlatans who brought you terms like “cloud computing” which still has yet to be defined in any concrete technical sense.

If you said “yes” and by “yes” you meant that you felt that the more things change the more they stay the same — then you are indeed prepared.

Cold War II, civil war in China, the breakup of the EU, abolishment of American drug laws, the DEA and an end to the Mexican civil war all at once — those are the kinds of things that will have a measurable impact on life. The so-called “internet of things” concept as heard in internet marketing is… well, not at all what the guy who coined the term “Internet of Things” meant.

We already have an internet of things. Has it cured cancer yet? Nope. But if we put RFID in every part of our bodies we will certainly be even more exposed to the will of outside actors. Not that the public has demonstrated that it cares about complete loss of its privacy, especially when “Google style conveniences in exchange for your life’s data” can be backed up by the rhetoric of fear necessitated by government “anti-terrorism” funding. (Yes, I mock this, and yes, I was a Green Beret in the US Army for 6 years — the direction that rhetoric is headed is toward government empowerment, and the government is exactly the least well equipped element of society to deal with terrorism.)

Want to see an internet of things? Tesla cars receive system updates across the network now, and can turn in performance data to help the maker improve on their designs and software. Open water jetski robots can follow automated routes and report hydrographic and bathyrithmic data back to a data processing facility to chart change over time. I was working on a (now defunct, but promising) design project to develop spotting scopes that were intelligent enough to peer data amongst one another within an operational space and change “spotter calls” into more generally interesting “shot requests” and aggregate shot providers in the area to engage targets based on type, effect and following damage reports. Whenever any peers had a network connection they could aggregate data externally.

Dude, we’re already there.

What we lack is standards. Oh, wait, nevermind… we actually have tens of thousands of those. What we lack is standards that people actually can use, that aren’t harder to learn and comply with than the handling of the basic user problems at hand are. These problems will mostly never be solved, not really. Truly understandable data must have a semantic foundation, and semantics are essentially arbitrary in most ways that matter in data processing. That means data must either be tagged in a non-trivial way or must be placed into a schema where relationships are what have meanings.

Note that “tagged in a non-trivial way” above means taking tagging systems to such extremes that they become their own ontologies. Think about that. It should make your face turn pale. That’s at least as difficult as developing an arbitrary formal language. (In case you didn’t notice, an “arbitrary formal” language is a oxymoron — though consortia and governments alike love nothing more than funding committee efforts to formalize the syntax of futile efforts in this area). Writing even trivial software using such a tagging system would require that programmers at every step of the system learn this arbitrary formal language of tagging before they do much of anything, and that’s a lot harder overall than just continuing on with our pile-of-ad-hoc-systems approach. Schema-based systems, while having some different tradeoffs (computationally natural descriptions of data as a “shape”, for example, is a really big win in practical application), ultimately suffer from the same complexity explosion at some level. In particular, applying a particular schema designed in the context of one problem domain will very often not fit in the context of another problem domain — and fully normalizing all data ever, ever would eventually require an infinite (and ever growing) number of relational definitions. Yech.

So… Internet of things? Yeah. We’re already living it. Don’t get too excited and don’t give into the hype. Just because you technically can read data from remote sensors or activate your house’s appliances with your phone (hint: you already can) doesn’t mean this is something you will want to do, or that the venture capitalists of the world will peel their lips off the adsearch cock for long enough to realize that there are more interesting things they could be funding than bounce-under ads and invisible iframe reclick-to-click javascript tech.

Rest easy. Humanity’s material circumstances will continue to get incrementally better (save the occasional dip due to predictably stupid things we do to ourselves) until The Singularity when we are all either suddenly eliminated due to obsolescence, or drive ourselves into a new mode of existence-as-slavery to whatever Google turns into (when all data is network accessible, privacy does not exist, all data is the private IP of a single aggregate, the rights of conscious uploaded entities don’t exist, the definition of “life” is still “way way way after birth”, and continued conscious existence equates to paying a service charge — that’s not really life). None of this is particularly dependent upon the hype surrounding the “Internet of Things”.

JSON and YAML: Not a pair that fits every foot (but XML sucks)

It is good to reflect on exactly how hard a problem it is to define a consistent cross-platform data representation. Most of the time (especially on the web) we just shovel data around, let things be inconsistent, avoid conflicts by pretending they don’t happen, and carry a general disregard to data consistently. This attitude is, sadly, what has come to characterize “NoSQL” in my mind, though in a strict sense that is not true at all (GIS and graph databases aren’t SQL systems, and some are very solid — PostGIS being the exception in that it is a surprisingly well made extension to a surprisingly solid SQL-based RDBMS).

Obviously this isn’t a good attitude to have when dealing with things more important than small games or social media distractions. That said, most of the code written today seems to fall into those two categories, and many a career is spent exclusively roaming the range between these two (and whether we should consider most of the crap that constitutes the web a “game” itself is worth thinking about, whether we think of SEO, mindshare in the blogosphere, StackExchange rep, Facebook likes/friends/whatever, pingbacks, comment counts, etc.). We focus so much on these trivial and often meaningless cases that an entire generation of would-be programmers has no idea what the shape of data is really about.

When you really need a consistent data representation that can survive the network (ouch! that’s no mean feat!), can consistently be coerced into a known, predictable, serialized representation, and can be handled by generated code in nearly any language you need ASN.1.

But ASN.1 is hard to learn (or even find resources on outside of telecom projects), and JSON and YAML are easy to reference and (initially) use. XML was made unnecessarily hard, I think as a cosmic joke on people who never heard the term “S-expression”, but very basic XML seems easy, even if its something you would never want to type by hand (though that always seems to wind up being necessary, despite our best efforts at tooling…).

Why not just use JSON, YAML or XML everywhere? That bit above, about a consistent representation — that’s why. Well, that’s part of why. Another part of why is that despite your best efforts to define things in XML or nest explicit declarations in YAML/JSON you will always wind up either missing something, or find yourself needing to change some type information you embedded as a nested element in your data definition and then need to write a sed or awk script just to modify later (and if you’re the type who thinks “ah, a simple search/replace in my IDE…” and “IDE” to you doesn’t basically equate to “Emacs” or your shell itself, you’re going to a gunfight with boxing gloves on — if you need a better IDE to manage your language then you really need a better language).

The problem with YAML/JSON/XML are twofold: they are not defined anywhere, so while you may have a standard of sorts somewhere, there is no way to enforce that standard. An alternative is to include type information everywhere within your tags as attributes (in XML) or nest tagged groups or create a massive reference chain of type -> reference pointer -> data entry in YAML (or nest everything to an insane degree in JSON), but then making changes to the type of a field in a record type you have 20 million instances of is… problematic.

And we haven’t even discussed representation. “But its all text, right?” Oh… you silly person…

Everything is numbers. Two numbers, actually, 0 and 1. We know this, and we sort of forget that there are several ways of interpreting those numbers as bigger numbers, and those bigger numbers as textual components, and those textual components as actual text and that actual text (finally) as glyph you see when you use “the typewriter part” or look at “the TV part” (or do anything with the little touchscreens we use everywhere these days nobody seems to have worked out a genuinely solid interface solution to just yet).

Every layer of that chain of interpretation I mentioned above can be done several ways. Every layer. Think about that for a second. Now, if you live purely in a single world (like modern Linuxes and probably newer versions of OSX) where there is only UTF-8, then about half the possible permutations are eliminated. If you only ever deal with unaccented characters that fall in the primary 127 defined by ASCII, then several more permutations are eliminated — and you should dance with joy.

Unless you deal with a bit of non-textual data in addition to the textual stuff. You know, like pictures and sounds and application-produced opaque binary data and whatnot. If that’s the case, you should tremble. Or… oh god, no… what if your data doesn’t stand alone? What if all those letters are supposed to actually mean something? “We have lots of data” isn’t nearly as important to customers as “we have lots of meanings” — but don’t ask a customer about that directly, they have no idea what you mean, because all the text stuff already means something to them.

Why OTP? Why “pure” and not “raw” Erlang?

I’ve been working on my little instructional project for the last few days and today finally got around to putting a very minimal, but working, chat system into the ErlMUD “scaffolding” code. (The commit with original comment is here. Note the date. By the time this post makes its way into Google things will probably be a lot different.)

I commented the commit on GitHub, but felt it was significant enough to reproduce here (lightly edited and linked). The state of the “raw Erlang” ErlMUD codebase as of this commit is significant because it clearly demonstrates the need for many Erlang community conventions, and even more significantly why OTP was written in the first place. Not only does it demonstrate the need for them, the non-trivial nature of the problem being handled has incidentally given rise to some very clear patterns which are already recognizable as proto-OTP usage patterns (without the important detail of having written any behaviors just yet). Here is the commit comment:

Originally chanman had been written to monitor, but not link or trap exits of channel processes [example]. At first glance this appears acceptable, after all the chanman doesn’t have any need to restart channels since they are supposed to die when they hit zero participants, and upon death the participant count winds up being zero.

But this assumes that the chanman itself will never die. This is always a faulty assumption. As a user it might be mildly inconvenient to suddenly be kicked from all channels, but it isn’t unusual for chat services to hiccup and it is easy to re-join whatever died. Resource exhaustion and an inconsistent channel registry is worse. If orphaned channels are left lying about the output of \list can never match reality, and identically named ones can be created in ways that don’t make sense. Even a trivial chat service with a tiny codebase like this can wind up with system partitions and inconsistent states (oh no!).

All channels crashing with the chanman might suck a little, but letting the server get to a corrupted state is unrecoverable without a restart. That requires taking the game and everything else down with it just because the chat service had a hiccup. This is totally unacceptable. Here we have one of the most important examples of why supervision trees matter: they create a direct chain of command, and enforce a no-orphan policy by annihilation. Notice that I have been writing “managers” not “supervisors” so far. This is to force me to (re)discover the utility of separating the concepts of process supervision and resource management (they are not the same thing, as we will see later).

Now that most of the “scaffolding” bits have been written in raw Erlang it is a good time to sit back and check out just how much repetitive code has been popping up all over the place. The repetitions aren’t resulting from some mandatory framework or environment boilerplate — I’m deliberately making an effort to write really “low level” Erlang, so low that there are no system or framework imposed patterns — they are resulting from the basic, natural fact that service workers form constellations of similarly defined processes and supervision trees provide one of the only known ways to guarantee fallback to a known state throughout the entire system without resorting to global restarts.

Another very important thing to notice is how inconsistent my off-the-cuff implementation of several of these patterns has been. Sometimes a loop has a single State variable that wraps the state of a service, sometimes bits are split out, sometimes it was one way to begin with and switched a few commits ago (especially once the argument list grew long enough to annoy me when typing). Some code_change/N functions have flipped back and forth along with this, and that required hand tweaking code that really could have been easier had every loop accepted a single wrapped State (or at least some standard structure that didn’t change every time I added something to the main loop without messing with code_change). Some places I start with a monitor and wind up with a link or vice versa, etc.

While the proper selection of OTP elements is more an art than a science in many cases, having commonly used components of a known utility already grouped together avoids the need for all this dancing about in code to figure out just what I want to do. I suppose the most damning point about all this is that none of the code I’ve been flip-flopping on has been essential to the actual problem I’m trying to solve. I didn’t set out to write a bunch of monitor or link or registry management code. The only message handler I care about is the one that sends a chat message to the right people. Very little of my code has been about solving that particular problem, and instead I consumed a few hours thinking through how I want the system to support itself, and spent very little time actually dealing with the problem I wanted to treat. Of course, writing this sort of thing without the help of any external libraries in any other language or environment I can think of would have been much more difficult, but the commit history today is a very strong case for making an effort to extract the common patterns used and isolate them from the actual problem solving bits.

The final thing to note is something I commented on a few commits ago, which is just how confusing tracing message passage can be when not using module interface functions. The send and receive locations are distant in the code, so checking for where things are sent from and where they are going to is a bit of a trick in the more complex cases (and fortunately none of this has been particularly complex, or I probably would have needed to write interface functions just to get anything done). One of the best things about using interface functions is the ability to glance at them for type information while working on other modules, use tools like Dialyzer (which we won’t get into we get into “pure Erlang” in v0.2), and easily grep or let Emacs or an IDE find calling sites for you. This is nearly impossible with pure ad hoc messaging. Ad hoc messaging is fine when writing a stub or two to test a concept, but anything beyond that starts getting very hard to keep track of, because the locations significant to the message protocol are both scattered about the code (seemingly at random) and can’t be defined by any typing tools.

I think this code proves three things:

  • Raw Erlang is amazingly quick for hacking things together that are more difficult to get right in other languages, even when writing the “robust” bits and scaffolding without the assistance of external libraries or applications. I wrote a robust chat system this afternoon that can be “hot” updated, from scratch, all by hand, with no framework code — that’s sort of amazing. But doing it sucked more than it needed to since I deliberately avoided adhering to most coding standards, but it was still possible and relatively quick. I wouldn’t want to have to maintain this two months from now, though — and that’s the real sticking point if you want to write production code.
  • Code convention recommendations from folks like Joe Armstrong (who actually does a good bit of by-hand, pure Erlang server writing — but is usually rather specific about how he does it), and standard set utilities like OTP exists for an obvious reason. Just look at the mess I’ve created!
  • Deployment clearly requires a better solution than this. We won’t touch on this issue for a while yet, but seriously, how in the hell would you automate deployment of a scattering of files like this?

On the Meaning of “Carbon Neutral”

I noticed that a few products in my house have begun to proudly proclaim themselves as being “carbon neutral” over the last few months. Apparently this is among the new set of empty phrases marketing people feel are necessary to distinguish their otherwise ordinary commodity products from identical products of comparable quality. It used to be “Made in U.S.A.” or “日本製” (depending on the neighborhood), then it was “low sodium”, then “waterproof”, then “low fat” then “low transfat” then “cholesterol free” then “omega-3” then something else I probably forgot.

The problem isn’t that any of these things aren’t potentially good selling points, its that they usually don’t apply to the things I see the labels on. For example, I remember seeing an electric wok that said “Made in U.S.A.” on the bottom. I’m not so sure that’s the best thing to concern one’s self with when buying a cooking apparatus that originated in another hemisphere. That’s like buying a tuna steak because the sticker on the package marks it as being “a peanut-free product” or believing that a piece of software is high quality because its written in Java (or even more uselessly, “utilizes Java technology!”).

This reminds me of my sister’s enlightening tale of the truth behind the now heavily regulated terms “organic” and “all natural” as applied to food labels. She did her undergraduate study in genetics and graduate work in nutrition, worked in colon cancer research for a while, started a dietary medicine program at a hospital in Virginia a few years back, and now (after reaching some disillusionment with the state of government-funded research) raises “range fed Angus beef” as a side interest. She is therefore directly governed by some of the more hilarious regulations the FDA has come up with.

Needless to say, her opinion on the value of these buzzwords has much more influence to me than whatever a “medicinal cannabis expert” has to tell me about the benefits of toking up or the local yoga girl at the gym has to tell me about the benefits of yogurt shakes or almond oil or peanut-butter enemas or whatever it happens to be this week (of course, she’s just right about the benefits of sex in exciting places). In short, the regulations governing terms such as “organic” and “natural flavor” (or even the way the term “X% fat free” is permitted to be used) are both economically draining legally apply due to the administrative overhead of regulatory compliance and yet so full of loopholes that there is really no clear distinction between a head of lettuce that is “organic” and one that isn’t so labeled. Essentially the only difference is the price of the market package.

Of course, the real difference is that the lettuce sporting an “organic” sticker on it is almost undoubtedly produced by a large agribusiness firm that can afford the overhead of doing all the pencil-drills necessary to proclaim their lettuce to be “organic”. Either that, or it is quite pricey lettuce only rich folks who feel the need to spend more to sate their moral thirst can afford, grown at an “organic” farm run by one savvy businessman and a flock of altruist peons bent on saving humanity from itself one vegetable at a time. I’m certainly not saying that large agribusiness is bad — ultimately its the only way we’re going to survive over the long-term (and here I’m including post colonization of space) — but that the terms used on packaging are enormously deceptive in nature.

But that’s food. It is a specific example where it is relatively easy to trace both the regulatory documentation and the research literature. Of course, very few people actually track all that down — other than unusual people like my sister who happen to be trained in genetics, directly involved in agriculture, and so habituated to both scientific and regulatory research that they find nothing daunting about navigating the semantic labyrinth the FDA has let agricultural regulation become in the US (and the phrase “let…become” could easily be replaced with “deliberately made of…”). I suppose the problem isn’t that few people track all that down, really; its more a problem that even if my sister were to go to the trouble of explaining what she knows to the average consumer they wouldn’t have the faintest clue what she was getting at. The average consumer is instead faced with an essentially religious (or at least dogmatic) choice of whether to trust someone that has a stack of official paper backing up her credibility, or a government agency and a huge food industry which are both populated by thousands of people who each have every bit as much officious documentation backing up their reputations.

And that brings me back to “carbon neutral”. We still chase the purported value of demonstrably empty terms such as “cloud computing”, demonstrably failed vehicles such as “social networking”, and demonstrably flimsy labels such as “organic” and “all natural”. But we don’t stop there. We are jumping head-first onto the “carbon neutral” bandwagon as well. The point isn’t that we shouldn’t be concerned with the terrestrial environment, but rather that we must at all times guard against political forces that constantly seek to invent new social mores and foist them on us by conjuring meaning into empty phrases like “carbon neutral”. It tricks you not just into buying ordinary thing A over ordinary-but-cheaper-thing B, but also into feeling morally superior. In this it is indistinguishable from other dogmatic rhetoric that engenders an unfounded sense of moral certainty. If we thought convincing people that a man in the sky doesn’t want them to fly airplanes into office buildings was hard, consider how much more difficult it is to convince average people who genuinely want to “do good” that reasonablish sciency words are nothing more than unfounded political siren songs trying to open one more door for the tax man.

So back to the reasonablish sciency phrase “carbon neutral”… what does it mean? This is where I have some semantic issues, mainly because nobody really knows.

Let’s say, for example, that we start a paper mill. We’ll make paper, but only from recycled paper and only using wind energy. This could probably qualify as being entirely “carbon neutral”. But so could the same paper mill if it planted its own trees. But what about the wind generators? They have to come from somewhere. What about the diesel-powered trucks that carry the old paper stock to the recycling mill? What about the initial material itself? Are we being carbon neutral if we don’t go replace as many trees as our recycled stock represents? How about the electricity used by the paper-compactors run by other companies we have no control over? What about our employees’ cars they use to get to work? What about all the flatulence the invite by eating pure vegan meals?

The initial production itself would almost certainly not qualify as being “carbon neutral” — which demonstrates that we have to draw an arbitrary line somewhere from which we can derive a meaning for the term “carbon neutral”. It is almost certain that something, whether directly or indirectly, involved an increase in carbon emissions (and the meaning of “direct” and “indirect” really should be their own battlegrounds here, considering what people think the term “carbon neutral” means) somewhere at some point, otherwise there wouldn’t be people to buy our recycled earth-friendly paper to begin with.

But what are “carbon emissions”? This is, apparently, intended to only refer to releasing carbon into the air. Consider for a moment how monumentally arbitrary that is. There are currently some well-intended, but enormously misguided efforts to “sequester” carbon by burying it in the crust of the Earth. This, of course, represents an enormously heavy emission of carbon into the environment, but we are calling this a “good” emission (actually, we refrain from using the word “emission” because we intend that to be a “bad” word) because it is going into the ground and not the air. Incidentally, it is also not going into something useful like diamond-edge tools or nano insulators or any other beneficial process that is desperate for carbon (which our planet happens to be poor in by cosmological standards).

So where did all this “bad” carbon come from? If you believe the press, its coming from our SUV exhaust, coal-burning plants, Lady GaGa (well, she might be a Democrat, in which case she can’t be bad), and pretty much anything else that humans use to modify local order at the expense of a probable increase in universal entropy.

Where did the carbon come from for the coal, crude, natural gas and bovine flatulence? Probably from the atmosphere and the sea. At least that’s what a biologist will tell you.

And here is a problem for me. Nobody has explained (not just to my satisfaction, but explained at all) where all the billions of tons of carbon necessary to create the forests that created the coal (and possibly crude oil) came from in the first place.

Well, that’s not quite true. In the first place it came from a long-dead stellar formation, some crumbs of which clumped together to form our solar system. That’s the first place. So the second place. Where did the carbon for all this organic activity come from in the second place? Was it distributed evenly in the early Earth? Has it always been a fixed quantity in the atmosphere? Does it boil out of the molten terrestrial substrate and gradually accumulate in the atmosphere and ocean?

If the forests grew in the first place then the carbon was in the air, at least at one point. If it is a fixed amount of atmospheric carbon then the growth of the forests and their subsequent demise and burial beneath sediment represents an absolutely massive sequestration of atmospheric carbon. If it is indeed a fixed amount, then the absolutely huge amounts of flora and fauna represented by these forests were not prevented from thriving in an atmosphere which contained a huge amount more carbon than the atmosphere contains today. If that is true, then either climate change is not affected much by the carbon content of the atmosphere, or a changed climate does not pose much of a threat to organic life on Earth.

Some parts of the fixed atmospheric quantity scenario don’t really add up. Despite a bit of effort I’ve only managed to scratch the surface of the ice core research literature, but a static amount of available atmospheric carbon doesn’t seem to be the story research efforts so far tell. This area of research has been made amazingly difficult to get a straight tack on ever since environmental sciences got overheated with politically-driven grants looking for results that validate political rhetoric instead of grants seeking research into what is still largely an observational field, but it seems fairly clear that there have been fluctuations in atmospheric carbon content that do not directly coincide with either the timing of ice-ages or the timing of mass terrestrial forestation. (The record is much less clear with regard to life in the ocean — and this could obviously be a key element, but it doesn’t seem that many people are looking there, perhaps because the current rhetoric is full of fear of rising sea levels, not full of hope for a marine component to the puzzle of eternal human salvation). That said, there must be some pretty massive non-human sources of atmospheric carbon which have been in operation millions of years before we evolved (as for where trillions of tons of carbon may have gone, I think the huge coal formations may be an indication).

While the idea that a carbon-rich atmosphere providing adequate conditions for thriving terrestrial life might seem odd (at least when compared with the “Your SUV is killing the Earth!” dialog), the idea that the Earth itself has both mechanisms to gradually ratchet up the total amount of carbon in the atmosphere over the eons and to drastically change the climate in spans measured in mere years (not decades, not centuries or millenia) without human or even atmospheric input is pretty scary.

A lot more scary than the idea that driving my kids to school might be damaging in some small way.

But this isn’t the way we are thinking. We are letting marketers and politicians — two groups infamous for being as destructively self-serving as possible — sell us a new buzzword stew, and we, the consumers, are ready to confidently parrot phrases such as “carbon neutral” about as if they mean something. “Oh, Irene, that salad dressing is 100% organic and carbon neutral — you’re such a gourmet!”

We’re clearly having the wool pulled over our eyes, except this time it doesn’t just play to our ego-maniacal craving to live forever (“If you eat gluten-free yogurt and drink positive-ion water you’ll live forever — and have huge tits/a thicker penis/ungraying hair/a tiny waist!”), it engenders a dangerous sense of moral superiority (“I’m doing this for the planet/global socialism/God/The Great Leader!”) which tends to eliminate all possibility of rational thought on a subject which could indeed affect us all.

What if, for example, the Atlantic currents are just panting their last, barely keeping us away from a global mass cooling event? We won’t just be blind to the threat because we’ve blown our research money on politically driven quests to generate the academic support necessary to pursue whatever new pork-barrel projects we come up with over the next decade or two — we will deny the very idea that a threat other than carbon emissions could possibly exist on moral grounds because we’ve already identified the “real enemy” (wealthy people in SUVs who come with the added benefit of being fun to hate). That’s dangerous.

Words mean things. We should remember that.

Keyboards, Machine Guns, and Other Daily Tools

I’ve got an annoying issue on my mind: keyboard layouts. This is brought on by the recurring need for me to travel to places where two different keyboard layouts are common and other programmers look at you funny if you have to glance down to make sure whether you’re hitting ‘@’ or ‘`’ or ‘”‘ or whatever is there just then. Neither of these regions, of course, use the keyboard layout of my region (Japan).

I was born in the US and grew up with that layout, but it is difficult to find US-layout keyboards here. Though I usually write only a few Japanese-language emails per day its just not practical to use anything but the local flavor — especially since I use かな input for Japanese, not the insane (to me) Romaji system so many people are accustomed to. Even if I did have a bunch of US-layout keyboards it would be crazy to switch among JP-layout laptops,  US-layout crash carts, JP-layout customer systems and US-layout workstations at my offices. So the “when in Rome” rule is in play and I’ve grown accustomed to this layout. It now works well for me.

“But what’s the big deal?” you may ask. Well, the main keys that do Latin letters and numbers are all generally in the same place, so it seems like this wouldn’t be a big deal. The problem is the crazy keys that do “top row” and wildcard stuff like bangs, hashes, quotes, backticks, at-marks, brackets, colons, semicolons, parens, etc. (Not to mention the Japanese-specific stuff like mode changes, 全角・半角, and other nonsense the majority of the world is spared.) All the parts of a keyboard that are pristine and rarely used on a typical email/pr0nz/games user’s keyboard are usually worn smooth on a programmer’s keyboard. Those frequent-use weird keys are the one that seem to be completely, unexplainably jumbled around when you compare US, JP and various European keyboards (so why even have different layouts…?).

But that brings up a point about how personal familiarity and user expectations play into the perception of a given tool as being “good” or “bad”. I grew up on US keyboards; I am intimately aware that they are not “bad” tools; they are bad tools for me right now. The idea that a concept being “easy” is not the same as it actually being “simple” intersects with the idea that a tool being perceived as “good” is not the same as it being objectively “better” than some given alternative.

(It is interesting to note that even the descriptive language must change between the two cases above. Though the basic pattern of thought is similar, “concepts” and “physical tools” are different in ways that magically prevent certain forms of direct comparison.)

I could easily take the position that US-layout is poo and that JP-layout is superior. I could go uber nerd and pretend that some statistical study on finger reach, frequency of use, angle of finger motion, key tension and travel variance, etc. matters when it comes to programming. (Programmers who use Dvorak do this all the time. I always wonder if they take themselves seriously, are consciously crossing the satire-reality divide as a joke, or if they have a getting-punched-in-the-face fetish.) In the case of programming layout doesn’t really matter — consistency does. To imagine that the charater-per-second count of input matters in programming is to imagine that input is the hard part of programming. Its not*. The problem is figuring out what to write in the first place.

[* Unless you are a Java or COBOL programmer. I didn’t realize how intensely verbose either of those were until I dealt with both again recently after years of absence.]

More to the point, what matters is which layout prevents the wetware halting problem in a specific person. When the human doing the work has to stop what he is doing to figure out something unrelated to the essential task at hand then he’s distracted and that sucks, and you have a wetware halting problem.

But it is still almost certainly true that some layouts are probably objectively worse or better than others for intense input jobs that are not intensely creative, like transcription. If that wasn’t true then the stenograph would never have taken off and court reporters would have been just happily flying through rapid-fire legal discourse in an uncompressed, fully textual medium on their typewriters — which would have been quite a thing to witness, actually. But they didn’t because the stenograph was objectively better than the typewriter for this. It follows that other sorts of tools can often be judged against one another in the same way.

Languages can be like that. Not the ones we speak, I mean real languages, the one you speak math to your computer with. The problem with judging that is that there are several dimensions to a programming language. They usually center around three themes: some concept of “safety features”, and some other concepts of “operational features”, and some other concepts of syntax-as-a-feature.

Safety is a bit of a mixed bag, because few folks seem to declare what they want from a language in terms of safety up-front, and rarely define whether it should be a feature of the language, the runtime or the compiler (until after the fact, of course, which is when everyone bitches about whatever the design decision turned out to be). There is the “safety from others” aspect, which The Management loves because they believe it permits them to demand that different elements of a program become invisible to other elements (the whole `public static blahblah` bit in Java). There is the type safety angle, which people are either really loud about or have nothing to say about (probably because once type systems start making sense to you you’re prone to freaking out about it for about a month). There is the runtime safety angle (Three competing approaches: “It will never run if its unsafe.” VS “Its OK if it crashes.” VS “Nothing is safe and it will always run but gradually grow more magical and mysterious over time and you will never really know bwahahahahaha!”).

Operational features are sort of hard to judge, though it seems like this would be easier somehow than the safety tradeoffs. I think this is because language design is still in its infancy and we’re still totally turned around backwards about the idea that “computer science” has anything to do with computers. So the language feature problem turns into the problem of finding the Goldilocks point of “just enough of the right hot features mixed in with the cool familiar ones to get work done” as opposed to having way too few or way too many features to make much sense to use in production.

Judging syntax is totally arbitrary — until you have to come back and read your own code after two years of not having seen it. There are sometimes languages X and Y where both have the same major features but the syntax of one makes it look more like line noise than code. Meh. This is why language arguments are never going to end (so I tend to have runtime arguments instead).

That’s a lot of dimensions. With all that in mind, I personally dislike Java. I dislike some of the things the JVM assumes should be true. I dislike that its taught first to people who really should learn more about the nature of programming first. I… ugh, I just really dislike it as a platform. It solved a certain sort of problem we almost sorta had in the 90’s, but now its just causing problems — and yet as an industry we are too brainwashed to even spot them.

And, to make this post meander even further, that reminds me of guns. No, seriously. There are several excellent machine gun, rifle and pistol designs employed in militaries across the world. Many of them are decent enough that, while some have a slight edge over others in some areas, I’d go to work with pretty much any of them.

For example: the M4 vs. the SCAR. Meh. The SCAR is indeed objectively better, but the M4 is familiar enough to me that I just don’t really care which one I wind up getting stuck with. On the other hand, I don’t have nearly as much faith in the AK-47, especially in an environment where precision, reaction time, quick on/off safety and partial reloading are critical. While they are famously resistant to neglect (which is often mistaken for durability) that’s really a key attribute for a rifle intended for the Mindless Commie Horde or the Untrained And Unwashed Mass Formation of insurgent/freedom-fighter/terrorist whose backers need cheap, trashy guns with which to arm their cheap, trashy goons. Indeed, the AK-47 is in real terms dramatically less good than the SCAR or M4 and there is a whole list of rifles and carbines I would consider before going to work with one. (That said it is not absolutely awful, just so much less good than the alternatives that I’d avoid it if possible — sort of like Java. I mean, at least its not the C++ of guns.)

Where this is really striking is with machine guns and pistols. On the pistol side there are a rather large number of designs that actually break down frequently during heavy use (granted, “heavy use” meant something very different for me than most people who don’t shoot for a living). This never happens in a James Bond movie, of course, but in real life it happens at the most inconvenient times. Come to think of it, there is never a convenient time for a pistol to break because usually if you’re using your pistol it means your real weapons already broke. Once again, despite the absolute superiority in design of the semi-automatic over the revolver, familiarity can overcome the technical deficiencies between the two (with lots of practice) and I would actually prefer to go to work with certain revolvers over certain semi-autos. (This is to say nothing, of course, of the issue of caliber…)

With machine guns, however, the differences in good vs. bad designs are vast. In nearly any modern military you’re absolutely spoilt. A “bad” gun is one that doesn’t have a TV built into the stock to ease the passage of long turns on security and automatically arrange for pizza to be delivered to an 8-digit grid. They are mindlessly easy to load, sight, barrel change, fire, strip, clean, carry, etc. The links disintegrate and can be re-used on unlinked ammo, all sorts of cool toys fit around the thing (which can, sometimes, make them start to suck just from the “too much Star Wars” problem), runaways can have their belt broken, they will eat through just about any garbage that gets caught in the links or even fire bent ammo, and they probably prevent cancer. They aren’t even unreasonably heavy (and its patently unfair to compare it to the uber lightness of an M4). Its amazing how well these things work. But when great machine guns are all you know you start complaining about them, wishing you had a 240 when you’ve been handed an M60 (because its possible to jam it up if you accidentally load it bolt-forward, usually lacks a rail system, or you’re an unsufferable weakling and won’t stop bitching because you didn’t get the lightweight bulldog version).

I’ve had the misfortune of having to go to work with old Soviet machine guns, though, and can attest that they are indeed objectively horrible.

When we say “crew served weapon” in modern armies we mean “the weapon is the centerpiece of the crew” not “this weapon is absolutely unreasonable to assign to any less than three people”. The term “crew served” may have had more semantic purpose back when operating the machinery actually took a crew — back in the days when tripods included full-sized chairs, ammo came on a horse-drawn cart, and vast amounts of oil and water were consumed in operation. But that was the early 1900’s. We still employ machine guns as “crew served weapons” because it is generally advantageous to have an AG with you and usually a good idea to actually set up a tripod if you find yourself facing off against a for-real infantry force. That is completely different than calling it “crew served” because wielding one is equivalent in complexity to running a mortar section.

Today a single person can easily maintain and operate an M240, M60, MAG58, 249, MG42, MG3, MG11, or whatever. Not so with, say, the PKM (or heaven forbid the SG-43). An RP-46 is actually better if you come to the field with American-style assumptions that a single person is adequate to handle a machine gun (the “zomg! PKM!” not-a-Soviet fanboi hype may have infected your brain already though and make this sound like a crazy statement — until you actually try both).

Let’s clear one point of nomenclature up straight away. The PKM is not really belt fed, it is chain fed, and the chain doesn’t disintegrate. Its also extremely strong. “Strong” as in you can support more than a single person’s weight from an empty belt. The longer the belt the more bullets, and this seems good at first, until you realize that holy shit it feeds from the wrong side (the right). This prevents a right-handed shooter from feeding the pig himself with his left hand and leaves the indestructible spent chain right in front of the shooter, or rather tangled around his feet the moment he has to get up and move which you have to do pretty much all the time because gun fights involve a lot more running around than shooting.

This little design whoopsie! has made be bust my face in the dirt or on the top of the gun more than once — not so convenient at interesting moments, and absolutely detrimental to my Cool Point count. Being at a tactical disadvantage and almost getting killed is one thing, but people actually saw that and it was embarrassing. Every. Goddamn. Time. It also hurts pretty bad (like, my feelings!).

But the failure of design doesn’t stop there. That stupid belt is nearly impossible to reload by hand without wearing gloves and using a lever to force the rounds into the thing. You have to at least find yourself some gloves and a stiff metal boxtop, wrench, ancient steel desk or something else firm that the butt of the round casings won’t slip too easily on to force those stupid rounds into their fully-enclosing holes. (And to you, the guy reading this who is thinking “I went to the range once and loaded a brand new, totally clean, uncorroded, never-been-fired-or-stepped-on belt — didn’t seem too hard to me”: sure, you might load 50 rounds into a pristine, lubricated belt by hand, but how about 5000 into a hundred chewed up belts?).

They also rust instantly, in accordance with the PKM Belt Rust Time Law: the time for a belt to rust to the point that it will malfunction if not serviced immediately, but not enough for it to be replaced by management will be exactly the amount of time since you last placed it in a sealed container and now. (Go check right now — you’ll see that this rule somehow always holds.) If you try oiling them to prevent that they gum up or actually start “growing hair” instantly. Its a never ending cycle of trying to keep the belts from making your life suck without giving up, throwing them all away and resolving to fight by using your bad breath and attitude alone.

This brings me to why the Soviets conveniently invented a reloading machine. Which also conveniently sucks. (Compare.) I can’t even begin to explain the inadequacy of this stupid machine, but it actually is the only way to maintain even a marginally reasonable reload rate for belts. There is, however, no way you could do this under fire. Or on Tuesday. (The machine jams spectacularly at random, Tuesday tending to be the worst day.)

I haven’t even begun to mention the inadequacy of the ammo crates. The standard ammo crates are insanely stupid. Actually, this isn’t a gripe reserved just for 7.62 ammo, its true for all commie ammo I’ve ever seen. The ammo cans aren’t like the infinitely reusable, universally useful, hermetically sealed, flip-top boxes found in non-backward armies. They are actually cans. Like giant spam cans, but without a pull-tab — not even a sardine-key. They come with a can opener. A huge one (but only one per crate, not one per can). You read that right, a can opener — and only one for the whoooole crate, so you better hope Jeeter doesn’t drop it in the canal.

You may be thinking “Oh, a can opener, I’ve got one of those that plugs into the wall, how could Jeeter drop such a thing in the canal?!” Glad you asked. When I say “can opener” I don’t mean the first-world type. I mean the part of your boyscout era Swiss Army Knife you never realized had an actual use. You know, the lever-kind where you hook the grabby part onto the crimp at the top edge of the can and pull to lever the pointy part down until it makes a tiny puncture, then slide over a touch and repeat until you’ve prized and ripped a gash large enough to do your business. Let that sink in.

(Here is a video of a guy opening one to illustrate… WTF. And that’s a really nice crate in pristine condition — which you will never, ever see while on contract in, say, Nigeria.)

Now remember, we’re talking about an ammo can. Like with bullets that people need to do their job, hopefully sometime this year, but perhaps much sooner under the sort of conditions that make calmly remembering details like where the fscking can opener is very difficult.

Once you’re inside the fun just doesn’t stop — no way. The thousand or so rounds inside are in boxes of 5 or 6 or so. Well, “boxes”, what I really mean is squarishly shaped topographic nets made of some material that tenuously holds together well enough to almost seem like a paper packing material — so it loosely resembles a little paper box. So this means that, unlike what you would have come to expect from armies in nations where maximizing busywork employment wasn’t the main and only goal, the can that you worked so hard to open isn’t full of pre-loaded belts. That would deprive someone of a government job somewhere and that’s just not Progressive. So inside there are dozens and dozens of those tiny, crappy, flimsy little cardboard boxes, each containing a few rounds. And before you start worrying that those stupid boxes are the end of the show, let me assure you that the party just keeps rolling along — each round is individually wrapped in tissue paper.

You just can’t make this shit up. Its amazing. How on earth could such a horrible, stupid, backward constellation of designs emerge from one of the two nations to achieve serious, manned spaceflight before the end of the 20th century? GHAH! Had WWIII ever occurred I wouldn’t have known whether to snicker and lay scunion, secure in my knowledge of how thoroughly the enemy had hamstrung himself, or feel pity and offer a face-saving peace deal to spare the poor saps who are forced to try to actually get anything done under these conditions.

But maybe its brilliant

A guy I worked with a few years ago called Mule had a theory that this was, in fact, an excellent design for a machine gun system in a Marxist paradise. His reasoning went something like this:

  • Nobody can use it alone, so you can’t get a wild hair up your ass and get all revolutionary — you need to convince at least a platoon to get crazy with you.
  • You employ a gazillion people not only in the production of a billion lovingly gift-wrapped-by-hand rounds of machine gun ammunition throughout the nation, you employ another gazillion or so to open and load the belts.
  • Its the ultimate low employment figure fixer — at least until the state digests enough of itself that this becomes suddenly unsustainable, of course (but that never stopped a socialist, despite the ever-lengthening list of failed socialist states).
  • You never really intended to go to actual war with the Americans in the first place (wtf, are you crazy?!?), so what you really needed was a police weapon of deliberately limited utility with which to suppress political dissent, not an actual infantry weapon of maximal utility under all conditions.

Mule’s theory was that this machine gun design — from the actual shittiness of the gun itself to the complete circus of activity which necessarily surrounds its production, maintenance and use — is a brilliant design from the perspective of the State, not the soldier. Furthermore, that the aims of the two are at odds is simply the natural outcome of being produced in a socialist/Marxist system. Mule was one of the most insightful people I’ve ever met (and I’m not being rhetorical — he really was a hidden genius).

Thinking about what he said has made me re-evaluate some of my assumptions of bad design. Perhaps the “bad” designs are excellent — not for the end user, but for whoever is in charge of the end user. And that brings me back to thinking about just why the Java programming language is so bad, yet so prolific, and how perhaps the design of Java is every bit as brilliant as the design of a good keyboard, differing in that each is brilliant from diametrically opposed points of view.

Java is the PKM of the programming world. Its everywhere, it sucks, it is good for some (Stalin-like) bosses, and the whole circus surrounding its existence just won’t ever go away. And sometimes those of us who know in painstaking detail why a 240 (or nearly anything else in common use) is better are still stuck using it to get real work done.

I should, perhaps, write another edition of this interdisciplinary mental expedition in text focused around the failings of JavaScript (oh, excuuuuuse me, princess, “ECMA Script”). Or Ruby. Or MySQL. Or SQL itself. Or HTTP. Come to think of it, subjects of my angry dissatisfaction abound.

Development Speed VS Quality

I’ve been working under some pretty insane time constraints lately. Two things jump out at me upon review of my work:

  1. I can, when cornered, churn out thousands upon thousands of lines of functioning code in a flash.
  2. The code works, but it is not particularly insightful or brilliant — it merely works.

The first bit is sort of cool: Typo Monster and Syntax Bear are no longer a part of my life (at least not in my Big Five). Nice.

On the other hand, the second bit isn’t so nice. It means that insightful development requires far more time than people tend to imagine. This has been a point of discussion for decades in engineering and especially software development, but its hard to fully appreciate until you get a chance to compare your own code, written once under duress, with code written for a similar problem domain at a pace that allows more time for grokness.

Now that I think of it, its not the pace exactly which is the problem, it is the way that certain forms of deadline and environmental pressures can re-tune the time budget in a way that extends typing/coding/input time at the expense of grok time.

This is particularly pronounced when it comes to data modeling — no matter if this means in the sense of managed state, a relational database schema, a no-schema blobbulation, a serialization scheme of some sort, or an object/class model.

This is a particularly touchy thing, since rearranging a data model of any sort entails major surgery when more than one thing relies on the way data is stored, but refactoring functions is relatively lightweight (so long as you understand what the intended mapping was to begin with).

I suspect that the vast majority of production code is written under deadline and that most of it relies on expedient and trashy data models. I further suspect that most corporate settings urge programmers to “be programming” and officially establish that “programming” means “typing” instead of “understanding”.

It is shocking to revisit something written in a hurry and simplify it in a flash of insight, and then contemplate what just happened. The surprise is all the more resonant once you fully realize how much less time is required to input insightful code than rushed, verbose-but-functional trash.

Looking to my left and right, it is not particularly evident that much production code benefits from a culture of sacred Grok Time.

More on a new Data Language

I’ve given more thought to the new data language I’m working on, and finally decided on a name as well.

The new baby shall be called RyuQ — a reference to where I live respelled to accommodate the mandatory “Q” in any shiny new query language. I’ve settled on a modified form of S-expressions, calling them SP-expressions to distinguish between real S-expressions and my mutated version. The (still infantile) language description is peppered with examples, so you can see pretty quickly if SP-expressions feel comfortable or more alien relative to SQL.

I have to say, the further I go on this the more I wonder what was going through the minds on the committee that created SQL. So many things are just so obvious when sticking to an algebraic notation rather than trying to form some new thing that is neither relational algebra nor relational calculus.

Until I have a complete implementation worked into a fork of Postgres I won’t be able to do anything but toy with this language — but the more thought I give it the less I am satisfied with the work I am currently compelled to do in SQL.