The Intellectual Wilderness There is nothing more useless than doing efficiently that which should not be done at all.

2020.12.26 16:19

Erlang: Writing a Tetris clone Part 3 – Gameplay rules, final features and deployment

Filed under: Computing,Games — Tags: , , , , , — zxq9 @ 16:19

The third video in this series moves on with implementation of gameplay rules, scoring, the “next” preview window and packaging and deploying the game using ZX as a shortcut.

Writing this is a lot of fun. As of this video the game part is finished, but there are a few things such as high score recording and maybe some network features that have not been implemented. I might make a fourth video that covers these but it might be more interesting to move on with another example to demonstrate techniques to accomplish similar tasks.

Quite a few details that were noticed and mentioned in the first two videos have been updated in the course of completing the code for this video, so it may be interesting to check the commit history if you’ve been following along.

As always, have fun making stuff!

2020.12.19 17:16

Erlang: Writing a Tetris clone Part 2 – Gameplay mechanics

Last night I was able to make the second video in my series about implementing a Tetris clone in Erlang. Yay!

In this video I start where I left off in the first video where I had ended with a data abstraction to represent the play field (called the “well” in Tetris lingo), a data abstraction for the game pieces, some colored sprites to draw the game board with, and a GUI that could draw a game board and a single random piece every time it was opened as well as print to stdout any key press events that I made.

Oh but by the way…

I have a small confession to make — I didn’t actually start where I left off. I mentioned the stopping point (the “Draw the board with a random piece on it” commit), then I mentioned the next thing I did which was implement basic movement (the “Basic (unsafe) movement” commit), then I completely blew past it and never explained the way I implemented unsafe movement in the first place. The trouble with having skipped that is that I had intended to discuss how keystroke capture actually works, where it is in the GUI code, and follow the event through the system so that people could get that idea into their head earlier than later because it is so basic to making a game!

So instead I’ll explain how that works and point out where it is in the code here in this post and cover it at the beginning of the third video.

Getting Input

If we look at the “Basic (unsafe) movement” commit there is a file called ertltris/src/et_gui.erl that is the code for the GUI process. In the init/1 function we see the wx server get started, a “frame” is created (the main window in wx parlance), some various widget elements and things are all established and on line 112 we see this:

ok = wxPanel:connect(Frame, char_hook),

This is connecting the Frame to a window manager event called char_hook. I do mention this in the video, but it is important to point this out here. I should also point out that I’m mistakenly calling wxPanel:connect/2 instead of wxFrame:connect/2 — which is technically incorrect, but due to the nature of the underlying inheritance among the C++ classes that make up wx and the way that’s all masked in the generated library wrappers that make up wxErlang, it actually doesn’t cause any errors. wxPanel and wxFrame are all ancestors of wxEventHandler.

Anyway… what connecting to this event does is tells wx that whenever the frame is the focus it should relay keystroke events to the program. Inside Erlang they arrive as messages that carry a #wx{} record that includes an event element that carries another record that provides all the relevant information about the event. You can receive these in the handle_event/2 callback function of wx_object. Pretty nice. This means you can deal with GUI events in very much the same way you can handle network events as well as inter-process messages within the Erlang runtime: everything is a message.


As I often say: Develop idioms.

A common idiom I use in wxErlang code is to match on an event type in handle_event/2, assign any relevant event data to variables, and then within a case inside the handle_event clause that matched that event type determine where we want to dispatch the event (if at all). You can see a very clear example of this on lines 191-203 of this commit.

handle_event(#wx{event = #wxKey{keyCode = Code}}, State = #s{frame = Frame}) ->
    ok =
        case Code of
             32 -> et_con:random_piece();
             88 -> et_con:rotate(l);
             90 -> et_con:rotate(r);
            314 -> et_con:move(l);
            315 -> et_con:rotate(r);
            316 -> et_con:move(r);
            317 -> et_con:move(d);
            _  -> tell("KeyCode: ~p", [Code])
    {noreply, State};

I could have made each one of those dispatch decisions inside the function head instead of using a case statement within a clause, but I find it much easier to read this particular style where we open a very compact dispatch span per event type we’re looking for than have a ton of handle_event clauses. In complex GUI applications you can wind up with a lot of special keystroke events and it becomes super cumbersome to match them all as function heads. Not just writing the function heads, but trying to find a specific event gets kind of messy because the code starts looking so scrambled.

Each of the different key codes matched in the dispatching part of that function correspond to some gameplay event. Note that the GUI doesn’t really care what the status of the game is at all, it just cares that it is relaying the mapped commands to the game control process and then carries on doing GUI stuff.

This is important to point out: Nearly all communication between the controller and the GUI is asynchronous. You almost never want blocking calls to pass between them. There may be a need for a blocking call in some special code, but this is almost always a bad idea.

Buh bye!

That pretty much sums up what I wanted to cover that I forgot to mention in the video. Input handling is really important. If you want to jump ahead, check out the way the handle_event/2 function evolved in the latest commits to see how menu commands are intercepted, or go check out the handle_event/2 function in the Erlang Minesweeper clone!

Don’t forget to give me all the delicious likes and stars and channel subs! BWAHAHA! I’ll catch you magnificent nerds in the next video!

2020.12.15 17:16

Erlang: Writing a Tetris clone Part 1 – Data types and basic GUI display

Over the last few days I wrote an Erlang implementation of the classic puzzle game Tetris and decided to do a few videos to explain how it works and why. This is part one of a three-part series.

  1. Data types and basic GUI display [this page]
  2. Gameplay mechanics
  3. Gameplay rules, final features, and deployment

This first video covers how I started the project (cheating using ZX to template a GUI project for me, of course), what data types I thought I would probably need from the outset (the game board and the game pieces), and my initial stab at getting pixels on the screen in a reasonably pretty way.

The videos roughly trace the commit history of the project as I developed it, with this first video covering the first 3 commits which get me from a GUI project template to displaying the game board and drawing a random piece in it each time I start the program (as well as catching keystrokes so I can examine what their values are).

2020.05.22 18:31

Getting started in DayZ Standalone v1.07

Filed under: Computing,Games — Tags: , , , , , , , — zxq9 @ 18:31

Happy free weekend! DayZ is be free to play from 2020-05-21 to 2020-05-25. What better way to spend your weekend while a pandemic sweeps the world than playing a post-apocalyptic zombie survival simulator… bwahahahahaha!

(Note: This is not a paid article and I have no connection to Bohemia Interactive — I just really like this game and wish more people understood what it’s about instead of suffering from the “expected Coke but got beer” shock and wind up missing out.)

Be warned: DayZ is hard mode all the way and you’ll die all the time at the beginning until you figure out how things work. This is actually hilarious, but not if you develop some pointless emotional investment in a freshly spawned character. Laugh at it. There aren’t any levels and you can gear back up pretty easily once you learn the basics of survival. Learning the basics of survival, though, can be hard. There is no manual (except for this post and the previous one) and the game is surprisingly realistic, making for an incredibly hostile environment (hence the “survival” part) and a near vertical learning curve.

This is a (relatively) quick guide on how to get your character from their initial hungry and thirsty starting condition to being well fed with a quenched thirst and hopefully enough gear and knowledge to start heading inland. A few critical details that directly affect survival have changed since previous versions of DayZ, so trying to follow guides for previous versions might leave you in even more shock than you’d already experience on your first visit to Chernarus (or especially Livonia).

First: Understand what DayZ by understanding what it isn’t

It isn’t an action shooter. It isn’t Rebel Fortress or Unreal Tournament or Battlefield. It isn’t WoW, either. It is none of those things. It is a hardcore zombie survival simulation. The all important noun there being “simulation” and the critical adjective being “hardcore”. I say “simulation” instead of “game” because there are no quests, no global map HUD, no global chat lobby, no ranking or guild system, no equipment “bank”, no auction house, none of that stuff. Comparing DayZ to an action shooter is like comparing a real flight simulator to Gradius III.

Your DayZ character must eat, stay hydrated, keep warm, avoid illness if possible, etc. There are 900+ items in v1.07 that actually have a function — so be mentally prepared for some things you didn’t think were possible or sometimes need to do things you’re surprised are necessary. Suffice to say there is a ton to learn and a single short guide like this written on the occasion of a free play weekend on Steam isn’t going to cover it all.

Picking a server

The first thing that trips up the freshest of fresh freshies is picking the wrong server. The basic mechanic is simple, you press “play” and the launcher links in whatever mods you’ve selected (if any) and launches the game. From there you will want to select “change server” and sort by ping to find one that has the lowest possible ping to your location. Anything under 200ms is workable (though not ideal) for PvE, and anything under 100ms is workable for PvP, though 50ms or less is best no matter what you’re doing.

In the server selection screen you’ll see at the top “Official | Community | Private”. It isn’t obvious to everyone at first glance that this is a clickable control (UI fail) but it is. If you can’t find a nearby official server, click on “Community” and sort by ping.

Example: I live in Japan. The official servers with the best ping times are in LA with around 130~250ms ping times from my location. That’s a bit annoying (but workable) for PvE, but hopeless for PvP. There are community servers local to me, though, with 30ms ping times (and fewer regional spoken language/accent issues). Note that with community servers you might pick one that has a great ping time but has mods installed. You’ll need to install and activate the same mods as the server to join it (mods are allowed to change the game in arbitrary ways, so the server and the client need to be on the same page). Starting out with vanilla DayZ is my recommendation your first time around, though.

With that out of the way, let’s get on to survival.

Hello, Freshie! A bit peckish?

You just spawned. Your character starts out with a sated appetite and hydrated, but that will change very soon. You’re a little more than halfway to getting hungry when you start out and you’ll get thirsty almost as soon as you start jogging somewhere (and jogging is totally OK — you don’t have to walk everywhere, but sprinting will wear you out and waste lots of precious water). You will have an apple or pear in your inventory to start with. Various types of food and drink provide both hydration as well as energy, water being the only one that provides only hydration and no extra nutrition.

Eat something

There is no manual (but I have made a cheat sheet for the controls), so it is useful to note that pressing <TAB> toggles your inventory screen. You can drag/drop things here. The square below your character’s image represents whatever he’s carrying. Drag a fruit to it, then press <TAB> again to get out of the inventory screen. You’ll notice a tool tip that has the left mouse button highlighted that says “to eat [HOLD]” or something similar, and this means if you hold the left mouse button down your character will start eating the fruit in his hand. Try it as soon as you spawn. It isn’t a big deal to lose your starting fruit, and that water and energy do more good in your body than in your pocket (infantry rules apply!).

Manipulating inventory

Inventory and item manipulation is pretty intuitive. Mess around with it and you’ll figure out what’s going on quickly enough.

The only counter-intuitive point is that the “tacky” point of a dragged object is the upper left corner of the item and that will be dictated by where the mouse pointer is, not necessarily where the item’s outline appears to be. Mess around with it and this will become obvious quickly. Also, note that items in inventory may be oriented the wrong way to fit, even if you have enough slots available — press the spacebar to rotate them while selected so they fit.

When you’re near a container (or a body) you will see those appear as inset containers in the “Vicinity” area on the left side of the screen. This is how you get stuff off the ground, out of containers, and off of bodies (of zombies, dead players, or unconscious saps that have been knocked out).


Now that you’ve had something to eat and if you haven’t been killed yet you should take your bearings (and if you did already die for whatever reason, just respawn and try again — it’s not a big deal). There is no map in the game interface but the environment is accurate enough that you can get your bearings from observing your surroundings. (There are maps in the game that your character can find, but nothing that is magically visible to the player).

The sun rises in the east and sets in the west, just like you would expect. If you can tell whether it is morning or evening you can know by the shadows which direction you’re moving. If it is morning and the sun is to your right (shadows point to the left) then you’re facing north, and this reverses in the afternoon. If shadows are super short (directly under trees) then it’s about mid-day and the short shadow tip is pointing north (or sometimes not quite visible).

The stars in the sky are actually placed accurately, and this makes the night sky a gigantic, free compass, just like in real life (Polaris, the North Star, is prominently visible and easy to navigate by on nights that aren’t overcast). On cloudless nights the light outside is actually quite good enough to see your immediate surroundings, so don’t be afraid to turn off any lights or extinguish any torches and navigate. Sometimes it’s much easier to see distance this way and is far less likely to give your position away to any zombies. Use the stars! If it is raining out and you don’t have a full rain suit, though, stay indoors for the night periods.

Time moves pretty quickly in the game (of course), with days seeming to be significantly longer than nights (perhaps this is on a planet with two suns? no idea). We’ll just handwave that away, but keep in mind that night isn’t to be dreaded and it doesn’t last too long.

There are billboards with maps with a big red “You are Here!” dot on them, usually somewhat close to a bus stop area out on the coast, or sometimes near a water pump in a larger town. There are also military and hiking maps you can find as loot drops that are extremely detailed. Get familiar with the map. If you do a web search for “DayZ map” you’ll find many nice ones, even interactive maps that have spawn locations listed. It is important to get the basic map in your head and figure out where you are as soon as you can.

The general layout of Charnarus is that the east and south are relatively straight coastlines, and this is the lower-risk/lower-reward area people spawn. As one heads northwest loot drops improve (military gear, more car chassies and parts, etc) and risk increases (to include hostile wildlife like wolf packs and bears, in addition to PvP-minded players looking for those sweet, sweet M4s and bodyarmor…).

Spend your time near the coast gearing for survival and then move inland to gear up for combat, base building, camping out, linking up with people, going cannibal, or whatever else it is you might want to do in the game world.

Survival priorities

You need to stay hydrated, eat well, keep warm, don’t get sick, and keep the red stuff in your body (don’t bleed out!).

To stay hydrated drink and eat.

To stay well fed eat and drink.

To keep warm wear well-maintained clothes and stay dry.

To stop bleeding when you’re hurt use rags or bandages to bind your wounds (put them in your hands and hold the left mouse button). You can have multiple wounds, so you may need to do this several times in succession (the “I’m cut” icon has a number on it representing how many open wounds you have).

To keep from getting sick don’t eat rotten food, don’t drink untreated water from rivers (or the ocean), and stay warm. To increase your immune system and reduce the recovery period for illnesses take vitamins if you can find some.

Survival tools: Water containers

If you can find a few water bottles, canteen, or a cooking pot, you’re set! You will eventually run across these. Almost every town has a water pump in it, at which you can fill your water container. These can be reused, so hold on to them after they are empty.

Survival tools: Bandages and rags

To make more rags (useful for more than just bandages) you can cut up clothing items you find as long as you have an edged tool in your hands. Do this frequently.

Survival tools: A trusty sharp thing

Tools are more important than weapons, and most heavy tools are very effective weapons anyway.

The biggest priority is having a blade of some sort. You need an edged tool to open canned food (or else you can’t eat it), strip bark from trees to use as kindling for starting fires, to craft fishhooks from bones, etc. The better suited the edged tool is to the job you’re doing the better it will work. For example, an improvised stone knife can be used to open a can of food, but you’ll spill 20~40% of it in the process because it’s sloppy. An axe is much better, and an actual can opener is perfect every time.

If you see a shovel, axe, hatchet, or pickaxe: GET IT

If you see a knife: GET IT

If you can’t find a knife, make one with two small stones. You will find these on the ground if you run down any forest trail or along the train tracks. Place one in your hand, then put another on in your hand (in the inventory screen you’ll get the option “combine” or “replace”, pick “combine), and then use the action button to craft an improvised knife. Stone knives wear out really fast, but they are free. They can get you by until you find something better.

Survival tools: Give a man a can of tuna, feed him for 12 hours; teach a man to fish, feed him until the bears show up

Besides a bladed tool and maybe something heavy to kill zombies with (can be the same: splitting axe, pickaxe, etc.), the next thing you need is a reliable source of food. At the very outset you can scrounge for canned food and soda in houses by the coast or killing zombies and looting their bodies, but eventually you’ll want to get into fishing and hunting.

Fishing is much lower risk than hunting, but you’ll need a fishing pole. Find a bush and get near it. You’ll see the option to cut down the bush by using the action button. This will happen in two stages, first you’ll strip a “long stick” off it, and if you keep going you’ll knock the whole thing over leaving some more sticks (there are a few varieties of this depending on the type of bush or small tree you target and whether you have a tool or are using your hands — experiment). You want the long stick. Now find a rope or make one if you have enough rags. Combine the stick with the rope and viola! we have a fishing pole. (Note: This is also the way bows are made, but they are disabled in v1.07, but will be returning soon.)

Now you need a fishing hook and some bait. To get fishing hooks you can either find them in random loot (I seem to never find them) or, more reliably, make some yourself out of bones. On the coast there are many places you’ll find chickens wandering around. Listen for them. Follow the sound the same way zombies follow you and SMACK! Kill yourself a chicken. Put a knife in your hand (or make a stone one if you haven’t found one yet) and butcher the chicken. You’ll get a few pieces of glorious chicken and a pile of bones. With your trusty knife in hand you can work those bones into fishhooks!

Hurray! Now the last thing to get is some bait. If you find some soft dirt, put a knife, shovel or pickaxe in your hand and you’ll see the option to “dig for worms”. Give that a try a few times. Once you have some worms, head over to the ocean or a river or lake (if you’re already inland) and give fishing a try. Fish provide a lot of nutrition, are unlimited in supply, and are safe to handle (unlike hunting wolves and bears).

How to combine worms with the fishing pole is not as obvious as it should be. Here’s a step-by-step (assuming you already have a pole, a hook and a worm):

  1. Remove the hook from the pole.
  2. Put the hook in your hand.
  3. Drag the worm to the hook to “combine”.
  4. Exit the inventory menu.
  5. Hold the left mouse button to “make fishing bait”.
  6. Once the fishing bait is complete, pick it up.
  7. Equip the pole.
  8. Add the baited hook to the pole.
  9. Go near the water (sometimes you need to look down toward it) and click and hold the left mouse button until you’ve caught something (hopefully not a pair of Wellies).

It goes a bit faster if you get a few hooks and worms ready all at once, and then commence fishing. This procedure should, in my opinion, be shortened to where you can combine a worm directly with a hook-equipped pole, but that’s not how it works at the moment. In any case, enjoy fishing! Fish are a super food, and after you prepare it with your knife it doesn’t even need to be cooked to be eaten.

Survival technique: Jacking zombies

Once you have a heavy thingy (sledgehammer, pickaxe, splitting axe, etc.) or pointy thingy (combat knife, spear, etc.) or just a bit of experience punching zombie heads in, you’re ready to jack zombies for loot, to clear routes, and for pleasure.

If you don’t sprint or make loud noises zombies generally won’t notice you until you get quite close. If you’re listening for them (use headphones) you’ll notice them long before they notice you. If you see a single zombie you can approach it from the rear or side at a sprint before it can react. As you near it, ready a blow with your weapon or fists and smack that sucker in the noggin until he goes down, then take whatever he has on him. You can find quite a lot of early loot this way (can openers, flashlights, soda, water, canned food, etc.).

Just don’t get cocky when a bunch of them are around. A single zombie isn’t hard to take down at all, even unarmed, but two gets interesting unless you have a weapon, and three or more can be the end of you if you get unlucky.

Whatever you do, don’t shoot in town unless you have a clear and immediate route out of there. A common beginner experience is “Oh, two zombies, I have a gun, simple [BANG!] … [5 seconds later] ZOMG WTF 15 ZOMBIES!” You’ll do it eventually. It might even be a good learning experience to try it on purpose just to see how fast things can get messed up when you’re wandering around alone!

Paramount to survival: Mindset

This is a toy. Remember that. You will die. Mess around the coast a bit and use your first few incarnations to figure out how things work. Familiarize yourself with crafting knives, fishing poles, finding and butchering chickens, fishing, getting solid shots with an axe on inattentive zombies, building fires (need wood, 2 or more rocks, and kindling (paper, tree bark, rags), and a firestarter (matches or a lighter)), basic cooking (need some meat and a long stick or a cooking pot), etc.

Don’t get invested in your character, get invested in the experience. When you die it doesn’t take long to gear up to survival level again, because “survival level” is really a skill not a piece of gear.

Once you are over the survival hump you’ll start meeting players and having wild experiences. Some people will kill you on sight (aka “KOS”, generally a jerk move, but lots of people think this is an action shooter because that’s been the way to make AAA titles without innovating or having any new ideas for the last 30 years).

Have fun out there, use voice chat with other players when you meet them, and always look on the bright side of life.

DayZ SA v1.07 Controls Cheatsheet

Filed under: Computing,Games — zxq9 @ 16:55

Just to make DayZ even more hardcore than it already is there is no manual and no tutorial area to cut your teeth. Instead, you just get your teeth smashed out and die until you figure out what’s going on (then you die in more interesting ways). Here is a cutdown of the basic default controls in the game.

For a more complete reference check your options menu.

Control Table (defaults)

MouseCharacter’s bearing
ALT/*Free look
V/ENTERToggle 1st/3rd person view (if 3rd is allowed)
W/A/S/DCharacter’s direction (foward/back and strafe left/right)
Q/ELean left/right
L CTRLWalk/Jog (hold for momentary toggle, double-tap to toggle)
L SHIFTSprint (hold for momentary, no toggling)
SPACEJump (character may use hands to assist)
CTap to toggle stand/crouch (hold to lay prone)
TABInventory screen
R MouseHold to raise/employ held item (raise gun, fists, etc.)
L MouseMain action (fire weapon, punch, select/drag item, etc.)
M MouseZoom (employs optics/weapon sights if at the ready)
FAlternate action (buttstroke with weapon, open door, etc.)
GDrop or throw held item
RChamber a round/cycle the action (clear a jam)
XToggle held weapon’s firing mode (semi/full auto)
PGUPElevate weapon sights (adjust zero up) if possible
PGDNLower weapon sights (adjust zero down) if possible
LActivate night vision/special optics
L CTRLHold breath (reduces aiming float)
(spacebar)Rotate selected/dragged item in inventory
` (backtick)Toggle item/action hotkey in HUD
09Item/action hotkeys (assignable in game)
Cycle voice volume (whisper/talk/shout)
/Text chat (you should really use voice if possible)
F1F11Gestures (play with this to learn them)
F12Screenshot (Steam default)


When driving vehicles the WASD movement controls work as expected, the mouse looks around, Q/E shift gears down/up respectively, and L Shift activates turbo (same as sprinting).


Many items in the game can affect other items or be used for crafting. To craft a stone knife from two small stones, for example:

  1. Open the inventory screen and place one in your hands (drag to the “held” box below the character)
  2. Drag the other one to the “held” box — the right side will say “replace”, the left will say “combine”. Drop it into “combine”.
  3. Exit the inventory screen and you will see the tool-tip tell you to press and hold the left mouse button (or “F” key) to craft a stone knife.

In the same way, fast loading of magazines can be accomplished by placing a magazine in your hands, then dragging the appropriate ammunition to “combine” and then holding the action button to pump rounds into it without dragging individual rounds in.


To eat an item, hold it in your hand. If it is a canned item you will need to open it before eating it. If it doesn’t have an easy opener you’ll need a can opener (best) or something with an edge like a knife (not too bad), an axe (spills quite a bit), or improvised stone knife (spills up to 40% of the contents, but better than starving).


To loot a container (locker, barrel, backpack/clothes on the ground, etc.) or a body (dead zombie or dead/unconscious player, etc.) simply stand near it and activate the inventory screen. You can freely move items around as long as they fit.

2020.01.29 17:27

Erlang: Minesweeper

Filed under: Computing,Games — Tags: , , , , , — zxq9 @ 17:27

I’ve been sick the last two days and utterly uninspired to do anything productive. I’ve instead procrastinated by writing a “minesweeper” clone in Erlang.

Why? I have no idea. I was just sort of thinking of simple desktop classics to mess around with that are de-facto standard to populate a GUI app launcher like Vapor… and several hours later I had this thing. By that point I figured I was invested enough to swap text for graphics, and poof! There we are.

I still need to add win/loss conditions, a wall-clock timer and some kind of score thingy, but anyway, this was actually a much more fun way to tool around on a sick day than I expected and makes me feel just barely less of a dirtbag than I would have been had I wrapped up in bed all day feeling crappy.

I hate being sick. Ugh.


I went ahead and finished it (except for recording scores — does anyone ever look at that since they can’t be sanely shared and aggregated?) and put it on gitlab just in case someone wants to see what a really hasty/disorganized codebase looks like.

It even has settings! Hahaha! “Settings” really being code for me messing around and seeing if I remembered how wxSlider widgets work (turns out I do and they are boringly easy to use).

If I find myself not feeling in the mood and going to the gym is out of the question, I suppose I could do one of these in 3D next time. Seems like “minesensor” would be a slightly more involved sort of game.

2019.06.20 12:44

You don’t have time to play games

Filed under: Games,Science & Tech,Society — Tags: , , , , — zxq9 @ 12:44

There is only one way to develop graduate-level expertise in more than one narrow specialty: self education.

The time commitment required is larger than most people are willing to consider — so large that the people who tend to apply themselves to the degree required don’t notice they are doing it.

2016.03.25 15:05

The basic Erlang service ⇒ worker pattern

There has been some talk about identifying “Erlang design patterns” or “functional design patterns”. The reason this sort of talk rarely gets very far is because generally speaking “design patterns” is a phrase that means “things you have to do all the time that your language provides both no primitives to represent, and no easy way to write a library function behind which to hide an abstract implementation”. OOP itself, being an entire paradigm built around a special syntax for writing dispatching closures, tends to lack a lot of primitives we want to represent today and has a litany of design patterns.

NOTE: This is a discussion of a very basic Erlang implementation pattern, and being very basic it also points out a few places new Erlangers get hung up, like the context in which a specific call is — because that’s just not obvious if you’re not already familiar with concurrency at the level Erlang does it. If you’re already a wizard, this article probably isn’t for you.

But what about Erlang? Why have so few design patterns (almost none?) emerged here?

The main reason is that what would have been design patterns in Erlang have mostly become either functional abstractions or OTP (here “OTP” refers to the framework shipped with Erlang). This is about as far as the need for patterns has needed to go in the most general case. (Please note that it very often is possible to write a framework that implements a pattern, though it is very difficult to make such frameworks completely generic.)

But there is one thing the ole’ Outlaw Techno Psychobitch doesn’t do for us that quite a few of us do have a common need for but we have to discover for ourselves: how to create a very basic arrangement of service processes, supervisors, and workers that spawn workers according to some ongoing global state or node configuration. (Figuring this out is almost like a rite of passage for Erlangers — and often even experienced Erlangers have never distilled this down to a pattern, even if many projects do eventually evolve into something structured similarly.)

The case I will describe below involves two things:

  • There is some service you want to create that is represented by a named process that manages it and acts as its sole interface. Higher-level code in the system doesn’t want to call low-level code to get things done, the service should know how to manage itself.
  • There is some configurable state that is relevant to the service as a whole, should be remembered, and you should not be forced to pass in as arguments every time you call for this work to be done.

For example, let’s say we have an artificial world written in Erlang. Let’s say its a game world. Let’s say mob management is abstracted behind a single mob manager service interface. You want to spawn a bunch of monster mobs according to rules such as blahlblahblah… (Who cares? The game system should know the details, right?) So that’s our task: spawning mobs. We need to spawn a bunch of monster mob controller processes, and they (of course) need to be supervised, but we shouldn’t have to know all the details to be able to tell the system to create a mob.

The bestiary is really basic config data that shouldn’t have to be passed in every time you call for a new monster to be spawned. Maybe you want to back up further and not even want to have to specify the type of monster — perhaps the game system itself should know what the correct spawn/live percentages are for different types of mobs. Maybe it also knows the best way to deal with positioning to create a playable density, deal with position conflicts, zone conflicts, leveling or phasing influences, and other things. Like I said already: “Who cares?”

Wait, what am I really talking about here? I’m talking about sane defaults, really. Sane defaults that should rule the default case, and in Erlang that generally means some sane options that are comfortably curried away in the lowest-arity calls to whatever the service functions are. But from whence come these sane defaults? The service state, of course.

So now that we have our scenario in mind, how does this sort of thing tend to work out? As three logical components:

  • The service interface and state keeper, let’s call it a “manager” (typically shortened to “man”)
  • The spawning supervisor (typically shortened to “sup”)
  • The spawned thingies (not shortened at all because it is what it is)

How does that typically look in Erlang? Like three modules in this imaginary-but-typical case:

  • game_mob_man.erl
  • game_mob_sup.erl
  • game_mob.erl

The game_mob_man module represents the Erlang version of a singleton, or at least something very similar in nature: a registered process. So we have a definite point of contact for all requests to create mobs: calling game_mob_man:spawn_mob/0,1,... which is defined as

spawn_mob() ->

spawn_mob(Options) ->
    gen_server:cast(?MODULE, {beget_mob, Options}).

Internally there is the detail of the typical

handle_cast({beget_mob, Options}, State) ->
    NewState = beget_mob(Options, State),
    {noreply, NewState};

and of course, since you should never be putting a bunch of logic or side-effecty stuff in directly in your handle_* function clauses beget_mob/2 is where the work actually occurs. Of course, since we are talking about common patterns, I should point out that there are not always good linguistic parallels like “spawn” ⇔ “beget” so a very common thing to see is some_verb/N becomes a message {verb_name, Data} becomes a call to an implementation do_some_verb(Data, State):

spawn_mob(Options) ->
    gen_server:cast(?MODULE, {spawn_mob, Options}).


handle_cast({spawn_mob, Options}, State) ->
    NewState = do_spawn_mob(Options, State),
    {noreply, NewState};

% ...

do_spawn_mob(Options, State = #s{stuff = Stuff}) ->
    % Actually do work in the `do_*` functions down here

The important thing to note above is that this is the kind of registered module that is registered under its own name, which is why the call to gen_server:cast/2 is using ?MODULE as the address (and not self(), because remember, interface functions are executed in the context of the caller, not the process defined by the module).

Also, are the some_verb/N{some_verb, Data}do_some_verb/N names sort of redundant? Yes, indeed they are. But they are totally unambiguous, inherently easy to grep -n and most importantly, give us breaks in the chain of function calls necessary to implement abstractions like managed messaging and supervision that underlies OTP magic like the gen_server itself. So don’t begrudge the names, its just a convention. Learn the convention so that you write less annoyingly mysterious code; your future self will thank you.

So what does that have to do with spawning workers and all that? Inside do_spawn_mob/N we are going to call another registered process, game_mob_sup. Why not just call game_mob_sup directly? For two reasons:

  1. Defining spawn_mob/N within the supervisor still requires acquisition of world configuration and current game state, and supervisors do not hold that kind of state, so you don’t want data retrieval tasks or evaluation logic to be defined there. Any calls to a supervisor’s public functions are being called in the context of the caller, not the supervisor itself anyway. Don’t forget this. Calling the manger first gives the manager a chance to wrap its call to the supervisor in state and pass the message along — quite natural.
  2. game_mob_sup is just a supervisor, it is not the mob service itself. It can’t be. OTP already dictates what it is, and its role is limited to being a supervisor (and in this particular case of dynamic workers, a simple_one_for_one supervisor at that).

So how does game_mob_sup look inside? Something very close to this:


%%% Interface
spawn_mob(Conf) ->
    supervisor:start_child(?MODULE, [Conf]).

%%% Startup
start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    RestartStrategy = {simple_one_for_one, 5, 60},
    Mob = {game_mob,
           {game_mob, start_link, []},
    Children = [Mob],
    {ok, {RestartStrategy, Children}}.

(Is it really necessary to define these things as variables in init/1? No. Is it really necessary to break the tuple assigned to Mob vertically into lines and align everything all pretty like that? No. Of course not. But it is pretty darn common and therefore very easy to catch all the pieces with your eyes when you first glance at the module. Its about readability, not being uber l33t and reducing a line count nobody is even aware of that isn’t even relevant to the compiled code.)

See what’s going on in there? Almost nothing. That’s what. The interesting part to note is that very little config data is going into the supervisor at all, with the exception of how supervision is set to work. These are mobs: if they crash they shouldn’t come back to life, better to leave them dead and signal whatever keeps account of them so it can decide what to do (the game_mob_man, for example, which would probably be monitoring these). Setting them as permanent workers can easily (and hilariously) result in a phenomenon called “highly available mini bosses” — where a crash in the “at death cleanup” routine or the mistake of having the mob’s process retire with an exit status other than 'normal' causes it to just keep coming back to life right there, in its initial configuration (i.e. full health, full weapons, full mana, etc.).

But what stands above this? Who supervises the supervisor?

Generally speaking, a component like mob monsters would be a part of a larger concept of world objects, so whatever the world object “service” concept is would sit above mobs, and mobs would be one component of world entities in general.

To sum up, here is a craptastic diagram:

Yes, my games involve wildlife and blonde nurses.

Yes, my games involve wildlife and blonde nurses.

The diagram above shows solid lines for spawn_link, and dashed lines to indicate the direction of requests for things like spawn_link. The diagram does not show anything else. So monitors, messages, etc. are all just not there. Imagine them. Or don’t. That’s not the point of this post.

“But wait, I see what you did there… you made a bigger diagram and cut a bunch of stuff out!”

Yep. I did that. I made an even huger, much crappier, more inaccurate diagram because I wasn’t sure at first where I wanted to fit this into my imaginary game system.

And then I got carried away and diagrammed a lot more of the supervision tree.

And then I though “Meh, screw it, I’ll just push this up to a rough imagining of what it might look like pushed all the way back to the SuperSup”.

Here is the result of that digression:

It wouldn't look exactly like this, so use your imagination.

It wouldn’t look exactly like this, so use your imagination.


Yep. All that. Right there. That’s why its called a “supervision tree” instead of a “supervision list”. Any place in there you don’t have a dependency between parts, a thing can crash all by itself and not bring down the system. Consider this: the entire game can fail and chat will still work, users will still be logged in, etc. Not nearly as big a deal to restart just that one part. But what about ItemReg? Well, if that fails, we should probably squash the entire item system (I’ve got guns, but no bullets! or whatever) because game items are critical data. Are they really critical data? No. But they become critical because gamers are much more willing to accept a server interruption than they are losing items and having bad item data stored.

And with that, I’m out! Hopefully I was able to express a tiny little bit about one way supervision can be coupled with workers in the context of an ongoing, configured service that lives within a larger Erlang system and requires on-the-fly spawning of supervised workers.

(Before any of you smarties that have been around a while and point out how I glossed over a few things, or how spawning a million items as processes might not be the best idea… I know. That’s not the point of this post, and the “right approach” is entirely context dependent anyway. But constructive criticism is, as always, most welcome.)

2014.11.12 21:23

Why OTP? Why “pure” and not “raw” Erlang?

I’ve been working on my little instructional project for the last few days and today finally got around to putting a very minimal, but working, chat system into the ErlMUD “scaffolding” code. (The commit with original comment is here. Note the date. By the time this post makes its way into Google things will probably be a lot different.)

I commented the commit on GitHub, but felt it was significant enough to reproduce here (lightly edited and linked). The state of the “raw Erlang” ErlMUD codebase as of this commit is significant because it clearly demonstrates the need for many Erlang community conventions, and even more significantly why OTP was written in the first place. Not only does it demonstrate the need for them, the non-trivial nature of the problem being handled has incidentally given rise to some very clear patterns which are already recognizable as proto-OTP usage patterns (without the important detail of having written any behaviors just yet). Here is the commit comment:

Originally chanman had been written to monitor, but not link or trap exits of channel processes [example]. At first glance this appears acceptable, after all the chanman doesn’t have any need to restart channels since they are supposed to die when they hit zero participants, and upon death the participant count winds up being zero.

But this assumes that the chanman itself will never die. This is always a faulty assumption. As a user it might be mildly inconvenient to suddenly be kicked from all channels, but it isn’t unusual for chat services to hiccup and it is easy to re-join whatever died. Resource exhaustion and an inconsistent channel registry is worse. If orphaned channels are left lying about the output of \list can never match reality, and identically named ones can be created in ways that don’t make sense. Even a trivial chat service with a tiny codebase like this can wind up with system partitions and inconsistent states (oh no!).

All channels crashing with the chanman might suck a little, but letting the server get to a corrupted state is unrecoverable without a restart. That requires taking the game and everything else down with it just because the chat service had a hiccup. This is totally unacceptable. Here we have one of the most important examples of why supervision trees matter: they create a direct chain of command, and enforce a no-orphan policy by annihilation. Notice that I have been writing “managers” not “supervisors” so far. This is to force me to (re)discover the utility of separating the concepts of process supervision and resource management (they are not the same thing, as we will see later).

Now that most of the “scaffolding” bits have been written in raw Erlang it is a good time to sit back and check out just how much repetitive code has been popping up all over the place. The repetitions aren’t resulting from some mandatory framework or environment boilerplate — I’m deliberately making an effort to write really “low level” Erlang, so low that there are no system or framework imposed patterns — they are resulting from the basic, natural fact that service workers form constellations of similarly defined processes and supervision trees provide one of the only known ways to guarantee fallback to a known state throughout the entire system without resorting to global restarts.

Another very important thing to notice is how inconsistent my off-the-cuff implementation of several of these patterns has been. Sometimes a loop has a single State variable that wraps the state of a service, sometimes bits are split out, sometimes it was one way to begin with and switched a few commits ago (especially once the argument list grew long enough to annoy me when typing). Some code_change/N functions have flipped back and forth along with this, and that required hand tweaking code that really could have been easier had every loop accepted a single wrapped State (or at least some standard structure that didn’t change every time I added something to the main loop without messing with code_change). Some places I start with a monitor and wind up with a link or vice versa, etc.

While the proper selection of OTP elements is more an art than a science in many cases, having commonly used components of a known utility already grouped together avoids the need for all this dancing about in code to figure out just what I want to do. I suppose the most damning point about all this is that none of the code I’ve been flip-flopping on has been essential to the actual problem I’m trying to solve. I didn’t set out to write a bunch of monitor or link or registry management code. The only message handler I care about is the one that sends a chat message to the right people. Very little of my code has been about solving that particular problem, and instead I consumed a few hours thinking through how I want the system to support itself, and spent very little time actually dealing with the problem I wanted to treat. Of course, writing this sort of thing without the help of any external libraries in any other language or environment I can think of would have been much more difficult, but the commit history today is a very strong case for making an effort to extract the common patterns used and isolate them from the actual problem solving bits.

The final thing to note is something I commented on a few commits ago, which is just how confusing tracing message passage can be when not using module interface functions. The send and receive locations are distant in the code, so checking for where things are sent from and where they are going to is a bit of a trick in the more complex cases (and fortunately none of this has been particularly complex, or I probably would have needed to write interface functions just to get anything done). One of the best things about using interface functions is the ability to glance at them for type information while working on other modules, use tools like Dialyzer (which we won’t get into we get into “pure Erlang” in v0.2), and easily grep or let Emacs or an IDE find calling sites for you. This is nearly impossible with pure ad hoc messaging. Ad hoc messaging is fine when writing a stub or two to test a concept, but anything beyond that starts getting very hard to keep track of, because the locations significant to the message protocol are both scattered about the code (seemingly at random) and can’t be defined by any typing tools.

I think this code proves three things:

  • Raw Erlang is amazingly quick for hacking things together that are more difficult to get right in other languages, even when writing the “robust” bits and scaffolding without the assistance of external libraries or applications. I wrote a robust chat system this afternoon that can be “hot” updated, from scratch, all by hand, with no framework code — that’s sort of amazing. But doing it sucked more than it needed to since I deliberately avoided adhering to most coding standards, but it was still possible and relatively quick. I wouldn’t want to have to maintain this two months from now, though — and that’s the real sticking point if you want to write production code.
  • Code convention recommendations from folks like Joe Armstrong (who actually does a good bit of by-hand, pure Erlang server writing — but is usually rather specific about how he does it), and standard set utilities like OTP exists for an obvious reason. Just look at the mess I’ve created!
  • Deployment clearly requires a better solution than this. We won’t touch on this issue for a while yet, but seriously, how in the hell would you automate deployment of a scattering of files like this?

Powered by WordPress