Tag Archives: strong types

Pure Declarations in Erlang

Over the last year or so I’ve gone back and forth in my mind and in discussions with other Erlangers about type systems in Erlang, or rather, I’ve been going back and forth about its lack of one and the way Dialyzer acts as our bandaid in this area. Types are useful enough that we need Dialyzer, but the pursuit of functional puritanism gets insane enough that its simply not worth it in a language intended for real-world production use, especially in the messy, massively concurrent, let-it-crash, side-effecty, message-centric world of Erlang.

But… types and pure functions are still really useful and setting a goal of making as much of a program as possible into provable, bounded, typed, pure functions tends to result in easy to understand, test and maintain code. So there is obviously some stress here.

What I would like to do is add a semantic that the compiler (or Dialyzer, but would prefer this be a compiler check, tbh) be aware of what functions are pure and which are not. The way I would do this is by using a different “arrow”, in particular the Prolog-style declaration indicator: :-

[Edit after further discussion…] What I would like to do is add a directive that Dialyzer can interpret according to a simply purity rule. Adding this to Dialyzer makes more sense than putting it in the compiler — Dialyzer is already concerned with checking; the compiler is already concerned with compiling.

The directive would be -pure(Name/Arity) (a compliment to -spec). The rule would be very simple: only guard-permissible BIFs and other pure functions are legal from within the body of a pure function. This is basically just an extension of the current guard rule (actually, I wonder why this version isn’t already the guard rule… other than the fact that unless something like this is implemented the compiler itself wouldn’t have any way of checking for purity, so currently it must blindly accept a handful of BIFs known to be pure and nothing else).

For example, here is a pure function in Erlang, but neither the compiler nor Dialyzer can currently know this:

-spec increment(integer()) -> integer().
increment(A) ->
    A + 1.

Here is the same function declared to be pure:

-pure(increment/1).
-spec increment(integer()) -> integer().
increment(A) ->
    A + 1.

Pretty simple change.

“ZOMG! The whold standard library!” And yes, this is true — the whole thing is out. Except that the most important bits of it (the data structures like lists, dict, maps, etc.) could be easily converted to pure functions with little more than changing -> to :- adding a single line to the definition.

Any pure function could be strongly typed and Dialyzer could adhere to strong types instead of looser “success types” in these cases. Some code that is currently written to take an input from a side-effecty function, pass it through a chain of non-returning and possible side-effecty functions as a way to process or act on the value, and ultimately then call some side-effecty final output function would instead change to a form where the side-effects are limited to a single function that does both the input and output, and all the processing in-between would be done in pure functions.

This makes code inherently more testable. In the first case any test of the code is essentially an integration test — as to really know how things will work requires knowing at least one step into side effects (and very often we litter our code with side-effects without a second thought, something prayer-style monadisms assist greatly with). In the second case, though, the majority of the program is pure and independently testable, with no passthrough chain of values that have to be checked. I would argue that in many cases such passthrough is either totally unnecessary, or when it really is beneficial passing through in functions is not as useful as passing through in processes — that is to say, that when transformational passthrough is desired it is easier to reason about an Erlang program as a series of signal transformations over a message stream than a chain of arbitrarily side-effecty function calls that collectively make a recursive tail-call (and that’s a whole different ball of wax, totally orthogonal to the issue of functional purity).

Consider what we can know about a basic receive loop:

loop(State) ->
  receive
    {process, Data} ->
        {ok, NewState} = do_process(Data, State),
        loop(NewState);
    {send_state, From} ->
        From ! State,
        loop(State);
    halt ->
        exit(normal);
    Message ->
        ok = log(unexpected, Unexpected),
        loop(State)
  end.

-spec do_process(term(), #state{}) -> {ok, #state{}} | {error, term()}.
do_process(Data, State) :-
    % Do purely functional stuff
    Result.

-spec log(category(), term()) -> ok.
log(Cat, Data) ->
    % Do side-effecty stuff
    ok.

We can see exactly what cases result in another iteration and which don’t. Compare that with this:

loop(State) ->
  receive
    {process, Data}     -> do_process(Data, State);
    {send_state, Asker} -> tell(Asker, State);
    quit                -> exit(normal);
    Message             -> handle_unexpected(Message, State)
  end.

do_process(Data, State) ->
    % Do stuff.
    % Mutually recursive tail call; no return type.
    loop(NewState).

tell(Asker, State) ->
    % Do stuff; another tail call...
    loop(State).

handle_unexpected(Message, State) ->
    ok = log(unexpected, Message),
    % Do whatever else; end with tail call to loop/1...
    loop(NewState).

I like the way the code lines up visually in the last version of loop/1, sure, but I can’t know nearly as much about it as a process. Both styles are common, but the former lends itself to readability and testing while the latter is a real mixed bag. Pure functions would keep us conscious of what we are doing and commit our minds in ways to the definite-return form of code where our pure functions and our side-effecty ones are clearly separated. Of course, anyone could also continue to write Erlang any old way they used to — this would just be one more tool to assist with breaking complexity down and adding some compile-time checking in large systems.

I would love to see this sort of thing happen within Erlang eventually, but I am pretty certain that its the sort of change that won’t happen if I don’t roll up my sleeves and do it myself. We’ve got bigger fish to fry, in my opinion, (and I’ve certainly got higher priorities personally right now!) but perhaps someday…

JSON and YAML: Not a pair that fits every foot (but XML sucks)

It is good to reflect on exactly how hard a problem it is to define a consistent cross-platform data representation. Most of the time (especially on the web) we just shovel data around, let things be inconsistent, avoid conflicts by pretending they don’t happen, and carry a general disregard to data consistently. This attitude is, sadly, what has come to characterize “NoSQL” in my mind, though in a strict sense that is not true at all (GIS and graph databases aren’t SQL systems, and some are very solid — PostGIS being the exception in that it is a surprisingly well made extension to a surprisingly solid SQL-based RDBMS).

Obviously this isn’t a good attitude to have when dealing with things more important than small games or social media distractions. That said, most of the code written today seems to fall into those two categories, and many a career is spent exclusively roaming the range between these two (and whether we should consider most of the crap that constitutes the web a “game” itself is worth thinking about, whether we think of SEO, mindshare in the blogosphere, StackExchange rep, Facebook likes/friends/whatever, pingbacks, comment counts, etc.). We focus so much on these trivial and often meaningless cases that an entire generation of would-be programmers has no idea what the shape of data is really about.

When you really need a consistent data representation that can survive the network (ouch! that’s no mean feat!), can consistently be coerced into a known, predictable, serialized representation, and can be handled by generated code in nearly any language you need ASN.1.

But ASN.1 is hard to learn (or even find resources on outside of telecom projects), and JSON and YAML are easy to reference and (initially) use. XML was made unnecessarily hard, I think as a cosmic joke on people who never heard the term “S-expression”, but very basic XML seems easy, even if its something you would never want to type by hand (though that always seems to wind up being necessary, despite our best efforts at tooling…).

Why not just use JSON, YAML or XML everywhere? That bit above, about a consistent representation — that’s why. Well, that’s part of why. Another part of why is that despite your best efforts to define things in XML or nest explicit declarations in YAML/JSON you will always wind up either missing something, or find yourself needing to change some type information you embedded as a nested element in your data definition and then need to write a sed or awk script just to modify later (and if you’re the type who thinks “ah, a simple search/replace in my IDE…” and “IDE” to you doesn’t basically equate to “Emacs” or your shell itself, you’re going to a gunfight with boxing gloves on — if you need a better IDE to manage your language then you really need a better language).

The problem with YAML/JSON/XML are twofold: they are not defined anywhere, so while you may have a standard of sorts somewhere, there is no way to enforce that standard. An alternative is to include type information everywhere within your tags as attributes (in XML) or nest tagged groups or create a massive reference chain of type -> reference pointer -> data entry in YAML (or nest everything to an insane degree in JSON), but then making changes to the type of a field in a record type you have 20 million instances of is… problematic.

And we haven’t even discussed representation. “But its all text, right?” Oh… you silly person…

Everything is numbers. Two numbers, actually, 0 and 1. We know this, and we sort of forget that there are several ways of interpreting those numbers as bigger numbers, and those bigger numbers as textual components, and those textual components as actual text and that actual text (finally) as glyph you see when you use “the typewriter part” or look at “the TV part” (or do anything with the little touchscreens we use everywhere these days nobody seems to have worked out a genuinely solid interface solution to just yet).

Every layer of that chain of interpretation I mentioned above can be done several ways. Every layer. Think about that for a second. Now, if you live purely in a single world (like modern Linuxes and probably newer versions of OSX) where there is only UTF-8, then about half the possible permutations are eliminated. If you only ever deal with unaccented characters that fall in the primary 127 defined by ASCII, then several more permutations are eliminated — and you should dance with joy.

Unless you deal with a bit of non-textual data in addition to the textual stuff. You know, like pictures and sounds and application-produced opaque binary data and whatnot. If that’s the case, you should tremble. Or… oh god, no… what if your data doesn’t stand alone? What if all those letters are supposed to actually mean something? “We have lots of data” isn’t nearly as important to customers as “we have lots of meanings” — but don’t ask a customer about that directly, they have no idea what you mean, because all the text stuff already means something to them.