The Intellectual Wilderness On Government: "There is nothing more useless than doing efficiently that which should not be done at all."

2016.02.2 23:51

zUUID: An Example Erlang/OTP Project

Filed under: Computing — Tags: , , , , , , — zxq9 @ 23:51

I was talking with a friend of mine yesterday about how UUID v2 seems to have evaporated. We looked into things further and found its not actually included in RFC 4122! One thing led to another and I wound up writing an example project that is yet another UUID generator/utility in Erlang — but this time it actually has duplicate v1 and v2 detection/correction and implements as close to what I can find is defined as UUID version 2 values.

As there are already plenty of UUID projects around I focused on making this one as readable as I possibly could — to include exported documentation, in-source documentation, obvious variable names, full typespecs, my silly little “pure” notation, blatantly obvious bitstring syntax, and the obligatory github presence.

Hopefully some folks newish to Erlang will come along and explain to me what confuses them about that code, the process of writing it, the documentation conventions, etc. so that I can become a better literate programmer. Of course, since the last thing the world needs is another UUID implementation I suppose I would have had better luck with something at least peripherally related to the web. (>.<)

2016.01.21 11:59

Erlang R18.0 Docs

Filed under: Computing — Tags: , , , — zxq9 @ 11:59

Language sites get knocked offline from time to time, so in addition to the Guile 2. manual I also keep a mirror of the Erlang R18 docs. I’ve never needed to link to them from here, but today has been a rough day around the internet for some reason so here we are. Depending on how long they stay down I might add the latest revision as well.

2016.01.20 16:59

Messaging: What Will It Do?

[Part 2 of a short series on messaging systems. (Part 1)]

Having implemented messaging systems of various sizes and scopes in all sorts of environments, I’ve come up with a few guidelines for myself:

  1. If messaging is not the core service, make it an orthogonal network service.
  2. If possible make the messages ephemeral.
  3. If the messages must persist use the lightest storage solution possible and store as little as possible.
  4. Accept that huge message traffic will mean partitioning, partitions will be eventually consistent, and this is OK.
  5. You don’t need full text search.
  6. If you really do need full text search then use a DB that is built for this — its a major time-sink to hack it in later and get it right.
  7. If the messages are threaded annotations over other existing relational data, swallow your pride and consult your DBA.
  8. VERSION. Your. DATA. And. PROTOCOLS.
  9. If anything about messages over existing data records feels hacky, awkward, or like it might put pressure on the existing DB, separate message storage and accept that some data integrity may be delayed or lost from time to time.
  10. Messaging is likely more important to your users than you (or they) think it is.
  11. The messages themselves are likely less important to your users than you (or they) think they are.
  12. If you can skip a feature, DO.
  13. YAGNI.
  14. You don’t need [AdvancedMessagingProtocol] (aka XMPP).
  15. If you *really* need XMPP it will be painfully obvious (and if you do chances are you certainly don’t need the extensions).
  16. [insert more things about avoiding feature creep]

Adding messaging to an existing system can get a little messy if you’re not really sure why you are doing it. If you know that your users really do have a need for messaging within your system, but you don’t know what features or type of messaging it should be, then think carefully about how you would want to use it as a user yourself.

Do users work together within your system to accomplish immediate goals? A concurrent multiuser system (multiplayer game, concurrent design tool, pair programming environment, etc.) benefits most from an ephemeral, instant chat system that centers around the immediate task. When the task is over the messages become meaningless and should be allowed to decay. It may be nice to give users an easy way to export or save significant messages, but that is really a client-side issue and is orthogonal to how the messaging system itself works.

Is the system used to coordinate real-world tasks or events? A real-world coordination system benefits most from a threaded comment/discussion system that centers around those tasks, and must only enhance but not replace the existing task-assignment and tracking features of the system. Threaded annotation is powerful here, but the threads need only persist as long as the task records do and messaging should never be mistaken for a task assignment tool (it can be leveraged as a part of task notification but never assignment or tracking). Remember ON DELETE CASCADE? This is where it is super helpful.

Do users self-organize into groups whose sole purpose is communication? Social group systems benefit most from mail systems implemented directly within the system they use to organize themselves. Such systems may also benefit from some form of ephemeral immediate chat, but it is critical that we keep in mind that that indicates a need for two different messaging systems, not One Message System To Rule Them All.

Many different flavors of message systems exist between the extremes of “persistent point-to-point mail” and “ephemeral, instant, channelized (group) chat”. Consider:

  • Persistent chat (Campfire, SocialObstructionist StackOverflow chat, etc.))
  • Message boards (“forums” — though this term is far broader in actual meaning…)
  • Usenet-style newsgroups
  • Mailing list systems
  • Email bridges
  • IRC + bridges + bots
  • Anything else you can imagine…

Some other things to think about are the nature of the users relationship to one another. That’s not just about communication channels and point-to-point delivery issues, it is also about what concept of identity exists within the system. Is the system one with strong authentication, total or partial anonymity, or a hybrid? This will dictate everything about your approach to permissions — from moderation, channel creation and administrative control to whether private messages are permissible and have a huge impact on what the implementation of a messaging system will require in terms of access to the original host system it is intended to support.

The issues of identity, authentication and public visibility are largely orthogonal to questions of persistence duration, storage and message-to-record relationship, but they can become intertwined issues without you realizing it whenever it comes time to design the storage schema, the serialization format(s), or the protocol(s). Of course these are three flavors of basically the same issue — though the modern trend is shy away from thinking about this either by hiding behind HTTP (like, uh, who even knows how to program sockets anymore? zomg!) and sticking your fingers in your ears when someone says that “schemaless JSON is the schema” or that XML can do the job of all three because it is the pinnacle of data representations. Consider whether this may change in the future. Keep YAGNI in mind, but when it comes to schemas, serialization and protocols it is always good to design something that can be extended without requiring core modification.

Messaging: Why Would You Do This?

[Part 1 of a short series on messaging systems. (Part 2)]

Messaging is a feature that seems to wind up on the ZOMG MUST HAVE!!1! feature list long after initial system deployment quite a bit these days. I’ve been getting asked about this a lot lately, so I’ve written down my generic advice here for reference. (If you want more specific advice feel free to contact me, though — its interesting and fun to hear about all the different things people are working on.)

I’ve been implementing and deploying messaging systems in one form or another since I first got involved with computers as a kid in the very late 80’s and early 90’s. Back then most multiplayer games were basically text message dispatch and routing systems with thematic computation surrounding those messages, so this was an area quite a few people my age dealt with — without realizing that it would one day be considered an “enterprise” feature. (And to think, we used to do this in Pascal, and assembler, and C, and hilariously inadequate variants of BASIC… as children! That nonsense was only reasonable to us because they were the only tools we had, those were labors of love, and we didn’t know any better.)

The most important thing to recognize about messaging systems is that the term “messaging” is hopelessly broad. It doesn’t really mean anything by itself. Every over-the-wire protocol is a message system definition, for example. Every drop-it-in-a-spool system is also a messaging system. Every combat notification system in a game is a messaging system. Every websocket thing you’ve ever done is a messaging system. This list could go on for quite a while, but I assume you get the idea.

Messaging systems take on different characteristics depending on their context of use. Some messaging systems are persistent, some are ephemeral, some have selective decay, some are instant, some are asynch, some are channeled, some are global, some are private, some are selective-access, some are moderated, some include extra semantic data that can be interpreted in the message body, some run everything through a central data service (“ooooh, ahhhh, ‘the cloud'”), some are peered, some are anonymous, some are verified, some broadcast, some are point to point, some are free, some are paid in per-message increments, some are paid by aggregate use, etc. The list keeps going as long as you can think of things that can be said about any system, actually.

It is important to keep in mind that most of the adjectives in the last paragraph are not mutually exclusive. Here is the fun part, though: if you need two that are, then you need two messaging systems. (That’s why they make chocolate and vanilla, after all.)

That’s a lot of different aspects to something as conceptually simple as “messaging”. The obvious problem there is that messaging is actually not simple at all in practice. When you decide that you need a “messaging solution” you really must carefully consider what that means for your users and your system. Your users’ state of mind, user experience, utility value of messaging and desire to use messaging in the context of their use of your system are all folded together.

Consider this: Email is a messaging system and we’ve all already got that, why is this not sufficient for your use? If you can’t answer that question quickly then sit back and let it stew for a bit — you will learn something about your users in the process of answering this question because it forces you to think for a moment as a user of your system instead of as a builder of it.

Let’s consider some reasons you might want to add a killer messaging feature to your existing product or system.

  • You can’t figure out the difference between “organic pageviews” and traffic resulting from the robot invasion.
  • You want more pageviews and you don’t care how you get them — messaging, you think, can be a source of bajillions of clicks!
  • Your users deal with data objects about which they constantly converse, but can’t do so within your system.
  • Your users have things to express that are best represented in a way that captures the history of their thoughts in sequence.
  • Your native applications do exactly what they should, but breaking out of it to send a message is distracting.
  • Your system already passes data among users, but requires human annotation to be fully meaningful.
  • Your system deals in machine-to-machine messages already, but users still have to call each other on the phone.
  • You are looking for a way to attach your brand to one more thing users see every day to reinforce the hypnotic effect you wish your brand had on them.
  • You require an anonymous(ish) way for users to communicate out-of-band relative to other online services.

A conditional flow would look something like this:

stupid_reasons = [pageviews,
                  domain_valuation,
                  brand_enhancement,
                  pagerank,
                  BULLSHIT_WEB_IDEAS(SEO && ALL_FLAGS),
                  ADS([ad_search, ad_rank, ad_tech, ad_{*}, ...]),
                  ...];

if (member(your_reason, stupid_reasons)) {
    if (idea_of(You) || you_are(TheManagement)) {
        admit(being_wrong);
        assert cancer_killing_internet = You;
        abort(stupid_idea);
        quick_grieve(dying_product);
    } else if (idea_of(TheManagement)) {
        abort(current_job, GRACEFULLY);
        // It won't last much longer anyway.
    }
    acquire(funding_contacts);
    contact(me, IRL);
    // I've got way better ideas.
    // You should work with me instead of sticking with losers.
} else if (knee_jerk(messaging_feature)) {
    abort(stupid_idea);
    focus(core_features);
} else {
    congratulate(You);
    panic(FEATURE_BURDEN);
    CAREFULLY(implement());
}

quick_grieve(hopeless_thing)
{
    shock(hopeless_thing, DEAD);
    anger(cant_coerce([model, users, investors, world]),
          TANTRUM_LEVEL(11));
    bargain([me, you, imaginary_entities, investors, users, world, HAL],
            perceived_value(hopeless_thing, PERSPECTIVE(MYOPIC)));
    mourn(hopeless_thing);
    accept(DEFEAT);

    return AGGRESSIVE_OUTLOOK([POSITIVE && EXPERIENCED]);
}

Did you find yourself at congratulations(), or somewhere else? Stop and think for a while if your reason belongs in the stupid_reasons list or not. How much will it actually enhance your users’ experience with the product or interaction with each other or you? Are you going to use it yourself, or are you going to continue to use email, IRC, whatever corporate chat service is popular this week, Twitter, etc? (And if you say “Well, I/we don’t use my/our product, so I don’t know…” then you are already rather far beyond admitting defeat, you just don’t know it yet.)

Be painfully honest with yourself here because messaging is one of those things that you can’t add in and then just take away later without consequences. Even if very few people use it and it turns out to be a technical burden to you someone will wind up using it no matter how shitty it turns out to actually be, a few of these people will come to depend on it, and to these people you will be an asshole if you ever remove it. Don’t be an asshole to customers. That’s how you wind up with a “[your_product]sucks.com” — and the kind of users who come to depend on weird internal messaging systems that suck are exactly the kind of people who will register a domain like that and talk shit about you. That crap will stay on the web forever and the stink may stay on your product forever, even if you change your product and they come back loving it.

So let’s assume that your system will exist in a context where it really does add value to the user’s life in some way. goto congratulations()! Messaging will be a big win if done properly and unobtrusively. As a reward for being diligent and thoughtful you now you get to wade through a swamp of design issues. But don’t worry, I’ll give you the fanboat tour and point the way through the muck.

Next we’ll look at some common features of messaging systems, what they mean for your implementation and most importantly what they mean for your users within the context of your system. [Continue to Part 2]

2016.01.15 17:47

間接的な影響

Filed under: Society — Tags: , , — zxq9 @ 17:47

I just realized that it is futile to drop hints to my wife about, say, a snack before dinner. The master play is to talk about snacks with my kids before dinner, and they will always find a way to deliver the goods.

anna_the_persuasive3

Evidence of real power: badass snacks.

2015.12.21 19:38

Pure Declarations in Erlang

Over the last year or so I’ve gone back and forth in my mind and in discussions with other Erlangers about type systems in Erlang, or rather, I’ve been going back and forth about its lack of one and the way Dialyzer acts as our bandaid in this area. Types are useful enough that we need Dialyzer, but the pursuit of functional puritanism gets insane enough that its simply not worth it in a language intended for real-world production use, especially in the messy, massively concurrent, let-it-crash, side-effecty, message-centric world of Erlang.

But… types and pure functions are still really useful and setting a goal of making as much of a program as possible into provable, bounded, typed, pure functions tends to result in easy to understand, test and maintain code. So there is obviously some stress here.

What I would like to do is add a semantic that the compiler (or Dialyzer, but would prefer this be a compiler check, tbh) be aware of what functions are pure and which are not. The way I would do this is by using a different “arrow”, in particular the Prolog-style declaration indicator: :-

[Edit after further discussion…] What I would like to do is add a directive that Dialyzer can interpret according to a simply purity rule. Adding this to Dialyzer makes more sense than putting it in the compiler — Dialyzer is already concerned with checking; the compiler is already concerned with compiling.

The directive would be -pure(Name/Arity) (a compliment to -spec). The rule would be very simple: only guard-permissible BIFs and other pure functions are legal from within the body of a pure function. This is basically just an extension of the current guard rule (actually, I wonder why this version isn’t already the guard rule… other than the fact that unless something like this is implemented the compiler itself wouldn’t have any way of checking for purity, so currently it must blindly accept a handful of BIFs known to be pure and nothing else).

For example, here is a pure function in Erlang, but neither the compiler nor Dialyzer can currently know this:

-spec increment(integer()) -> integer().
increment(A) ->
    A + 1.

Here is the same function declared to be pure:

-pure(increment/1).
-spec increment(integer()) -> integer().
increment(A) ->
    A + 1.

Pretty simple change.

“ZOMG! The whold standard library!” And yes, this is true — the whole thing is out. Except that the most important bits of it (the data structures like lists, dict, maps, etc.) could be easily converted to pure functions with little more than changing -> to :- adding a single line to the definition.

Any pure function could be strongly typed and Dialyzer could adhere to strong types instead of looser “success types” in these cases. Some code that is currently written to take an input from a side-effecty function, pass it through a chain of non-returning and possible side-effecty functions as a way to process or act on the value, and ultimately then call some side-effecty final output function would instead change to a form where the side-effects are limited to a single function that does both the input and output, and all the processing in-between would be done in pure functions.

This makes code inherently more testable. In the first case any test of the code is essentially an integration test — as to really know how things will work requires knowing at least one step into side effects (and very often we litter our code with side-effects without a second thought, something prayer-style monadisms assist greatly with). In the second case, though, the majority of the program is pure and independently testable, with no passthrough chain of values that have to be checked. I would argue that in many cases such passthrough is either totally unnecessary, or when it really is beneficial passing through in functions is not as useful as passing through in processes — that is to say, that when transformational passthrough is desired it is easier to reason about an Erlang program as a series of signal transformations over a message stream than a chain of arbitrarily side-effecty function calls that collectively make a recursive tail-call (and that’s a whole different ball of wax, totally orthogonal to the issue of functional purity).

Consider what we can know about a basic receive loop:

loop(State) ->
  receive
    {process, Data} ->
        {ok, NewState} = do_process(Data, State),
        loop(NewState);
    {send_state, From} ->
        From ! State,
        loop(State);
    halt ->
        exit(normal);
    Message ->
        ok = log(unexpected, Unexpected),
        loop(State)
  end.

-spec do_process(term(), #state{}) -> {ok, #state{}} | {error, term()}.
do_process(Data, State) :-
    % Do purely functional stuff
    Result.

-spec log(category(), term()) -> ok.
log(Cat, Data) ->
    % Do side-effecty stuff
    ok.

We can see exactly what cases result in another iteration and which don’t. Compare that with this:

loop(State) ->
  receive
    {process, Data}     -> do_process(Data, State);
    {send_state, Asker} -> tell(Asker, State);
    quit                -> exit(normal);
    Message             -> handle_unexpected(Message, State)
  end.

do_process(Data, State) ->
    % Do stuff.
    % Mutually recursive tail call; no return type.
    loop(NewState).

tell(Asker, State) ->
    % Do stuff; another tail call...
    loop(State).

handle_unexpected(Message, State) ->
    ok = log(unexpected, Message),
    % Do whatever else; end with tail call to loop/1...
    loop(NewState).

I like the way the code lines up visually in the last version of loop/1, sure, but I can’t know nearly as much about it as a process. Both styles are common, but the former lends itself to readability and testing while the latter is a real mixed bag. Pure functions would keep us conscious of what we are doing and commit our minds in ways to the definite-return form of code where our pure functions and our side-effecty ones are clearly separated. Of course, anyone could also continue to write Erlang any old way they used to — this would just be one more tool to assist with breaking complexity down and adding some compile-time checking in large systems.

I would love to see this sort of thing happen within Erlang eventually, but I am pretty certain that its the sort of change that won’t happen if I don’t roll up my sleeves and do it myself. We’ve got bigger fish to fry, in my opinion, (and I’ve certainly got higher priorities personally right now!) but perhaps someday…

2015.12.8 23:31

Rich Hickey: Simple Made Easy

Filed under: Computing,Overlooked Resources — zxq9 @ 23:31

This is an excellent presentation of the “soft tech talk” type given by Rich Hickey about the difference between a thing being “easy” in the sense that it is near to our experience and it actually being “simple” in the sense that understanding the idea does not force us to understand a lot of other ideas that are partially folded into it. His point is that we should be careful about the way we use the words “easy” and “simple” and be rather more specific than we tend to be, because this hides up a rather large and common class of complexity problems that we tend to gloss over within programmer society because we are merely familiar with a huge class of unnecessarily complicated constructs that have nothing to do with the business problems we are supposed to be solving.

Rich Hickey: Simple Made Easy

If only it were glaringly obvious to us when we were engaging in the glossing-over of non-essential complexity in architecture. I suppose one way of at least trying to become conscious of this is the frequency of use of a certain alarm-bell phrase: “just”. Ward Cunningham’s wiki actually has a whole page on it, along with another page that enumerates and discusses/argues a few other linguistic red flags of note.

2015.12.7 22:25

Bad Art: Arbitrarily Ordinary Sounding Names

Filed under: Bad Art — Tags: , — zxq9 @ 22:25

arbitrary_namesDriving home today I saw a sign that made me realize how overused yet utterly meaningless some common company names are.

2015.12.6 17:43

Its a Small, Small World

Filed under: Society — Tags: , , — zxq9 @ 17:43

A question on the Worldbuilding SE site about military contractors / mercenaries / “evil henchmen” / whatever caught my attention and I responded. A discussion started between myself and another guy who has contracted before in the same places, but on the tech side of the business instead of PSD. I’m preserving the conversation here because it illustrates the point I was trying to make in my answer, but doesn’t really fit within the answer and will certainly be deleted by the mods:


@zxq9 I’m impressed by your answer. I worked as a contractor as well, and ran a bar at night in one country (Middle East). Lots of the guys I worked with were former Ranger / SF. I did tech instead of PSD, but some jobs we were responsible for our own security, travel, etc. All the guys I saw ended up going back to the States (or the Philippines…) after contracts; nobody really hung out in theater after jobs. Are those little communities you’ve referred to really that common? Because I’ve yet to see people hang around after the fact.
hathead 15 mins ago

@hathead If you were ever in Baghdad you might remember that along the main E-W road between the traffic circle and Triple Canopy’s base Olympia was a mish-mashed neighborhood with a few shops and restaurants, and a couple of smaller contract company offices working out of houses. (Near the “Hot Tomato” restaurant that was always weird.) That neighborhood is exactly the sort of spot I’m talking about. Kabul has something similar. I think most of that may have been invisible to you if you didn’t work PSD or base def (and therefore have your own wheels) — but its there. Similar in Kampala.
zxq9 10 mins ago

@zxq9 – I remember the neighborhood but didn’t spend a lot of time out there. People bought alcohol there. I worked on Bes-maya, I guess you know where that is. My glory days were in Kabul, though. There we all lived on the economy and would pop into the embassy / base to do work, or speed out to Bagram. Kabul was always weird- you can get whatever you needed and half the guys at the bar were wanna-be journalists. Tourists even popped in sometimes. This neighborhood you’re talking about in Baghdad – it seemed like it was within the confines of base (checkpoint controlled). Was it not?
hathead 7 mins ago

@hathead I also forgot to mention… There are contracts in the southern Philippines, Indo, Malay, Cambodia and Thailand , too — and there are two SF association houses (the front half of which are really bars, like Garfield’s in AC and Tilac II in Pattaya). The neighborhoods near there are full of older SF guys who married locally and never left, and sometimes its easy to source logistics or meet the right folks to get a crew together locally. Pace of life is a bit slower there, but the shape of things is fairly similar — but not everyone wants to have anything to do with contracting.
zxq9 6 mins ago

@hathead The neighborhood wasn’t, but right next to it was the former finance minister’s house and apartments for his wives — which was turned into a business development center. The edge of it was right on the NW edge of the circle and had walls + a guard force from TC (which I was in charge of for 6 months once). That place gave the impression the neighborhood was controlled, but it was just that one facility. The volume of alcohol trade there always surprised me! A lot better than the “parts cleaner” thing the Poles had going on down in al Kut.
zxq9 3 mins ago   edit

@zxq9 – half the ex-Army guys I worked with had houses in the Philippines. You probably know some of the same guys. It’s not easy getting work as a tech with a PMC but I managed to find it with a big name (closed now, but you’d know them) and it was a lot of fun. Hanging out in Turkey now. It’s not hot enough yet that a worthwhile expat community has established. Yet.
hathead 2 mins ago

@zxq9 – The Europeans had fine alcohol, it was just a matter of finding it. They had a bar on Liberty that was wild. Panties stapled to the ceiling. I only saw it once; kept out of that sort of thing mostly. Did brew my own beer, though.
hathead 1 min ago

@hathead Likely do know some of the same folks. They’ll put this thread in chat or delete it pretty soon. My url is the same as my alias here: zxq9.com . Drop me a line. Turkey is an interesting place right now, too!
zxq9 46 secs ago

@zxq9 – I’ll drop you a line, and delete my previous chats if I’m still able.

Bad Art: Karl VS Carl

Filed under: Bad Art — Tags: , , , — zxq9 @ 13:10

If you've never seen the Walking and Talking Dead bad lip reading videos you're missing out...I’m so behind on Western media that the first time someone asked if I knew about the Carl Poppa video, I totally had the wrong idea.

Older Posts »

Powered by WordPress