Author Archive

The Great Blockchain Race

Tuesday, December 12th, 2017

There is a big hustle going on right now over blockchain-based systems, most notably digital cryptocurrencies. It is as if the public just became aware of the word “blockchain”, saw that Bitcoin posted some crazy value gains, and decided “Oh? It went up? That means it is going to be a safe bet that it will go up forever!” and just hopped in with both feet.

Despite blockchain’s inherent scalability problems…

Despite the totally insane energy cost behind every single transaction going forward…

This has, of course, attracted the attention of The Sneakies. The Sneakies are people who realize that running a confidence game on a single person is moderately difficult, but running one on a large population that doesn’t really have the time or interest to dig into the details is quite easy — especially if you have a piece of cake in one hand, and even easier if they are panicked about something at the same time. Fear and hope are a powerful combination when aligned.

Since about 2014 an interesting proliferation of digital currencies (most being cryptocurrencies, but some even being created by banking consortia — har har!) has occurred. Some try to attract attention by spreading FUD about Bitcoin (not that the things they say about Bitcoin are inaccurate, but the same criticisms usually apply to the newly proposed currency as well), some try to attract attention using a “proof-of-work” system analogous to the original Bitcoin algorithm (“Get in now on the ground-floor!”), some try to leverage pre-existing FUD about Trump or the Euro or whatever. Most use a subtle combination and target a specific demographic (Antifa sympathizers, Randite Objectivist libertarians, Neo-Commies, Neo-Nazis, retirees and other “near-deads”, veterans, even Neo-Pagans).

Catching a trend? This is how trends that become confidence scams start to look.

Are cryptocurrencies the future of lightweight value exchange? Yeah, probably something like that. But we already have something more concrete backed by violence: actual currencies that can be electronically divided, transferred and calculated at a much lower cost to energy.

So what will happen? The early miners are punching out now — because while the run has been great and Bitcoin & co will be worth more than $0 even after the market correction, nobody knows when the correction will come. Full disclosure: I’m holding some Bitcoin. Mostly stuff I mined a few years ago. The value is sort of preposterous at the moment. Will I cash in? Maybe — but who knows what sort of pain that might cause me with tax services? It might not even be worth it unless I’m prepared to be shady about things.

But the scammers are starting to cash in, and it won’t be too much longer before one of two things happen:

  1. Scary but predictable: The Bitcoin “whales” cash in and the market collapses, causing a race to the bottom (like a short-call on everyone who has been betting against the Yen, Dollar, Pound and gold)
  2. Crap your pants scary and unpredictable: A quantum breakthrough or algorithmic development makes the entire blockchain transparent and manipulable — POOF!

I’m not saying these sort of efforts are a bad idea, just that they are unrefined and this is unexplored territory.

Also, as a parting thought… Every piece of software used for running crypto wallets, miners, etc. right now is rushed into production with little to no validation or security testing whatsoever. Maybe that isn’t the best way to safeguard something many non-techhies are hoping to be The Next Big Thing. Many of these platforms require Oracle’s Java, for example, and cannot even run on IBM’s JVM or the OpenJDK. Maybe that’s also not a good plan. That’s like having all your eggs in one big basket inside another basket of baskets. Whoops.

18 U.S. Code § 793 – Gathering, transmitting or losing defense information

Thursday, December 7th, 2017

Quite a few high-profile instances of leaks, breaches, infractions, cracks and “extreme carelessness in the handling of” classified information have been in the news over the last few years, and while folks like to talk a lot of fluff about whether this or that instance was truly vile or truly virtuous, I’ve never actually seen anyone reference the underlying rules regarding defense information.

So here it is: 18 U.S. Code § 793

Cornell Law has the text posted here as well.

(Personal) Guidelines for Software Projects

Monday, December 4th, 2017

A few guidelines for non-trivial, large projects you actually care about and want to maintain for more than a month or so.

1. Typespecs

Learn to use them. If you are writing a large, complex project in a language that doesn’t support this or have tooling for it then use a different language. Yes, it actually saves so much heartache that it is important enough to switch.

Why? Because for-real type checking can tell you, without the futility or religious interference of unit testing, whether or not your program is valid. A valid program is not necessarily a correct program, but an invalid program is necessarily an incorrect one. (Also, it is worth keeping in mind that classes are not types. There is a subtle, and critical, difference.)

2. Property testing, not unit testing

Don’t simply write a few “unit tests” and assume things work. They don’t. As Rich Hickey (the creator of Clojure) so aptly put it: “What is the one thing that is true about all bugs found in the wild? Every one of them passed all the tests!” It can be useful to engage in regression testing, but regression testing is a subset of integration testing and even crosses over with user testing (the ultimate of all) and project documentation and history management.

When you write code, it has bugs.

  • Some are syntactic: You forgot some ant poop somewhere (things like: : ; . ,), failed to close a brace or paren, or misspelled something.
  • Some are structural: You passed in a foo type but the function is defined as accepting bar (statistically this is the greatest category of compilable, invisible errors — reference point 1 above).
  • Some are scheduling and timing: You have races and deadlocks all over the place and never knew it because they don’t usually get triggered and are super complex to work out in your head.
  • Some are semantic: The program does precisely what you told it to do, but you told it to do the wrong thing (the most frequent place where protocol failures creep in).

You write every one of these kinds of bugs into your programs every time you write a non-trivial program. I can’t just tell you to knock it off and tighten your shot group because I do the same stuff because it is impossible to avoid! If you write all these stupid bugs into your programs, what do you think lurks in your hand-written test code? MORE BUGS!

So what do?

In the same way that we can write a type specification for a function (declare its domain and codomain, basically) we can also write a specification for the function’s valid inputs, and outputs and the expected rules the output should follow (its range and image, basically). This defines the properties of the function.

Neat-O. But what would we do with such a specification? Property declarations are like me explaining to you what a function does, but not how it manages to do it. To test whether our implementation of the function does the expected thing and lacks corner cases, however, we can use a property-based testing system to generate tests for us on the fly and run them to check whether the expected properties of the function hold true. Not only that, smart property based testing systems not only find bugs (values that are defined as valid but produce invalid results that violate the property specification) but can quite often home in on specific broken cases and give you a good indication what sorts of values are problematic. That is to say, a property-based testing engine equipped with good property definitions can locate the corner cases for you.

Why wouldn’t we do this by hand? Because typically unit tests cover a handful of most-common cases with their expected values and that’s about it. Property based testing is much less merciful and also much less prone to error because a property based tester will generate an endless stream of tests according to the provided properties and run them for as much CPU time as you’re willing to give for testing. You are never going to write millions of different test cases for your code. A property based testing engine will do precisely that if you give it the CPU time to do so. Compared to how testing is done in most projects this is like having nuclear power in the age of wooden stoves.

This is magical.

3. DO USER TESTING

When you release something that has worked for you so far, that’s about as much confidence as you should put in an alpha release. “Works for me!” are the bold last words of many an abandonned project.

Don’t be That Guy. Don’t release That Project as a final. Be clear its a beta or even alpha, and development is an ongoing thing, forever. Manage expectations, your users (paying or community) will reward you for being honest.

When you release a project understand that this is your beta period, even if you’re on a relatively mature version. In a sense all significant features go through their own little beta phase. This is true in part because you’ve no clue if power users are going to find a way to break it (they will) or if it will be instantly appreciated and adopted by the userbase (random gamble there). Whatever you think is important or intuitive might have never even occurred to them.

Power users are going to push the button the wrong way and don’t know how to deal. That’s actually a good thing if you maintain a relationship with your users, because you’re basically getting directions straight from the affected party about how to make your program better. This is important whether you’re doing community open source for some sweet Ego Points, or trying to feed the kids at your soul-crushing job.

No amount of unit testing (which we’ve already sort of debunked — write typespecs, don’t blindly churn out unit tests) or property testing (which is vastly superior to unit testing, but misses a lot of side-effecty issues, which are often the central purpose of your program) can catch everything. No amount of integration testing will uncover everything that is wrong with your program. None of these tests will tell you whether your program sucks to use and is reviled by users. But user testing will.

4. Don’t be afraid to change stuff

You have a version control system for your code. You use git. Or something. It doesn’t matter, though, because you have something that does version control for you and creating a new branch is painless. (Unless you’re not using a version control system… then you really need to start. You don’t have to submit to the dark cabal of Ruby hipsters that controls github, but you should at least be using git locally.)

If you have an idea try it out. It is probably a great idea in spirit but won’t be so great in reality until you’ve shaken a bit of the stupid, self-indulgent fantasy out of it. You can’t do that without exploring the idea in actual implementation and that sort of exploration requires hacking up your pristine project a bit until you discover exactly why, in mechanical terms, the Universe hates your idea. Once you know exactly why the Universe hates you and your ideas you can adjust your plan to accommodate the whims of the math gods, tame the vagaries of digital magika, and tap out the proper incantations in much less time than you could had you just held endless meetings about it.

Break stuff. Remember the Cardinal Rule of Hacking:

“If you understand what you’re doing, you’re not learning anything.”
– Some guy (who was not actually Abraham Lincoln)

Sometimes the best sign of progress is a change in the error messages you are getting.

Simplicity follows complexity. Until you write a godawful fugly version of your solution you don’t really understand the problem. If you don’t fully grok the problem how can you ever hope to come up with a solution? Only after you have encountered all the little gotchas that made the code ugly in the first place are you ready to rewrite that steaming pile of (working) poo into an elegant solution that is almost guaranteed to have fewer bugs if for no other reason than increased transparency and better organization of the code.

(But note that you could stick with the ugly version for a bit in a pinch — so not all is lost. Getting something working at all is better than having a bunch of great ideas that don’t exist in reality.)

5. Don’t be afraid of new languages

At this point in my life I’ve written code in about 30 or 40 languages. I don’t know the exact number. I have written a lot of code and gained intimacy with about 10 of those. That’s a lot of languages by some standards and not many at all by others. It is enough, though, that I have come to realize that most languages are minor syntactic variations on a couple of basic paradigms, and really none of that crap matters too much.

It’s all shitty. All languages suck. Some suck a little less than others. Try to find one from the handful that sucks dramatically less than others in a specific domain, then get comfortable with it as a go-to tool for that domain. But remember that it is just a tool. Jackhammers are tools, but I don’t see anyone building houses with them.

When you hop on to a new project that someone is already working on you’re going to have to pretty much adhere to the rules of their house, and that means dealing with whatever annoying language they wrote their awesome project in.

Want to hack on Freenode‘s core implementation? Better not mind dealing with network code and file operations in Java (eek!). And what if you don’t even know Java or it has been years since you saw it last and everything is different now? This is the concern that should worry you the least of all.

If you squint a little projects basically are languages. They have their own semantics (the project libs, its functions, it type specification, its class definitions, its decision tables, its… whatever its got that is relevant). They have their own sort of syntax. In fact, every very large project I’ve ever worked on tended to actually follow Greenspun’s Tenth Rule and if it was a concurrent system (so common today) they even tend to follow Virding’s First Rule. (That becomes less of a joke and more of a law of nature the longer you do this and the more you know about both lisp and OTP.)

What does this mean? It means that learning the language a program is written in is the easy part. Learning the libs of that language tend to take about twice as long as learning the language itself. Learning the internals of a large project, however, tend to take about ten times longer than that. So where is the real cost in effort here? It isn’t in the adoption of a new language. It is in the adoption of a new project because every project is a tarbaby.

6. JUST OPEN YOUR EDITOR YOU PROCRASTINATING SACK OF POO!

Getting started is the hardest part of writing anything, whether prose, code, or poetry is sitting down and typing out something.

How to tackle the procrastination problem? Easier said than done: OPEN YOUR EDITOR

3 to 5 letters is all you need: `vim` or `emacs` and away you go!

Once you’re fully in the Matrix, write a function or spec or something. It doesn’t matter what you try to do: it will be wrong. And then you’ll have been wrong, but not exhausted yet. And suddenly you’ll realize that you are the one being wrong on the internet today and that situation just cannot stand. So you’ll start fixing it. And tinkering on it. And before you know it you’ll actually have some something productive, the curse of social media will be temporarily suspended, and you’ll finally stop feeling so crap about yourself (for a few minutes, anyway).

Erlang: Naive Matrix Multiplication

Saturday, October 21st, 2017

Someone asked what was surely a homework question today on StackOverflow about matrix multiplication in Erlang. I set out to answer him in as simple a way as possible, but wound up writing a naive matrix generation and multiplication module.

The code to the module might be of interest to new Erlangers, as it adheres both to the style of zuuid and includes many examples of using a combination of list operations and explicit recursion to cut clutter and make the meaning of otherwise complex operations clear.

Here is the code:

%%% @doc
%%% A naive matrix generation, rotation and multiplication module.
%%% It doesn't concern itself with much checking, so input dimensions must be known
%%% prior to calling any of these functions lest you receive some weird results back,
%%% as most of these functions do not crash on input that go against the rules of
%%% matrix multiplication.
%%%
%%% All functions crash on obviously bad values.
%%% @end 

-module(naive_matrix).
-export([random/2, random/3, rotate/1, multiply/2]).

-type matrix() :: [[number()]].


-spec random(Size, MaxValue) -> Matrix
    when Size     :: pos_integer(),
         MaxValue :: pos_integer(),
         Matrix   :: matrix().
%% @doc
%% Generate a square matrix of dimensions {Size, Size} populated with random
%% integer values inclusive of 1..MaxValue.

random(Size, MaxValue) when Size > 0, MaxValue > 0 ->
    random(Size, Size, MaxValue).


-spec random(X, Y, MaxValue) -> Matrix
    when X        :: pos_integer(),
         Y        :: pos_integer(),
         MaxValue :: pos_integer(),
         Matrix   :: matrix().
%% @doc
%% Generate a matrix of dimensions {X, Y} populated with random integer values
%% inclusive 1..MaxValue.

random(X, Y, MaxValue) when X > 0, Y > 0, MaxValue > 0 ->
    Columns = lists:duplicate(X, []),
    Populate = fun(Col) -> row(Y, MaxValue, Col) end,
    lists:map(Populate, Columns).


-spec row(Size, MaxValue, Acc) -> NewAcc
    when Size     :: non_neg_integer(),
         MaxValue :: pos_integer(),
         Acc      :: [pos_integer()],
         NewAcc   :: [pos_integer()].
%% @private
%% Generate a single row of random integers.

row(0, _, Acc) ->
    Acc;
row(Size, MaxValue, Acc) ->
    row(Size - 1, MaxValue, [rand:uniform(MaxValue) | Acc]).


-spec rotate(matrix()) -> matrix().
%% @doc
%% Takes a matrix of {X, Y} size and rotates it left, returning a matrix of {Y, X} size.

rotate(Matrix) ->
    rotate(Matrix, [], [], []).


-spec rotate(Matrix, Rem, Current, Acc) -> Rotated
    when Matrix  :: matrix(),
         Rem     :: [[number()]],
         Current :: [number()],
         Acc     :: matrix(),
         Rotated :: matrix().
%% @private
%% Iterates doubly over a matrix, packing the diminished remainder into Rem and
%% packing the current row into Current. This is naive, in that it assumes an
%% even matrix of dimentions {X, Y}, and will return one of dimentions {Y, X}
%% based on the length of the first row, regardless whether the input was actually
%% even.

rotate([[] | _], [], [], Acc) ->
    Acc;
rotate([], Rem, Current, Acc) ->
    NewRem = lists:reverse(Rem),
    NewCurrent = lists:reverse(Current),
    rotate(NewRem, [], [], [NewCurrent | Acc]);
rotate([[V | Vs] | Rows], Rem, Current, Acc) ->
    rotate(Rows, [Vs | Rem], [V | Current], Acc).


-spec multiply(ValueA, ValueB) -> Product
    when ValueA  :: number() | matrix(),
         ValueB  :: number() | matrix(),
         Product :: number() | matrix().
%% @doc
%% Accept any legal combination of scalar and matrix values to be multiplied.
%% The correct operation will be chosen based on input values.

multiply(A, B) when is_number(A), is_number(B) ->
    A * B;
multiply(A, B) when is_number(A), is_list(B) ->
    multiply_scalar(A, B);
multiply(A, B) when is_list(A), is_list(B) ->
    multiply_matrix(A, B).


-spec multiply_scalar(A, B) -> Product
    when A       :: number(),
         B       :: matrix(),
         Product :: matrix().
%% @private
%% Simple scalar multiplication of a matrix.

multiply_scalar(A, B) ->
    multiply_scalar(A, B, []).


-spec multiply_scalar(A, B, Acc) -> Product
    when A       :: number(),
         B       :: matrix(),
         Acc     :: matrix(),
         Product :: matrix().
%% @private
%% Scalar multiplication is implemented here as an explicit recursion over
%% a list of lists, each element of which is subjected to a map operation.

multiply_scalar(A, [B | Bs], Acc) ->
    Row = lists:map(fun(N) -> A * N end, B),
    multiply_scalar(A, Bs, [Row | Acc]);
multiply_scalar(_, [], Acc) ->
    lists:reverse(Acc).


-spec multiply_matrix(A, B) -> Product
    when A       :: matrix(),
         B       :: matrix(),
         Product :: matrix().
%% @doc
%% Multiply two matrices together according to the matrix multiplication rules.
%% This function does not check that the inputs are actually proper (regular)
%% matrices, but does check that the input row/column lengths are compatible.

multiply_matrix(A = [R | _], B) when length(R) == length(B) ->
    multiply_matrix(A, rotate(B), []).


-spec multiply_matrix(A, B, Acc) -> Product
    when A       :: matrix(),
         B       :: matrix(),
         Acc     :: matrix(),
         Product :: matrix().
%% @private
%% Iterate a row multiplication operation of each row of A over matrix B until
%% A is exhausted.

multiply_matrix([A | As], B, Acc) ->
    Prod = multiply_row(A, B, []),
    multiply_matrix(As, B, [Prod | Acc]);
multiply_matrix([], _, Acc) ->
    lists:reverse(Acc).


-spec multiply_row(Row, B, Acc) -> Product
    when Row     :: [number()],
         B       :: matrix(),
         Acc     :: [number()],
         Product :: [number()].
%% @private
%% Multiply each row of matrix B by the input Row, returning the list of resulting sums.

multiply_row(Row, [B | Bs], Acc) ->
    ZipProd = lists:zipwith(fun(X, Y) -> X * Y end, Row, B),
    Sum = lists:sum(ZipProd),
    multiply_row(Row, Bs, [Sum | Acc]);
multiply_row(_, [], Acc) ->
    Acc.

Hopefully reading that on a blog won’t drive anyone too nuts. I’ll probably include an expanded version of that (or something related) in a convenience library eventually. Unless I forget. Meh.

Web Designers: Stop making SPAs for inherently web 1.0 style sites

Saturday, October 14th, 2017

It is 2017. What’s with the drive to make everything an SPA whether it needs to be or not? This is getting a little ridiculous. I’m going to ramble on below a bit because I’ve got a hankering to do so — pay this no mind.

All around the web I see sites that are best represented as a collection of inter-linked documents, and all around the web I see many of those being changed into single-page application (SPAs). Even more stupid is when the SPA in question was built by some naive dope who included a little bit of almost every JS framework in existence — including a random selection from the thousands of obsolete and dead ones.

What is the goal? What’s the deal? Do web authors today not know how the web was actually intended to work originally? That document publication is actually its reason for existence in the first place and that “web applications” are a new thing that is a backhack to an incomplete standard that only sorta-kinda-works?

Granted, the reason it only sorta-kinda-works is due mostly to the problems inherent in the fact that only a single language is allowed in scripts… which is ridiculous. Was nobody paying attention to the Guile2 approach all those years? The only lesson learned from the Java applet and Flash experience seems to have been that “it sucks to force users to install runtimes as plugins”. Ugh.

Anyway, back to web applications…

I get it. For the moment we don’t have a solid distinction between “a document browser” and “an application browser” so we are stuck with this insufficient worst-of-both-worlds nether region of “applications that masquerade as documents”. And that drives anyone nuts who has given this much thought.

Not that a lot of people have considered the difference deeply. I imagine that is probably because very few new coders today have ever written more than a line or two of code intended to run natively on a user’s local system. Nearly everyone has written thousands of lines of code intended to run natively on server-side systems, but even that is getting wonky because many youngsters today don’t know how to deploy without using Docker yet lack the faintest inkling as to what problems Docker actually is intended to solve and wind up bypassing better solutions when they exist.

Tools shine when they are used in a focused way, performing they job for which they were intended. The web is the same way. Yes, it is a big jumble of crap. So let’s just leave that there. Networks are a big jumble of crap, too, and so are human societies — so we’ve adopted dirty ways of dealing with the dirt. The jumbly pile of shit that is the web is one of our ways of dealing with that. Everything times out. Everything is sent in text. Protocols are bloated and redundant. There isn’t even a proper definition of what “valid” HTML and XML and JSON and whatever else is in most cases. Its all racing toward a singularity where everything is uniformly stupid. But… whatever, it sort of kind of still works — and humans just barely work themselves, so that’s par for the course.

The original web was designed to function as an insecure document publication system where documents could be interlinked. We realized that we could include more interesting stuff by expanding the definition of “document” to include more than just text, and quite recently with HTML5 the way in which documents can be written is only a few orders of magnitude behind, say, LaTeX, in its ability to arrange things on the screen (that’s feature lag is not entirely the fault of the HTML5 authors).

This gives a lot of freedom to website authors — perhaps too much.

If a website is a set of news articles or academic papers (or even tweets) then you really don’t need a SPA, you need a more traditional sort of “web site”. It can be dressed up all pretty with shiny things sprinkled around, of course, but we don’t want a SPA that mysteriously changes state in ways that users cannot bookmark things, can’t easily send links to one another to specific resources (something Twitter got right despite some initial confusion over how to frame their content), etc.

If a website is actually just a delivery front end for a graphical RPG, well, obviously the game part of the site is probably best designed as a SPA, but the rest of the site — the forums, armory, character pages, beastiary, fan wiki, manual, guild rankings, lore pages, etc. — are absolutely best presented outside of that SPA as an actual website.

See the difference?

The game example is actually quite useful to contemplate for a variety of reasons. I’ll probably come back and cut this post down to just that part. Either that or eventually come back and rewrite the first bits to more accurately convey the humor with which I, as a graybeard resident in cyberspace for about 30 years now, view the state of the web today.

Whatever you do, dear reader, have fun coding, and remember: Don’t outsmart yourself!

Las Vegas shooting prediction: Most casualties were not due to gunshot wounds

Wednesday, October 4th, 2017

Looking over the data for large stampedes and crowd crush events at concerts and sporting events, and comparing this to what I know personally from a career spent mostly handling various weapons in a tactical environment, I expect that we will discover fairly soon that the vast majority of casualties during the Las Vegas shooting — both injuries and fatalities — were actually due to stampede, and not anything to do with gunshot wounds at all.

Of course, in the confusion this issue has become politicized to an absolutely ridiculous degree by various anti-gun factions, and much of the US and European media is loathe to report anything other than anti-gun statistics for the moment, so we are seeing language tailored to evoke images of hundreds of people with actual gunshot wounds and zero people with stampede injuries.

For example: “Shooter in Las Vegas [blah blah blah] over 500 wounded.” This makes the reader or listener immediately envision 500 people actually wounded, as in due to violent trauma — and deliberate violent trauma at that. Which in this case would be exclusively due to gunshot wounds. But we have never seen a breakdown of causes of bodily harm by type, and this data will take a while to assemble.

By the time we do see these stats most people will not really be interested because immigration in Europe or stubborn people in Madrid/Barcelona or NFL SJW activity or whatever else will steal the spotlight and public attention before then. In other words, people will be distracted with another issue-of-the-day by then and forget that the new factoids they see relate to a previous event they felt very strongly about at the time it occurred.

Watch for this one.

Asian Governments Making Social Moves Together

Tuesday, October 3rd, 2017

I expect Asian governments to manifest a low-key but characteristically firm and absolute (and often official) position against Islam. Actually, I don’t expect it, I’m watching it happen and just now recognizing a fairly uniform trend. Something is going on in Asia with regard to this, and I don’t know quite what it is, but there is no doubt that doors are closing all across Asia for Muslims in general.

I think the timing is not a coincidence — the nature of Islamic threats are changing, becoming more diffuse, and taking on a different character just as a new generation of indoctrination is beginning across the West and Asia.

  • Myanmar has found something much more compelling than mere domestic political expediency to engage in its current operations (ISIS returners, as are turning up in Malaysia, Indonesia and the Philippines, is one possibility).
  • China has begun confiscating the Koran and categorized it as a book containing extremist political sentiment.
  • Thailand is readying a firm move against the southern Muslim rebels — and at the same time ISIS returners are very effectively influencing the young generation throughout the old Pattani region.
  • Saudis and other donors are standing up madrasas throughout Malaysia and Indonesia, and the Malaysian government is both unable to stop the trend while at the same time higher-ups in Putrajaya are strangely blind to the problem while also complaining about it.
  • The Philippines is obviously on a “you’re with us or against us” path politically and socially. And a certain of portion of the younger Muslim generation today is much more willing to take that as a challenge instead of an offer to pledge fealty (or at least negotiate terms).
  • Japanese are, at least anecdotally, becoming increasingly uneasy with the idea of accepting any Muslims, even as guest workers. The striking thing there is that ten years ago (well after 9/11) the topic of religion would never have been mentioned discussing this issue socially, but now it is brought up. This change over the last year or two coincides with the first mosque in Kyoto trying to promote itself via online ads and Japanese demonstrating an instant and strong aversion to the very concept of proselytization. They are now in “wait and see” mode socially — to watch and see how things turn out in Europe.
  • South Koreans seem to be on the same page as the Japanese — the attitude toward Islam having soured considerably over the last five years or so. Once again, this is anecdotal, but the subject has come up more than once, and many South Koreans keep up with news of attacks in France, Sweden and the UK.
  • Indonesia is seeing the rise of extra-judicial Islamic enforcement gangs.
  • Malaysia is seeing a similar rise in extra-judicial Islamic enforcement gangs, but the effect is somewhat muted by considerable repression by the special police and more active engagement with the group leaders.
  • Returners, returners, returners. ISIS veterans are flooding into various part of Asia, fresh off a tour in Syria, North Africa, Iraq or Afghanistan with ISIS and keeping in touch with one another. Of course, nobody feels comfortable with that. Unlike in Europe, though, well-known jihadis are not left to their own devices and most go missing somewhere in transit — but it is clear and evident that many are still returning and building new lines of communication and influence locally.

Any one of these issues, from official government actions to simple social reactions, would be grounds for certain groups to rally large responses — Islamic groups as well as Western-based political groups with strong anti-Asian nationalist agendas (something I’ve always found very odd). But the only thing making the news is Myanmar right now, and that’s a pretty hopeless fight to try to pick in terms of political pressure. Myanmar is about as pliable as North Korea as long as China is on their side, and China is indeed on their side with regard to this detail.

I do not see a future where Asian governments will feel compelled to do anything other than increase their resistance to an increased domestic Muslim presence. I fully expect that religious questions will be incorporated on visa applications to places like China eventually (not that repression of religion is anything new there).

I have no idea how any of this is going to turn out, but I find this trend notable and the timing troubling. I don’t know exactly what is triggering this much activity just now (why not a decade ago?), but something is clearly going on. It could be the outcome of some government assessments, or simply a change in the domestic social outlook, or both — but something is going on with this. And, of course, it is impossible to say “they are wrong”. It is just what they are doing and I’m just pointing it out.

Erlang: Silly way to see if your shell supports VT100 commands

Thursday, September 21st, 2017

There are a few cases where it can be useful to use VT100 terminal commands in shell interaction scripts to draw frames, progressbars, menu lines, position the cursor, clear the screen, colorize text, etc.

I actually have a small library of utilities like this I might eventually release, but its a pretty niche need.

Anyway, within that niche need, here is a really silly way to see if the terminal you run your shell in supports VT100 commands. (If you’re on Linux using pretty much any prepackaged terminal then your terminal supports VT100 commands, but that is not always so true on Windows, depending on how you are accessing your shell.) Paste the following into your shell:

Z =
  fun() ->
    {ok, S} = gen_tcp:connect("towel.blinkenlights.nl", 23, []),
    ok = gen_tcp:send(S, "\r\n"),
    Q =
      fun R() ->
        receive
          {tcp, S, B} ->
            ok = io:format("~ts", [B]),
            R();
          {tcp_closed, S} ->
            done
          after 60000 ->
            ok = gen_tcp:close(S),
            timeout
        end
      end,
    Q()
  end.

And then do Z().

(I remember seeing this first years ago and had forgotten it was even a thing! Sysop excuses is still live at port 666 as well, btw…)

Erlang: Converting text strings to Erlang terms

Monday, September 4th, 2017

We all love file:consult/1 and are familiar now with its inverse function. And of course everyone knows how comfortable it is to use the BIFs term_to_binary/1 and binary_to_term/1,2 to communicate over the network between nodes and even among other networked thingies written in other programming languages using BERT-RPC.

But we still have a gap.

There is not a very well known way to convert a text string that represents Erlang terms directly into a list of actual Erlang terms without writing to a file first and then calling file:consult/1. Most of the time you will never have this problem. But when you do encounter this problem it can be mighty annoying to figure out the steps to convert the string or binary to internal Erlang terms (to the point that I sometimes see people actually write to a temporary file just so they can then call file:consult/1 and then delete the file).

So, let’s take a look:

scan_binary(Bin) ->
    TermString = binary_to_list(Bin),
    scan_string(TermString).

scan_string(TermString) ->
    {_, Strings} = lists:foldl(fun break_terms/2, {"", []}, TermString),
    Tokens = [T || {ok, T, _} <- lists:map(fun erl_scan:string/1, Strings)],
    AbsForms = [A || {ok, A} <- lists:map(fun erl_parse:parse_exprs/1, Tokens)],
    [V || {value, V, _} <- lists:map(fun eval_terms/1, AbsForms)].

break_terms($., {String, Lines}) ->
    Line = lists:reverse([$. | String]),
    {"", [Line | Lines]};
break_terms(Char, {String, Lines}) ->
    {[Char | String], Lines}.

eval_terms(Abstract) ->
    erl_eval:exprs(Abstract, erl_eval:new_bindings()).

You’ll notice that I did not simply use string:lexemes(TermString, [$.]) (the successor to the now obsolete string:tokens/2) to break the original into discrete strings. That is because each string requires a period at the end or else erl_scan:string/1 will reject it. It is dramatically more efficient to run through the string a single time breaking at the periods and adding them back than traversing it once to break it into segments, then traversing every resulting string again just to add a period at the end (which also means an extra traversal of the list of that list to make the adjustments!).

Everything in that happens in scan_string/1 can, of course, crash if there is anything wrong in the input. If used as-is it should probably be run inside of a try..catch clause (and you should almost never, ever be using try..catch in Erlang to begin with, but this is one of the very few cases it is probably a good idea to). That could be accomplished by wrapping it in a non-insane function such as:

-spec maybe_scan(String) -> Outcome
    when String  :: string(),
         Outcome :: {ok, [term()]}
                  | {error, Reason :: term()}.

maybe_scan(String) ->
    try
        Terms = scan_string(String),
        {ok, Terms}
    catch
        error:Reason -> {error, Reason}
    end.

You’ll notice that I have a specific scan_binary/1 and a scan_string/1 also. I haven’t played around with this enough yet to feel comfortable throwing a full-blown io_list() at this, so my assumption is that you’re either reading data in from a file and will have a binary to start with, or would have a string that arrives or is constructed somewhere internally and know that you should flatten it yourself before calling scan_string/1 or maybe_scan/1.

How did I arrive at this?

The larger problem I have had to solve just now is unpacking and reading in configuration data from a large number of tar archives that I receive over the wire. While I could unpack them to disk, then read the file I want with file:consult/1, it is dramatically faster to unpack only the file I wanted from the archive in memory (as the archive itself has never been written to disk anyway), and that leaves me with a binary string of the file contents, but nothing on which I can call file:consult/1. Dhoh!

My solution to that problem was the above. This function has done its work now and I don’t need it anymore, but it strikes me as not such a crazy situation for other programmers to run into at some point so I’m leaving this here for my future self. I’ll probably include this function in a future version of a convenience library, and at that point I will either refactor it to break down all the possible error returns in a proper way (crash reports from within list operations inside list comprehensions can be mysterious), or decide that the details of an error from, say, erl_scan would be more confusing than its worth and instead provide a more generic return from some interface like maybe_scan/1.

Trump on the DPRK: Exerting Maximal Regime Change Influence

Friday, August 11th, 2017

Sitting within the target zone for a North Korean retaliation causes one to contemplate a bit on the state of things. Trump has doubled down on his bellicose rhetoric of “fire and fury” over the course of the last day, and quite a few people are flipping out, as anyone could have predicted. I have received several emails and calls from friends wishing me well if things go south, expressing hopes that various cabinet personalities can reel Trump in and so on.

All of this assumes Trump is nuts. That is far from an accurate portrayal of the situation.

Washington is faced with a very tough choice right now, but one that has only one real option available: Does Washington wait until American cities sit under nuclear threat from a country with a decision making apparatus that is only a single person deep (meaning, ultimately, the strike decision is left up to personal whim and intent), or does it sacrifice non-Americans to protect Americans?

Obviously, the choice is clear: risk Americans instead of risking Americans. To think that any other nation would do any differently is to believe we exist in a parallel universe where altruism reigns, feelings are reasonable goals of achievement and love conquers all. We do not live in that universe.

Let’s be clear: the US will not allow tens of millions of Americans to sit at risk of a North Korean leader who wishes to advance an extortion game against Washington. It will avert that by risking tens of thousands of foreign lives (mostly South Koreans, but also some Japanese and possibly Chinese as well). Even though I live within the zone that might get splatted, I really can’t see any other way for things to be — and let’s remember: this is tens of millions of American lives VS a few tens of thousands of foreigners from Washington’s perspective. Not much of a choice there, even if one is a hardcore humanitarian.

So now that we have established the American calculus, and we’re not deluding ourselves into thinking that management of a nuclear-armed, globally-strike capable North Korea is part of our menu of options, what is Trump going to do about this? How about get the Chinese or Russians to do something instead? Well, that route has already been explored and exhausted. The Chinese enjoy North Korea being a useful problem regionally, so do the South Koreans to some degree, the Russians love having the DPRK act as a consistent policy spoiler for everyone involved, and even the Japanese have leveraged the existence of North Korea from time to time. It was a useful problem for pretty much everyone for quite a long time, and that’s why it has been allowed to fester for so long.

But now things have gotten serious.

The US cannot wait longer than next spring to strike. The decision on exactly when to strike is dependent on weather, mostly. If the Americans believe that the advantage leans to their side in cold weather then we will see a strike sometime between late November and early March. If the advantage would go to the Americans in warmer months then we will see a strike sometime between now and December. Expect the US to ramp up a strike capability from now until whenever and just sit on it to mask the moment of their intent. Sure, nonessential being relocated from the American garrisons in South Korea would be a telltale sign, but I don’t know if Washington would even telegraph its intent that way rather than letting the chips fall where they may. This is serious business, after all. On the other hand, Washington may evacuate nonessential personnel right away and just remove that as an indicator all together very soon. Who knows.

Back to the rhetorical bit Trump threw out the other day and then doubled down on today…

Trump is doing everything but being explicit about his threat to either glass North Korea entirely or commit to a massive conventional strike that comes very close to that. Looking at Trump’s negotiating style since the 1980’s it is very likely that he intends to do exactly that if the situation does not improve — he is not known for bluffing. He also would not have made this decision alone. China has already stated that they would defend North Korea in the event of an American strike, so by elevating it to the level of an absolute conflict Trump is essentially guaranteeing that there would not be any chance for any action to escalate to becoming a regional war because there would not be a North Korea left to defend.

That sounds crazy, but it is not. It ensures a limited scope to the conflict from the start, and that is wise.

From the North Korean perspective, though, it does one more thing: it places every single leader and peasant and their families under threat of annihilation if Pyongyang does not change course in some way. The Chinese have been trying to effect a regime change in Pyongyang unsuccessfully for a few years now. Beijing can’t do it, it is very likely that nobody outside of North Korea can short of a war. Trump’s appeal to an absolute level of violence here is an overt signal to the North Koreans that it is up to them to effect regime change or face total annihilation. There is plenty of hidden opposition to Kim Jong Un in Pyongyang — but unless they feel that Trump is more dangerous to them than their own leader they are unlikely to feel motivated to move. After all, North Korea has had spats with the West hundreds of times over the last several decades — so often that there is almost a script for this sort of thing.

Trump is going off script. He is doing so to evoke a specific survival reaction in the upper leadership in Pyongyang, specifically a reaction against Kim Jong Un. This is probably the best chance anyone has of deposing him: turning his own leadership against him. They might die if they go against Kim Jong Un. They will certainly die if they go against Trump. This is how mutinies are made from the outside. On the outside chance that it comes to an American strike Trump has already guaranteed that a Chinese retaliation would be pointless. A massive strike (nuclear or conventional) would be a huge shock to the world, but the populations of the world are already experiencing hyperbolic rhetorical shock — when the volume has been turned up to 11 for so long there isn’t really anywhere left to go.

Trump is not crazy and his staff have certainly planned out (and are constantly revising) attack plans on North Korea designed to execute a strike devastating enough to limit the scope of any follow-on actions from anyone in the region. He has since moved on to working an influence play directly aimed at the North Korean leadership. This is how the game is played. People today are not used to being forced into situations where one bad option is balanced by an even worse one. Sometimes there is no unicorn to come save the day. The world is only going to turn more harsh in the coming decade — we probably will only remember this as a side show (if we even care to remember it at all).