Erlang: R20.3 doc mirror

April 15th, 2018

The main Erlang website has been super snappy for the last several months so I had slacked off on mirroring the documentation. Today it seems there are a few problems (DDOS-type symptoms, but I have no idea what is going on) so I’ve gone ahead and mirrored the R20 docs.

I also updated the “Erlang Stuff” page — though that page is going to get a few more changes once the Erlang tooling suite I’m working on is out, as now tinder, firekit, flint and zx are now all incorporated into the new (much better) thing… but more on that thing later once I’m done.

The Steins

April 10th, 2018

And then there’s the family of Stein.
There’s Ger, there’s Epp, and there’s Ein.
Ger’s stories are bunk.
Epp’s programs are junk.
And no one can understand Ein.

No Basis

February 7th, 2018

I had a conversation with a Marxist today. It became a bit heated on her side because she simply could not believe that I was unmoved by her arguments and was extremely frustrated with my explanation of the history of property, hierarchy and territorialism. She was particularly enraged about hierarchy and territorialism because they clearly originate in the animal kingdom and that presents a very difficult argument against her alleged desire to “live in the natural state of humanity” as our brains are hardwired for these things. That being the case there is no simple solution to the eternal question “Quis custodiet ipsos custodes?” and that really throws a wrench in the gears of the leadership tier of her hoped-for utopia.

Trying to turn the topic a bit to let her explore things on her own, I probed her a little more about labor-value theory and immediately hit a wall. I was trying to figure out why the conversation was so utterly unsatisfying until it dawned on me: she had never actually read Marx. She had no useful knowledge about philosophy at all, never having read any ancient, classical or even modern philosophers, scientists, psychologists, humanist thinkers, historians or economists.

She lacked any basis for recognizing a novel thought, much less being able to connect the thought at the leading edge of discourse back to its roots in a prior school of thought or even its relation to any other well-known concept. In fact, she didn’t know there were fundamental questions about whether knowledge is even possible, much less that this question divides philosophical tradition almost perfectly in two.

Through the encounter two things stuck out.

First, that she was so willing to promote violence as a reasonable tool for implementing Marx’s vision despite her not knowing anything serious about that vision and an accurate model of Marx’s vision actually lacking any economic theory whatsoever. That last bit didn’t phase her even a little. (And yes, you read that right, despite all the talk of “production” there is no theory to which a Marxist leader can refer to determine who should produce what when and how much is needed when. This should be terrifying to Marxists, especially in the absence of market feedback via price fluctuations, but they really seem not to care.)

Second, that despite her extreme, almost violent opposition to my positions, she wanted me to take her out to dinner next week. This was not her being reasonable, though, she just grew to hang on my words for some reason (though really, I didn’t say much in total word count). She started to like me, despite actually telling me at one point that I am “like the Nazis”. I usually get along well with women and like to think I nearly understand them, or at least some of them, but this was baffling. Then again, I haven’t dealt with many young Western women lately and clearly times are changing (she is in her mid-20’s, from France, and claims feminism is her religion; I am in my late-30’s, white living in Japan, and am unabashedly male and good at being in charge — this would be a disaster on every level).

I’m no philosopher — I’ve never even attended university — but it blows my mind that someone would commit so utterly and completely to a violent political philosophy without knowing what that philosophy was. This can only be described as a form of moral blindness (and not the good kind).

I can only imagine that most Marxists, whether or not they know much about what Marx actually proposed, have a similarly limited set of initial information against which to compare Marxist thought.

Confounding Beginner Question: What is an Erlang Atom and Why is it Useful?

February 1st, 2018

Like other Erlangers, I tend to take the atom data type for granted. Coming from another language, however, you might be puzzled at why we have all these little strings that aren’t really strings.

The common definition you’ll hear most frequently is something like:

An atom is a label. Its only meaning is itself.

Well, that’s true, but that also sounds a bit useless to someone coming from Python or R or JavaScript or whatever. So let’s break that down: what is a “label” useful for in programs?

  • Variable names are labels.
  • Function names are labels.
  • Module names are labels.
  • The strings you use as keys in a key/value data structure are labels.
  • The enums and label macros you might use in C for semantically significant internal values are almost exactly like atoms

OK, so we use labels all the time, why don’t any of those other languages have atoms, though? Let’s examine those last two reasons for a moment for a hint why.

In Python strings are objects and while building them is expensive, hashing them can be done ahead of time as a cached operation. This means comparing two strings of arbitrary length for equality is extremely cheap, because it is reduced to a large integer comparison for equality. This is not true in, say, C or Erlang or Lisp unless you build your own data structure to carry around the pre-hashed data. In Python it is simple enough to say:

if 'foo' in some_dict:
  # stuff
else:
  # other stuff

In C, however, string comparison is a bit of a hassle and dealing with string data in a cross-platform environment at all can be super annoying depending the age of the systems you might be interacting with or running/building your code on. In Erlang the syntax of string comparison is super simple, but the overhead is not pre-paid like in Python. So what is the alternative?

We use integer values to represent keys that are semantically meaningful to the program at the time it is written. But integers are hard to remember, so instead of having magic numbers floating all around the place we typically have semantically significant integer values aliased from a text label as a macro. This is helpful so that I don’t have to remember the meaning of code like

if (condition == 42) launch_missiles();
if (condition == 86) eat_kittens();

Instead I can write code like:

#define UNDER_ATTACK    42
#define VILE_UNDERBEAST 86

if (condition == UNDER_ATTACK)    launch_missiles();
if (condition == VILE_UNDERBEAST) eat_kittens();

It is extremely common in programs to have variables or arguments like condition in the above example. It doesn’t matter whether your language has matching (like Erlang, Rust, logic languages, etc.) or uses explicit conditionals like the fake C example above — there will always be a huge number of micro datatypes that carry great semantic significance within your program and only within your program and it is as useful to be able to label these enumerated values in a way that the human coders can understand and remember as it is useful for the computer to be able to compare them as simple integers instead of going to the trouble of string comparison every time your code needs to make a decision (because string comparison entails an arbitrarily long sequence of integer comparisons every single time you compare two strings).

In C we use those macros like above (well, not always; C actually does have super convenient enums that work a lot like atoms, but didn’t when I started using it as a kid in the stone age). In Erlang we just use an atom right there in place. You don’t need a declaration or definition anywhere, the runtime just keeps track of these things for you.

Underneath the hood Erlang maintains a running table of atom label values and translates them to integer values on the way into the system and on the way out of the system. The integer each atom actual resolves to is totally unimportant to you, so Erlang abstracts that detail away, but leaves the machine comparing integer values instead of doing full-string comparisons all over the place.

“But Erlang maps don’t do string comparisons on keys!” you might say.

And indeed, you would be right. Because map keys might be any arbitrary value each key is hashed on the way in, and every time keys are compared the comparing term is hashed the same way, so the end comparison is super fast, but we have to hash the input value first for it to mean anything. With atoms, though, we have a shortcut, because we already know they are both unambiguous integer values throughout the system, and this is a slight win over having to hash first before comparing keys.

In other situations where the comparison values cannot be hashed ahead of time, like function-head matching, however, atoms are a huge win over string comparisons:

-module(atoms).
-export([foo/1, bar/1]).

foo("Some string value that I don't really recall") ->
    {ok, 1};
foo("Some string value that I don't really care about") ->
    {ok, 2};
foo("Where is my cheeseburger?") ->
    {ok, 3};
foo(_) ->
    {error, wonky_input}.

bar(dont_recall) ->
    {ok, 1};
bar(dont_care) ->
    {ok, 2};
bar(cheeseburger) ->
    {ok, 3};
bar(_) ->
    {error, wonky_input}.

I’ve slowed the clockspeed of the system so that we can notice any difference here in microseconds.

1> timer:tc(fun() -> atoms:foo("Some string value that I don't really care about.") end).
{16,{error,wonky_input}}
2> timer:tc(fun() -> atoms:foo("Where is my cheeseburger?") end).
{13,{ok,3}}
3> timer:tc(fun() -> atoms:foo("arglebargle") end).
{12,{error,wonky_input}}
4> timer:tc(fun() -> atoms:bar(dont_care) end).
{9,{ok,2}}
5> timer:tc(fun() -> atoms:bar(cheeseburger) end).                                      
{10,{ok,3}}
6> timer:tc(fun() -> atoms:bar(arglebargle) end).                                        
{10,{error,wonky_input}}

See what happened? The long string that varies only at the tail end from two options in the function head takes 16 microsecond to compare and return a value. The string that differs at the head is evaluated as a bad match for the first two options the moment the very first character is compared. The total mismatch is our fastest return because that string never need be traversed even a single time to know that it doesn’t match any of the available definitions of foo/1. With atoms, however, we see a pretty constant speed of comparison. That speed would not change at all even if the atoms were a hundred characters long in text, because underneath they are all just integer values.

Now take a look back and consider the return values defined for foo/1 and bar/1. They don’t return naked values, they return pairs where the first member is an atom. This is a pretty common technique in Erlang when writing either a library intended for 3rd party use or when defining functions that have side-effecty operations that might fail (here we have pure functions, but I’m just using this as an example). Remember, the equal sign in Erlang is both an assignment operator and an assertion operator, when calling a function that nests its return values you have the freedom to decide whether to crash the current process on an unexpected value or to handle the “error” (in which case for your program it becomes an expected condition and not an exception).

blah(Condition) ->
    {ok, Value} = foo(Condition),
    do_stuff(Value).

The code above will crash if the tuple {error, wonky_input} is returned, because the expected atom 'ok' does not match the actually returned atom ‘error’.

bleh(Condition) ->
   case foo(Condition) of
       {ok, Value}          -> do_stuff(Value);
       {error, wonky_input} -> get_new_condition()
   end.

The code above now does not crash on that error return value and instead moves on to get another condition to try out, because the error tuple matches one of the case conditions that is defined as a return value. All this can happen really fast because atoms comparisons are really integer comparisons, and that means we save a ton of processor time (and space) by avoiding string/list or binary comparisons all over the place.

In addition to atoms being a much nicer and dramatically more flexible version of global enumerated types that let us write code in a more natural style that uses normal-language labels for program semantics, it turns out that function and module names are also atoms. This is a really nice feature in itself, because it allows us to write highly dynamic code with a lot less confusion about what types both sides of a call needs to be as well as making the code easier to read. I can even implement my own version of apply/3:

my_apply(Module, Function, Args) ->
    Module:Function(Args).

Of course, there is a whole pile of reasons why you will never want to actually write a function like this in a real program, but that’s the sort of power we have without doing any type casting magic, introspection, or on-the-fly modification of our program, references or memory space.

Once you get used to using atoms and matching you’ll really start to miss them in other languages and wonder how you ever got along without them. Now run off and start writing some code to practice thinking with atoms. They will become natural to you before the day is out.

American Geopolitical Imperatives: A (rough) Primer

January 31st, 2018

[2018-01-31: The following is an incomplete rough draft of an article I’m writing to have a single place to reference whenever anyone asks me sweeping questions about what I mean by “geopolitics” and “geopolitical imperatives”. This note will be removed once it is more complete. Thanks to Joe Armstrong for convincing me that it is better to publish incomplete ideas sooner than to publishing complete ideas never.]

The word “geopolitics” has been getting thrown around a lot in the media the last two or three years but unfortunately I’ve yet to see an accurate use of the term in general media. A handful of actual geopolitical analysts occasionally get a five minute spot somewhere, but even that is rare. (Indeed, I find the terms “analysis” and “assessment” almost universally misused as well.)

Despite the term being heavily abused in the media, geopolitics is a distinct discipline with clear definition. Geopolitics is a lens for interpreting world events, a tool for separating at least a little bit of personal bias and emotion from our interpretation of the world by modeling it as a physical system where aggregate human action is constrained by geographical factors. The geography a group of humans inhabits dictates a few imperatives on the group that must be satisfied if the group is to be successful relative to other groups. This version of geopolitics is “pure” and often leaves out the social aspect of human interaction. Of course, culture can play such a huge role that I occasionally step outside of pure geopolitics and extend to socio-geopolitics, but generally speaking the “geo” part of the term is the dominant factor.

Interpreting the world from a geopolitical perspective can be emotionally painful or shocking at first. That isn’t hyperbole. It can actually be so disturbing that many people who are initially interested in “how the world really works” run screaming from the first few lessons with an experienced mentor. Your political biases will get eaten wholesale, much of what you believe to be true about the world will be overturned, your sense of personal morality will find itself under constant attack (as individual morality can never be effectively applied to large groups), many of your unconsidered assumptions may turn out to be totally false on closer examination, etc. It can be deeply disturbing, even terrifying, for people whose views are heavily shaped by ideology.

Have strong feelings about Marxism? Capitalism? Abortion? Religion? Some particularly divine or truly vile historical figure? A favorite period of history? A favorite culture? A favored artifact of technology or warfare or art? A favorite language? A favorite style of cuisine? Lifestyle? Sexual preference? Religiously informed political view?

Geopolitics will eat your babies and murder your friends before it finally turns on you and leaves you feeling that everything your moral compass was based on is a fabrication.

That is, of course, until you get reoriented and realize that none of what was destroyed mattered — because it was all mistaken models of how the world works based on limited information, the missing parts of which you filled in by projecting your personal sense of individual morality on governments, national groups, companies, tribes, periods of history where the moral compass may have pointed a totally different direction, and so on. You were imagining the world as it should be instead of how it actually is, and snapping out of that is the shocking part. You don’t have to be changed by geopolitics, nor do your views about individual right and wrong. Your views about governmental right and wrong will definitely change, though, and perhaps most importantly, you will come to realize that the publicly touted reasons for many events are exactly backwards. All of this can be at odds with personal views to which you may discover you have developed a profound emotional attachment, and letting go of emotional attachments is uncomfortable for a reason.

Cause and effect are often reversed in public discourse. Sometimes it is a deliberate trick of rhetoric, but for the most part it is just how people recall and justify events after the fact, especially when they don’t have time to really dig into the details.

Today we are going to take a closer look at that phenomenon I mentioned above: geographical imperatives and how the laws of physics and the unwritten rules of human behavior conspire to force them on groups of humans. Most nations have impossibly difficult sets of imperatives, or imperatives that are difficult to establish clear priority order for or sometimes even discover without an in-depth investigation. Sometimes a group of humans itself doesn’t even a coherent body, which obviously confuses things a bit.

I’ll hit the low hanging fruit as an exploratory exercise today and examine the most well-known case in the world: the geopolitical imperatives of the United States of America.

Maps are Important

This thing below is called a map. They are really, really important if you want to understand why governments do pretty much anything on the strategic level.


Low-res image compliments of Wikimedia Commons user Martin23230.

There are many kinds of maps. This is a simple orthographic projection of one side of the Earth that shows some of the major political boundaries and none of the terrain other than separating the always-dry bits from the always-wet bits. That’s not very useful for many kinds of analysis, but this kind of simplification is really useful for understanding relative distances at a glance.

Our emotional attachment to particular issues tends to cause some distortion in our recollection of the map and particularly our memory of adjacent nations, the exact location of major waterways, the distance between significant points, and especially the details of the terrain involved that may have a profound effect on how a strategist will interpret the map based on how difficult it is to engage in a given activity in a given area.

As an example, I can’t remember how many times people have asked me about bizarre 9/11 or Afghan War conspiracy theories, the underlying premise of the theory being that they feel certain the U.S. military could absolutely flood Afghanistan with soldiers and war matériel if the need arose by using the USAF’s tremendous fleet of cargo craft.

It is hard to illustrate the sheer scale of how wrong this is.

The USAF does not have nearly enough airplanes to lift the daily tonnage of logistical support supplies necessary to run the Afghan campaign. It is truly mind-boggling how much stuff goes into a campaign of that scale and how much logistical support the operations involved require. Is the campaign dramatically bloated for the actual impact it has? Yes, definitely, but it isn’t 10x bloated. Maybe double or triple, but not bloated by a factor of 10. To support the Afghan campaign by air would require that it be reduced by about 100 times at the very least. I forget the figures right now, but we actually ran a calculation like this a few years ago and it came out to something between 100 and 1000 times shrinkage would be required to support the campaign by air alone.

That’s a lot of shrinkage. It is clear that the (ridiculously long) list of goals of the campaign would have to be dramatically reduced to make such a shrinkage possible. (See? This discussion is already about trade-offs — a trend that will continue.)

Why so many trade-offs in the Afghan campaign? Why is this so hard?

Geography.

Afghanistan is a very high altitude, landlocked buffer zone surrounded by buffer zones. The roadways are limited, there is extremely limited internal rail support, and no ocean access at all (so no heavy shipping). The maximum lifting capacity of a C-17 is about 70 metric tons and a little less than 70 or so are available to the government at any given time (of about 200 total in the military system).

That still sounds like a lot of lifting capacity, and it is! Relatively speaking. Compare that to 10 to 20 tons of capacity in a typical shipping container multiplied by a typical capacity ranging from 5,000 to 15,000 containers per ship when you have ocean access.

But back to airplanes… even if you reduced the mission by a factor of 100 and even if you could cancel every other military operation in the world to make every plane available to support the Afghan theater — where are you going to put all those planes? There are only two airbases in Afghanistan that can handle an even partially loaded C-17, and tarmac space is extremely limited, as are repair facilities and fuel supplies. Just consider how absolutely impractical it would be for C-17’s to fly fuel, spare parts and alternate crews in for all the other C-17’s, especially given crew rest, parking space and emergency repair availability. Even supporting C-17 operations at that scale with just C-17s is problematic. (Don’t even get me started on environmental factors like the weather…)

Nope. Trying to support an entire heavy-presence military operation with air supply alone is a non-starter.

Heavy things are really hard to move, and the higher the altitude the harder that gets. That’s also why helicopters are only partially useful in Afghanistan compared to how much utility we got out of them in Iraq. (And in Iraq we had plenty of aircraft issues, not the least of which were sand eating the engines or totally obscuring the ground — issues that, once again, are dictated by geography and cannot be changed no matter how much you hope and pray and write policy papers nobody will read.)

Keep that in the back of your mind. Logistics drives just about everything, and logistics is a super hard problem humanity has not come anywhere close to solving yet (sort of like computer security…).

American Geopolitical Imperatives

The American geopolitical imperatives are, in priority order:

  1. Delegate as much political control downward as possible
  2. Make the East Coast unprofitable for European navies to attack
  3. Achieve continental integrity from East to West coast
  4. Defend the Mississippi / New Orleans outlet and the maritime passage past Cuba
  5. Enforce hegemony across the Americas
  6. Establish global naval supremacy
  7. Establish supremacy in space
  8. Maintain a balance among other powers to prevent the rise of a rival superpower

That’s it in all its cold, calculated magnificence. As lists of national imperatives go this is one of the shortest and most inherently achievable lists in the short history of geopolitical analysis. Quite remarkable, actually. In light of recent political insanity and media hyperbole please, let’s take a moment to consider what is not not that list.

No mention of race. No mention of immigration. No mention of education reforms or welfare programs or any other form of social assistance programs. No mention of slavery. No mention of “social safety nets”. No age of consent. No duty to uphold the sanctity of homosexual marriage unions within the non-CIS-but-genderqueer branch of the pony community. No requirement that the government spy on everyone. No requirement that the government not spy on anyone. No mention of monogamy or polygamy or asexuality. No mention of baby murder VS women’s health. No preference for one religion over another. No mention of FOSS vs closed source. No promotion of atheism. No stance on narcotics enforcement. No mention of net neutrality. No mandate that women get the vote. None of that stuff.

Why? Aren’t these pivotal issues? Well, not really, no. The physical laws that made this particular geography come to be do not care about any of those things. In fact, they don’t care, period. Concern for your preferences is not a rule of the universe. At this initial level of evaluation we are considering these issues in isolation of all the other stuff.

There is an order in which geopolitical study of a region must be conducted or else we will find ourselves hung up on this or that favorite issue and miss the forest for the trees. We must consider the terrain first (its shape, what is in it, what resources are available, climate, terrain, lines of drift, proximity to other key terrain, etc.), then the existing infrastructure, then the existing and demonstrated capacity of the inhabitants, then the disposition of the available resources within the area, then a similar study of the adjacent areas, then gaming out a regional set of possibilities, etc. We won’t get into a detailed analysis here (which involves using numeric techniques to try to discover at least some of the bias in our assumptions), but we will touch on each of the elements in a cursory way while assessing the imperatives discussed below. Of course, once you go through the process once you iterate over it again and again — any time you learn anything new that might be significant.

Note that I haven’t mentioned anything about the details of culture or the nature of a given nation or people. At the outset we must look at the group that inhabits a given terrain in abstract: they have no race, names, history, culture, language — nothing. We don’t know who they are yet because we aren’t considering them as actual people yet, just hypothetical inhabitants. Remember that while each issue may be very important to a given individual none of them are important from the much higher level of geopolitics.

From a strategic perspective these cultural and personal aspects are distractions, issues that only rarely become geopolitically significant by themselves. While certain cultures have proven over time to be remarkably more robust and effective than others, past success does not necessarily predict future success. The problem lies with the nature of “success” and “failure”. Most people are destined for failure. Most organizations are not very organized. One team doesn’t typically win because it is so squared away as much as their rivals are just so screwed up that it is a wonder they even stick together (and often they don’t — consider the fractures that surfaced in Libya when Qaddafi died).

There isn’t anything stopping any pack of historical losers from flipping the switch and getting organized and engaged one day. We cannot assume that just because some particular collection of people has failed in the past it can’t — perhaps as a result of a radical change in leadership or cultural mores — suddenly start to flourish, and obviously the inverse is true as well. This is a critical point to understand because economics is not a zero-sum game, failure is always an option, and generation is a lot slower and more input-intensive than destruction (which is why violence and appeasement go together so well).

Geography stays the way it is for a long time. The world of men surges and ebbs and gets all mixed around at a breathtaking pace. Our interpretation of a map is grounded in the practical aspects of the logistical capabilities and resource technologies of the day. Before the invention of the maritime clock and reliable long-range maritime navigation, maps had intensely inaccurate coastlines and bloated interiors swelled to the edge of the page, chock full of details describing and naming what could be found on land and where. The oceans were relegated to the edges and shown as bleak and frightening aquatic wastelands full of monsters and giant serpents. Today we regard such maps as cartographic oddities, laughable and charming in their blatant inaccuracies — unless you actually grab a sea map and try to use it to walk somewhere without an accurate compass in hand. The bloated and “inaccurate” interiors were so bloated and detailed because they were full of next-hill landmark data, crucial for navigating by land back in the era they were drawn.

After the advent of blue-water navigation coastlines on maps became radically more accurate an displayed interiors became less detailed which corresponded with them suddenly becoming less geopolitically significant. A radical change in resource utilization (invention of the internal combustion engine or lithium battery, for example) or a radical change in heavy logistical technology (reliable maritime navigation, advent of the cargo airplane, development of the nuclear ballistic missile submarine, or commercially viable space shipping) is the kind of technological change required to change our interpretation of geography. Those changes occur so slowly that we usually see them coming and we mark eras by them: “The Golden Age of Sail” or “The Space Age“, etc. The nature of human societies changes at a pace that is imperceptible when you are young and requires study to even catch on to, but feels blindingly fast when one’s own life is reflected upon in hindsight. In fact, this is one of the chief lessons of history. So we leave out the particulars of the type of people or their cultural details and focus instead on the facts of geography and the practicality of the technology of the era in pure geopolitical assessment and analysis, and only dig into the cultural mess when conducting socio-geopolitical assessment.

One note of warning here: communication technologies not regarded as geopolitically significant.

I can hear it now: “Why not? People can talk to each other instantly today! Since all violence begins with misunderstanding clearly communication technology will bring everyone together and produce world peace ™ and that will certainly be geopolitically significant!” Oh, where to begin.

First off, does it seem like the world is becoming more polarized or more unified now that we can all scream at each other about every little detail of our lives? Clearly, in the days of ISIS, renewed violent political agitation in the West, and an EU council that doubles down on a catastrophic immigration policy “misunderstanding” is not the root of conflict. Perhaps knowing too many unpleasant realities about the people next door is, though. It’s a tough one to figure out, really — but one we don’t have to care much about when viewing the world with an eye to geopolitics, because anger isn’t what drives nations, imperatives are.

To actually do anything anywhere you have to actually get there and that means logistics always trumps communication. In fact, even critical business decisions today still travel at the pace of a man. (And yes, that is a gendered noun. I’m really courting controversy these days!) Why did it take roughly a week between the Russian invasion of Georgia in 2008 and the collapse of the European markets? Because it took about that long for people to make a concrete assessment of the event and the logistical and military threat that was posed to the profoundly overextended Yen carry-trade in Eastern Europe. This investigation did not involve a couple of Skype calls or Tweets or whatever — it required travel and personal meetings before final decisions were rendered. You don’t simply make a trillion-dollar decision based on a segment you see on TV one day. You don’t build a strategically significant aircraft factory in France without sending people to actually see the site with the Mark 1 Model 0 eyeball and report back. All this activity hinges on logistical technology. In the past checking the situation in East Europe would have taken a month, not a week, and an American or Japanese inspecting a new factory site in France would have taken several months to a year — and it wouldn’t have mattered much whether they could have Tweeted about it the whole time or not.

Occasionally a constellation of issues does arrange itself in a special way so that they have a socio-geopolitical impact in aggregate — but evaluating this is much more subjective (meaning, subject to personal bias and therefore much less reliable), even though occasionally such analysis really is warranted. It bears mentioning that any effort at socio-geopolitical assessment (technically not “analysis”) can only be performed after a geopolitical analysis has been performed because socio-geopolitics is more complex, and the complex always leverages the underlying simpler, pure analysis.

With all that in mind, let’s run through the various imperatives and see what we find. We are only going to think through what these different imperatives mean and how they have impacted history a bit, not actually engage in an exhaustive analysis — as that would entail bringing the context of a given point in time into play. This article is already over-long for a blog post as-is.

1. Delegation of Power

The first imperative itself is unique among the others. It is the only one that implies anything about domestic politics. While it does carry some implications, an implication is not a specification.

Geography dictates that Washington must delegate political control downward as much as possible. This doesn’t tell us how a government should work or how power should be wielded domestically or whether women should get the vote or whatever other detail, but it does indicate that large central government programs will always tend to exist in opposition to American strategic interests. It doesn’t tell us anything about how that should manifest, it is simply the fact that inefficiency is part of the nature of large organizations because large organizations grow exponentially complex and complexity is the enemy of both efficient operations and effective leadership. Central administration of a territory and population as large as the United States would require an absolutely massive and profoundly complex central government — and keeping such a government running would be simply impossible.

It is important to note here that the rest of the imperatives are set in conflict with this to some degree because a central authority is clearly necessary to pursue most of the imperatives that follow. So here is the first problem any polity based in the American geography must solve: How to create a durable political structure that can endure attempts at organizational corruption over several generations while maintaining a balance between delegation of civil authority and retention of central control of military and political action internationally?

Not surprisingly, the Constitution was written in an attempt to achieve exactly this kind of balance. Hands-off domestic policy from the central government, but strict subordination of the state and local governments (and, in a time of war, resources) to the central government in international affairs. The Constitution was written to protect the first imperative and establish a system by which a central authority can be created that is capable of pursuing the rest of the imperatives. Mythos aside, the U.S. Constitution is not a document that enshrines brotherly love, freedom, state independence, or civil liberty for their own sake. It is a sober document that describes a system of government that might give the United States a shot at real world power and prevent politicians from derailing that rise to power once some minor level of temporary success had been achieved.

Why have this split between civil and international authorities? Because the American geography, while having excellent natural lines of drift for navigation and logistical operations, is incredibly diverse and inhabited by a series of populations that all arrived from somewhere else. The culture of each state will inevitably be a little different. The moral standards in each community are therefore different. Expected anchors of social loyalty are different everywhere you go. The primary focus of economic activity changes from locale to locale. And all of that is a good thing. This is the real meaning of having useful diversity. It is just way too complex to administer centrally.

The geography of the U.S. represents a Goldilocks situation geopolitically:

  • Too diverse and spread out to administer centrally
  • Too well enclosed to be easily subverted and taken apart in piecemeal
  • Too diverse and spread out to subvert with a single rhetoric
  • An abundance of natural lines of drift and navigation corridors cut across the terrain, weaving the concerns of one region with that of its neighbors — too much so for any long-term fragmentation to endure
  • An incredible abundance and diversity of various terrain, most of which have extremely high inherent economic value, situated in such a way that they are mostly mutually supportive, or if not, are located close enough to major waterways to make heavy marine traffic possible
  • All this is protected from the most potent of geopolitical rivals by inherently defensible buffers (the Atlantic and Pacific Oceans)

To understand how that is unique and what it means, let’s compare with two counterexamples: Singapore and Russia.

Singapore is small enough to administer centrally. This is, of course, a somewhat unique example, but it is a useful exercise to think carefully through why Singaporean style civil law would not work if applied uniformly across the entire United States. It is also interesting to consider how Sinagpore’s unique strategic military and economic position in the world are able to leave it with enough resource to even engage in the game of central domestic control.

Russia’s meaningful core (politically, culturally and economically) is Moscow. But the Moscow region is in such a horrible defensive position that it must capture massive buffer zones surrounding it and force them into its political orbit. This effort, of course, stands against all reason in terms of military viability and economic stability. As a result, Russia winds up being forced to behave as a centralized security state to suppress constant attempts at secession whether it wants to or not. Russia is somewhat larger than the United States, less populated, but at least as diverse as the U.S. in terms of economy and terrain. A centrally administered security state in Russia must necessarily also become a centrally administered welfare state and that is always a failed proposition over the long-term — but Moscow simply has no other choice. To feed its enormous resource needs in maintaining all that extra territory and central administration it is necessary that Russia control the Ukrainian breadbasket to force Ukrainian farmers to sell their grain against their own interests: at extreme expense up-river and over land to markets with very little profit potential instead of selling it down-river and overseas to far more lucrative markets. (This is also why the Russians can never let go of the Kerch Strait, Crimea or political proxy control of Ukraine as a whole — whenever they haven’t outright annexed Ukraine already.)

The next six American geopolitical imperatives are all quite different from the first. The second through seventh are what I call “imperatives of achievement”. They are clear goals that can be achieved and considered “complete”. This makes formulation of a strategy for achieving them and recognizing moments of special opportunity that apply to any of them much more straightforward than trying to figure out a way to accomplish either the first or the last imperative. With imperatives of achievement a glance at a map tends to make the reason both for their central importance and their order of priority clear.

It is worth mentioning that it is no surprise that the first and last of the American geopolitical imperatives are different from the ones in the middle. The first task in assessing a nation’s circumstance is to determine what system governance best suits the terrain and culture, the next step is to figure out what things that government needs to achieve to secure primacy in its part of the world, and the last task is devise a strategy for how a nation can most effectively safeguard its achievements at the lowest ongoing cost to itself. If the first imperative is out of alignment with reality then it is a useful indicator that severe misalignments or imbalances in the nation’s power structure can be found.

On the point of misalignments between the chosen political strategy and the terrain, China has been a great example of that every time its separate inner kingdoms have been united under a single body that attempts to rule over them with the same level of strict central authority which which each sub-region has been historically ruled. That is the situation right now, actually. At the very high level, China’s terrain suits a federalized, delegated style government better than an authoritarian, centralized one, and it appears indeed that China is actually closer to facing a massive economic collapse due to the influence of this misalignment rather than becoming capable of sustaining credible power projection overseas. In view of the prevailing Chinese culture throughout history, that this is a theme in Chinese history comes as no surprise. It is actually quite possible that due to changes in lifestyle and life expectations among Chinese that the next version of China may wind up looking more like the American federal model than the current centralized Chinese.

As an individual who believes that would be great for both Chinese people and the countries I have a personal interest in I can certainly hope that this is true or even believe that it should be true, but as a geopolitical thinker I am only permitted to assess this as a possibility and settle with a dispassionate “time will tell”.

It is worth mentioning here that so far no combination of political structure, culture, technology and strategy have ever come close to predicting any possibility of stable governance over the entire world at once. Until we are quite far into the spacefaring future it is simply an impossible goal. Not just improbable, actually impossible. That is why you will never encounter a set of rational geopolitical imperatives of achievement that include “establish total world political dominance”. That is the sort of fantastical thing that pan-national socialists, ethnic supremacists and Dr. Evil fantasize about, but it just isn’t in the cards for the foreseeable future. Geopolitical imperatives tend to have a natural way of petering out at some limit that is itself dictated by the geography and circumstance at hand. The limit is not always obvious, but it usually is.

2. Making the East Coast an Unfavorable Target

Making the East Coast unprofitable to attack is not the same as making the East Cost impossible to attack. The British discovered that letting go of the Americas is a lot more profitable than trying to hold on and control it directly. As long as the balance of benefits is negative Washington has nothing to worry about, even if all it controls directly is the Eastern Seaboard. As long as this is true Mexico and Canada also cease to be threats. Despite being directly adjacent, neither Mexico nor Canada has anywhere near as good a geographic base from which to support the kind of economy necessary to field a military that would pose a threat to the existence of the United States without extremely heavy support from an external sponsor. As just established, such support would not be worth the eventual outcome.

3. Coast-to-coast Territorial Integrity

With the East fully secured, the rest of the continent opens up, either to immediate takeover or to gradual colonization. It doesn’t really matter because the East is the key to meaningful control of the Mississippi Drainage Basin. Even if another power were to control the entire Louisiana Purchase, Washington could choke off its access to international waters and contain its international influence to the Gulf of Mexico at a very low cost in terms of naval investment. Using the now fully secure Eastern Seaboard to support an effort to capture and control the middle of the continent predicts an inevitable need to also control the West Coast, both as a buffer beyond the Rocky Mountains (though the Rockies do provide a nice barrier themselves) and as a way of accessing the Pacific Ocean to become an influential power in both sides of the other hemisphere.

With territorial integrity secured from coast to coast the U.S. enjoys a position no other country can match: it has the entire Pacific and Atlantic Oceans as its buffers. Typically speaking, oceanic buffers require dramatically less political maintenance than territorial buffers do.


The Mississippi River Drainage Basin, compliments of Wikimedia user Shannon1

On that note, a brief digression on the (currently explosively political) Westward Expansion is in order…

The period in which the U.S. expanded its control westward was a fascinating period, a roughly 200 year span chock full of all kinds of things people seem to forget are integral to human behavior: indigenous ancillary military operations, public/private contract military forces, imperatives being justified by invented rhetoric, corporate assassinations, anti-immigration political platforms, “No Country for Old Men” problems, Game of Thrones level intrigue, Thomas Jefferson violating his own principles of government in the interest of leapfrogging almost an entire imperative in just two strokes (the Lewis & Clark Expedition and the Louisiana Purchase), actual swashbuckling pirates mixed up in serious national efforts, etc.

Study of this period has become heavily politicized for a number of reasons, but like it or not there was simply no other way things could have gone and the world to have ended up with the United States of today. If you think a world without the United States would have been better in some way, consider what would have come of your current ideals (or if you would even exist to hold your current ideals) had the Crown, the Nazis, the Soviets, the Spanish or even the Aztecs secured hegemony over the Western Hemisphere. The geography dictates that someone would have eventually established hegemony over the Americas, and that nation would be one of an only possible three superpowers in the modern era. The U.S. doesn’t sound so awful considering the alternatives — but then again, we are not allowed to care about what should be, only what actually was and is and possibly may be.

In any case, it doesn’t matter who was in charge in the East (there is even a possibility it could have been a more autonomous European colony at the time of the expansion), projecting our biases into the past onto historical figures who could never have imagined our current political, emotional and personal biases today is a pointless exercise. None of that gets us any closer to understanding the world the way it actually was or how the version of the world that our history passed through back then became the version of the world it is right now. So stay away from that. Remain interested in history for its own sake and you might just learn more about what happened in the past and come to understand something more about yourself in the process.

4. Security of the Caribbean Routes

The enormous power of the central United States in terms of agriculture and economic output is all just wasted potential with no way to effectively recruit it if the heavy transit routes down the Mississippi and Missouri Rivers do not connect to a secure passage to the open ocean beyond the Gulf of Mexico. This is even true when it comes to shipping heavy things (or a large volume of light weight things) from the center of the United States to the coasts (particularly true before the advent of transcontinental railroads and interstate highway system), hence the central importance of the riverboat culture Mark Twain wrote about.

From the late 1700’s to today the river country has been critical to nearly everything the U.S. does that has a strategic impact.


A heat map of shipping lanes through the Caribbean. (Derived from an image by Wikimedia user Grollech.) Compare again with the map of the Mississippi Drainage Basin above to see how much of the country’s power is bottlenecked first at New Orleans and then either the northern coast or western tip of Cuba.

While the Louisiana Purchase went a long way to securing imperative #3 by acquiring the Mississippi Drainage Basin from a cash-strapped France, the importance of that achievement can never be fully realized until routes from New Orleans to the rest of the world are secured through the Caribbean.

As you can imagine from the enormous American shipping activity surrounding Cuba, strategic analysts tend to think of Cuba as a cork sitting half-way in the bottle of American commercial power. That is why the U.S. has had a seemingly weird obsession with Cuba over the last two centuries, why the Spanish American War was inevitable, why Soviets found it absolutely mandatory to screw the place up, and why even today Moscow sends high-profile air and naval missions to the island. Much like the Strait of Hormuz, the mere threat of militarizing the waters around Cuba would cause a market panic that would ripple the world over and hurt just about everybody in some way. The calculus of any spoiling naval play around Cuba comes out in whether the harm done to the attacker would be small enough relative to the harm done to the American-led system to be worth it — and the solution to that is always entirely context dependent (so far it has never been quite worth it to go totally overt against the U.S. in this region, as clearly demonstrated in the Cuban Missile Crisis).

For the most part this balance has worked out in Washington’s favor, but not to the extent that the Soviets stayed entirely away. It is no accident that Che Guevara‘s movement was sponsored by the Soviets and that Fidel Castro‘s Cuba was quickly re-evaluated by Moscow (which initially distrusted him, thinking he was a CIA plant) which was looking for a way to distract the Americans closer to their own territory. Their movement evolved, as these things tend to do, from a passionate individual fight against political oppression to a fight for the title of Supreme Oppressor — a position that cares little for preservation of liberty, freedom, equality, egalitarianism, Marxist principles, or even anti-Americanism. They were indigenous leaders useful to the cause of undermining American hegemony in order to buy the Soviets breathing room elsewhere on the globe and were reshaped that way by a vastly superior outside power. The Americans had major tank and missile forces stationed a mere 2,000km from downtown Moscow — it was unthinkable to the Soviets that they could not at least support a strategic distraction equally close to the American heartland. It would not have mattered if their political rhetoric was initially based on Communism, Anarchism or Pastafarianism; the Soviets needed to make a spoiling move in the game somewhere and indigenous leaders would have been found to run the show.

5. Dominance Over the Western Hemisphere

The Monroe Doctrine is often cited by folks who seem to have never read it or understood the context in which it was delivered. It is often mistaken for a document that mandates Washington “meddle” in South American internal affairs. This is not accurate. The Monroe Doctrine states that Washington will actively resist the efforts of non-Americans powers to meddle in South American affairs — more specifically, it is a policy of opposition to European colonialism in the Western Hemisphere, with Washington assuming the role of lead rallying defender. The doctrine was delivered first as part of a speech (part of James Monroe’s State of the Union Address in 1823), not a formal policy letter, and the speech was directed at the Russians — a mere 40 years after the founding of the United States itself.

While it formally opposed European colonialism in the Western Hemisphere, it also did not reject existing claims. It established a grandfathering of colonial claims: new claims would be resisted directly, and successful independence movements in the New World would not be interfered with.

By 1823 the United States had not established clear naval supremacy to the level required to proactively enforce the Monroe Doctrine, but Washington did have the military capacity to make European adventurism in the New World unprofitable anywhere, and more importantly to the European calculus, the budding economy of the new American nation had enough capacity to outproduce any given individual European power if fully recruited in a time of war.

With the Europeans lacking a way to subvert new American territories to create organized and sponsored overt opposition to the United States, the default hegemon of the Western Hemisphere became Washington. This is an incredibly powerful position, but, as before with the relationship between the Mississippi Drainage Basin and the shipping routes out of the Gulf of Mexico, the power inherent in this position would not be fully realized until the United States established global naval supremacy — a detail that was so far beyond the thinking and reasonable capacity of the day that it took 120 years and World War II to propel the United States into that position.

6. Global Naval Supremacy

Control of the world’s oceans gives the U.S. a number of irreplaceable and unmitigable advantages. With its primacy in the Americas assured and naval supremacy across the world’s oceans the U.S. is the most secure and insulated nation in the world in terms of military threat. Sure, another nation could attack, but no nation can possibly invade. Once the U.S. controls the “blue water”, the details of what happen across it are of no consequence to the Americans so long as no hegemon appears anywhere else.

The power of this position for a major trading nation is difficult to overstate. With the U.S. having dominance over the oceans, just sitting back and not doing anything with that control promotes not just global trade, but global trade at the good graces of the U.S. Navy. In this way Washington is able to keep it in everyone’s interest to just let the status quo continue. As time has moved on the U.S. Navy’s maritime technologies have advanced by several generations, and not only that, the U.S. Navy is one of only three navies in the world that is fully drilled (the other two being the British and Japanese). The entry cost to even trying to compete with the U.S. Navy is too high for most nations to contemplate — a situation that sounds a lot less bad to those nations when they have good reasons to assume that the ongoing U.S. Naval policy of open navigation will continue.

Naval dominance is the most critical factor in global American military dominance, and has remained so for over a half a century. (This leads into and supports the dominant American land warfare force composition and activation doctrine, but we don’t have time to get into that here. Maybe another time.)

7. Supremacy in Space

It is becoming hard to name a communications, scientific or military technology today that does not involve a satellite in some way. This is significant. If a nation’s satellite network went out for some reason quite a few things taken for granted day-to-day would suddenly be rendered inoperable. While only a few nations have the ability to reliably place large payloads in space (especially very distant or difficult to achieve orbits), several have the ability to destroy most of the things in space by causing a debris cascade (often referred to in popular press as the “Kessler Syndrome“) — even North Korea or Iran could potentially do this. These two concrete capabilities are all-or-nothing: you either get a satellite into a useful orbit or you don’t, you either kill everything in LEO or you don’t. And, obviously, killing everyone’s satellites would be considered an act of war by someone even though the act itself would wind up affecting everyone at the same time.

There is a little room, though, for a third power play in space: denial of entry or re-entry from space. The American Ballistic Missile Defense system actually has this capability, though it is normally only discussed in the context of ballistic missiles. There is little technical difference between shooting down an ICBM and shooting down a launch rocket, either on ascent, in flight, or on descent. The American BMD system creates a much more powerful and selective tool for space dominance than any other nation currently has in its arsenal: the capability to act as a gatekeeper to and from space.

Are there any scenarios where Washington might want to exercise this ability? Not in the near future, probably, because spacefaring in general is still quite new. It wouldn’t really have made much different to the ancient Greeks if the Japanese had complete dominance of the Atlantic and Pacific oceans for the same reason — they had not yet reached the point that such maritime control would have been relevant.

American dominance in space is a tool that can be used to guide the future development of trade, colonization and commerce in space in much the same way that Washington’s naval dominance allows Americans to take the lead in trade, even when they are not the primary party engaged in a given industry. The trick here is making the entry cost to credible competition in space (especially militarized competition in space) too high for it to be judged worthwhile. This is why the Space Shuttle was sought, and also why it was abandoned in favor of deregulation of spaceflight and space development within the domestic United States. The first American imperative that delegates control and protects the economic and scientific capability of the U.S. is the same one that is being leveraged by the Americans today to guarantee that the most advanced space technologies still are developed there, whether or not NASA is in charge or even involved in the effort.

Dominance is in an interesting period of filtering right now, actually. Only a handful of nations could ever hope to catch up to the U.S. in terms of dominance in this area, and all but two are either mired in regional issues or will soon be bogged down in domestic ones. The E.U. is on the verge of splitting first East/West before dissolving entirely, Russia is halfway through re-establishing regional dominance (which its imperatives mandate), France, Germany and the U.K. (along with the rest of Europe) will soon be knee-deep in a domestic and cultural correction event, China is very likely to undergo either a political upheaval or a civil war once the European market closes to it temporarily (of course, whether a civil war is the same thing as a “political upheaval” is a matter of opinion in geopolitical terms), and India would likely have its efforts shut down by the Chinese were they to begin a serious manned space program before China is already well established there. The only nations that are potentially free and capable to compete directly and uninterrupted in space are Japan and the U.S. during the near- to mid-term. It is very likely that American space companies will come to dominate this area unless the Japanese decide to engage in a major space-based energy gambit — but at the moment a Japanese space gambit seems less likely than a Japanese attempt to dominant deep-space robotics.

Whatever the outcome, once the world recovers from the upcoming period of instability it is almost certain that the Americans will have established as firm a lead over spacefaring as they have over seafaring, and it is very likely that the Japanese will either be operating to a comparable standard or have chosen to take the lead in deep space construction and robotics. In a certain sense it is 1492 all over again; these issues are somewhat distant in the future and involve a number of very fast-moving and highly technical fields, but the impact on the future history of humanity is no less significant.

8. Maintain a Balance of Powers Overseas

The final imperative is a spoiling imperative: a mandate that the US prevent something from happening elsewhere instead of causing something to happen in a geographically dictated location.

As long as the Americans dominate space and the oceans the quest for achievements is at an end. At this point it is in the American interest to be relatively non-interactive with the world so long as no other regional hegemons or superpowers exist. As long as powers are relatively balanced against one another across the world, Washington’s best play is to sit back and promote trade — not because it can pull shady tricks and coerce people to do whatever it wants, but because a system of free trade is the most likely way to get nations addicted to the status quo of American naval and space supremacy, and generally raise the wealth status of everyone on the planet over time. All of this is pretty awesome, of course, but only if you belong to a nation which happens to lie across the oceans from the Americas and is satisfied with being stuck on your third or fourth imperative forever. The final three imperatives of every nation are the same, so as long as no nation expands to the point of conflicting with Washington’s 6th imperative headbutting with the U.S. can be avoided. There are a few places where this predicts Russia, Chinese and Japanese conflict with the United States, and this is indeed exactly what we see.

Of course, nations do not experience emotions, so one can never feel “satisfied”. That’s not how it works. Nations will always continue to struggle against one another, so the Americans must always be watchful but only intervene once things are getting unbalanced. That is their best play, anyway, whether or not the remain focused on that is another question entirely.

James Mickens: Life as a Developer

January 31st, 2018

Below is a somewhat whimsical talk given by James Mickens last year titled Life as a Developer: My Code Does Not Work Because I am a Victim of Complex Societal Factors That are Beyond My Control.

(If you recall, Mickens is the guy who did the insane/awesome Monitorama talk about the cloud a few years ago.)

As usual for dear begoateed James it is full of silly digressions, but the theme is quite an important one today: unnecessary complexity is increasing and that trend will conspire with our own limited capacity for comprehension, information retention, research of the new and obscure, and interpersonal communication to ensure that if we keep going as we are the whole world will eventually succumb to entropy and leave us in a big ball of digital hellfire.

Life As A Developer: My Code Does Not Work Because I Am A Victim Of Complex Societal Factors That Are Beyond My Control…. from NDC Conferences on Vimeo.

Laws Derived From Moral Panics Are Dangerous

January 29th, 2018

Politicians love morality laws.
They really love sex laws.
They really love child protection laws.
And they really really love child sex laws.

Manufactured moral panics are awesome if you are a career politician. By leveraging a moral panic you can whip people into a frenzy, warping their perception of reality to the point that the only thing that matters is whatever you made seem important. This is true even if the issue actually affects a statistically insignificant number of people. Consider the number of annual non-suicide homicide deaths involving firearms compared to the number of annual drunk driving deaths — the first number isn’t even paid attention to, as the anti-gun lobby will cite every death involving firearms regardless of circumstance, while the second number isn’t mentioned in public safety debates unless the subject is specifically drunk driving (but that’s not a hot topic now, so nobody cares). Repeated often enough, any message that induces a strong emotional response will stick and make people think the (actually rare) issue is happening to everyone all the time.

In this article I’m going to show you why moral panics, of any nature, are good for politicians and bad for you. If you think carefully about an example of a moral panic taken to its conclusion you will understand that the side effect of the moral moral panic (an inevitable national vulnerability) is far more destructive to society than the subject of the panic (one side of basic human behavior that has been generally kept in check by the another side of basic human nature since the beginning of time).

Let me revisit my first paragraph for a correction. A moral panic is not useful to politicians “even if” the issue affects an insignificant number of people in reality, it is “especially if”. The fewer people actually affected by the issue the more confident you can be that the issue will not backfire: Almost nobody will come forward to present a concrete counter argument, particularly if the issue is significantly emotional for a large enough portion of the population that people begin hiding their true thoughts on the matter in the interest of avoiding social ostracism. After all, anyone presenting a counter argument must be evil because they hold such views.

Imagine trying to bring a counter argument about the effects of cocaine use in the 1980’s at the height of the Cocaine Wars. If you came out and said “I’ve used pure cocaine as a mild stimulant in limited doses for years and it has not affected my health, made me an addict, or driven me to kill anyone. It is my own body, anyway.” You would immediately be socially ostracized and find yourself under federal investigation for what you had just said. The number of people who actually used pure cocaine at the time and were willing to admit it to support you would have been far too low to change the tide of public opinion — and good luck getting any of those people who secretly agree with you but haven’t used cocaine before to break their silence. Of course, this is exactly how cocaine was used before the drug wars started and that period of American history coincides with the greatest economic expansion and greatest increase in quality of life standards in human history. But those details wouldn’t save you because the moral panicky flavor of the issue itself would already make you seem like the Devil Hisself for having even cast doubt on the illicit status of the substance.

Remember the bizarre world of square cops Duke stepped into when he accidentally attended the narcotics enforcement convention in Fear and Loathing in Las Vegas? Granted, he was only trading one equally bizarre world for another, but the experience was illustrative enough for Thompson to include it in his book.

Here is a clip from the movie to jog your memory, since nobody reads actual books anymore.

Age of consent laws and the problems that surround them are mostly non-issues. But they are really, really useful non-issues if you are a politician. Age of consent laws are pretty freaking new and yet the world has gotten along just fine without them so far. But how?!? Before age of consent laws it wasn’t just non-stop adolescent hedonism in the streets. People have families and enough people are non-evil enough of the time that things have generally worked out OK. The world got along just fine without drug laws until just recently also.

Consider the following:

  • There was alcohol prohibition in the U.S. until 1920, at which point America embarked on a decade-long journey of gangland violence, abuses of alcohol went from being the occasional crying shame to being a literal pathway to either extreme abuse or a life in the prison system, and the government went to war against its own citizens. Great. This doesn’t seem like it was a very big win.
  • There was no drug prohibition in the U.S. until the first nuisance tax on the distribution of marijuana in 1937. Fast forward and the U.S. is once again mired in gangland violence and the government is openly at war with its own citizens based on random chemical combinations they make at home from ordinary materials. All that plus the effect artificial scarcity of the relatively benign naturally occurring drugs has had on the market: instead of cessation of narcotic consumption, people consume profoundly more dangerous and addictive alternatives and destroy their bodies much quicker at a higher public medical cost than previously. All of this while drug use has expanded instead of declined. There is also the small detail of the civil war in Mexico that is driving people north in record numbers — a war that the lowest members of American society are funding through drug purchases and the middle-class is subsidizing through massive government outlay in the form of anti-drug operations funds. Entire agencies exist solely for the purpose of pursuing the drug war! Once again, not a big win here.
  • There were no age of consent laws until quite recently, but that didn’t matter so much in the face of strong family traditions, social taboos regarding sexual interactions, relatively strong sexual morals across all segments of society, and a strong social preference for publicly adhering to near-puritanical views on sexuality in general. Age was not the core issue, but the society had strong views on sexual propriety. Fast forward to today and in some states you can get sent to jail for having a girlfriend a few months younger than you, women are assumed to have zero capacity for thought until they turn some arbitrary number, men can have their careers destroyed by a rumor, and yet at the same time little girls are all over YouTube twerking,  incidence of early teen pregnancy is skyrocketing, and even very young girls are making a game of engaging in highly promiscuous teasing games with older boys and men. Kids are obviously having sex at a rapidly increasing rate in spite of the law, but when both partners are extremely naive about life in general the outcome is far worse than whatever was going on before.

See a trend? I’m not saying that substance abuse or sex are light-weight issues — quite the opposite — but that government intervention really seems to consistently backfire on every social issue that is normally handled by families. Education is another shining example of the Cobra Effect in action, but that deserves an article of its own some day.

The Thesis

In the current era, where almost everything valuable you own is a networked computing device, moral panics are a source of strategic technical vulnerability. In the prefacing discussion above I discussed drugs because that is a major freak-out issue for some people and child sex because it is a major freak-out for other people. I avoided diving into a discussion about terrorism not because it isn’t a similarly dangerous issue, but because we all know what happens if you are perceived to be talking about Islam (I wouldn’t want anyone to take that the wrong way, of course). I assume there is a lot of crossover between the drug-panic and child-panic demographics, and I’m pretty sure the terrorism-panic and holy-crap-invasion-by-hijra-panic demographic covered pretty much everyone (even though that second one is WrongThink), but hopefully I’ve got you upset over one issue or another by this point.

An Example

Consider Android boards. Actually consider them for a moment.

Let’s say our goal is to stop terrorists or save the children or catch the drug dealers. Based on the premise that people do nearly everything through their smartphones these days we create a regulation that mandates all phone makers provide a hardware backdoor for law-enforcement and intelligence services in every new Android board produced. This is hardly a far-fetched proposal, and in fact there are proposals to do exactly this already on the table today.

So now, in this hypothetical-but-likely world, every new Android board that is now network enabled across a huge spectrum of wireless bands, can be equipped with wired ethernet, USB, etc., contains a GPS unit, accelerometer, thermometer, microphone, camera, etc… and is backdoored at the hardware level.

“Well hurrah!” you might say. Surely with the fantastical power of these backdoors into everyone’s phones the kids are safe, drug use has totally stopped, the terrorists are automagically banished from the Mortal Plane, and flowers have sprung forth in full bloom!

Absurd hyperbole? Yes. Of course it is. Backdoors into phones will just drive criminals to do business in other ways as always. Universal backdoors in computing devices are far less useful to law enforcement and intelligence officials in practice than regulators imagine when they formulate such rules. We have many examples to draw from already, and overall it certainly appears that while backdoors are of limited utility to law enforcement, they are super useful to criminals, enemy governments, and despots.

Part of the problem of being a good guy is that having data one everybody means that you don’t have time to check the data on anybody. You’re still in target identification mode while the bad guys already have a laser-focus on a target ahead of time. As law enforcement you wind up becoming the Precrimes Division — and that’s downright spooky. Trying to profile for criminality in aggregate winds up creating magical (and highly unpredictable) categories of “unusual” behavior patterns that, while not actually criminal, can mistakenly flag a normal citizen for scrutiny. This is terrifying for a number of reasons, not least of which that it tends to force people into conformance with artificial social norms that are invented by aggregate software analysis (the way we do with adsearch results, for example) rather than actual knowledge of criminal activity.

Even if we stopped here and didn’t pursue the Android example that follows, the situation already poses a strategic economic challenge. Without room for safely breaking with prior social behaviors there is little hope of social, economic or technical innovation moving forward. The last thing Americans seem to do well is conform with a static, centrally controlled society. (A spontaneously self-orded, semi-static society, sure — but the moment you tell an American who is sitting down that he’d better remain sitting is the moment that guy will stand up just to spite you.)

Back to our warehouse full freshly minted of government-compliant Android boards…

A year goes by after the passage of the new mandate for phone backdoors without incident. Things seem static, quiet, calm, wonderful. But one thing is never static: the market. Nobody is making phones with last year’s (or even last quarter’s) boards. Time and technology have moved on and new boards are being produced, leaving the old models which have been produced-but-unsold sitting in a warehouse waiting to be sold in bulk for a few dollars a piece, rotting in their surplus obsolescence — obsolescence for the phone market, anyway.

In this world you work as a product manager at a company that needs to develop a new, “smart” building utility control unit that should first and foremost be capable of controlling the lights and thermostat, but also must be extensible — perhaps becoming a more universal facility control device: door lock awareness/control, outlet consumption tracking, etc. (and I’m actually simplifying the example as this tech is already here on the fringe now and the example is itself a bit of date).

As a project manager you have a choice:

  1. You can get a bunch of old-school hardware engineers together to develop a new device, then get systems people on developing the software to run it, and eventually go through the product testing phase and get protocol people to make it comply with other devices it might need to talk to in the future, etc.
  2. You can acquire a box full of obsolete Android boards that already have a universally understood operating system on them (Linux/Android), complete drivers, and comply with whatever communication standards you might encounter right out of the box — and can outsource or offshore hasty development of some crapware to make the thing almost sorta-kinda work. For pennies on the dollar compared to #1.

As we say in engineering, the three favorable attributes to a project are: Cheap, Fast, Accurate; pick two.

Nobody will ever opt for Option 1 these days. Option 1 is expensive and/or time consuming despite the end result being a really proper “MIT design”. Investors and executives might be willing to throw a few million dollars away on toy mobile social app development these days (seriously) but they will never authorize development of a serious engineering project that isn’t instantly gratifying unless you can somehow link a buzzword like “blockchain” or “IoT” or “cloud” or “big data” to it somehow.

So you’re left, of course, with Option 2 — and that only because you spun the product as an IoT device. Despite the tendency to marry disaster early and outsource the software bits in an Option 2 project (behold! the fateful project grenade!), some rare managers might do a really solid job of Option 2 by, for example, hiring experienced local programmers for the software instead of offshoring the development of crapware.

So what is the end result? Any customer that buys your new thermostate/control device just placed a universal computing system onto its network with the following attributes:

  • Has an out of date OS your company is never going to pay to patch
  • Is equipped with a vast array of powerful wired and wireless networking interfaces
  • Has a GPS device
  • Defaults to trying to send data back to your servers (because IoT…ugh)
  • Has a microphone
  • Has a camera
  • Has a variety of old backdoors that your government mandated be put in place just in case someone had Japanese loli cartoons on their phone however many years ago

Those backdoors are still alive and operational, and by design are impossible for you to do anything about or even detect during a normal inspection. Of course, these backdoors will eventually be figured out by or leaked to other governments and The Bad Guys (and of course, those two factions may be the same depending on context).

Elements of the hypothetical situation above are already true. The control device development process — that choice between a custom device and shortcutting by using obsolete, discount Android boards — is something I’ve seen happen several times in real projects already. The missing piece is (hopefully) the universal backdooring part. Of course, backdooring is already happening a lot (you did check some of the linked resource material in the article, right?) but the universal mandate for backdooring is not yet in place — imposing something like that would require something like a moral panic to put in place.

Final Impact

The eventual impact of these backdoors is far-reaching and unpredictable, but it is certainly dangerous and strategically risky to have backdoors of any nature in widely deployed devices. We already are seeing IoT devices (security cameras, printers, door locks, routers, fence monitoring systems, teddy bears, thermostats, coffee makers, etc.) getting cracked on the network and enlisted in DDoS botnets on a scale that dwarfs anything that one could have ever hoped to accomplish by cracking notebook and desktop computers.

The risk inherent in placing wirelessly enabled, GPS enhanced control devices into service on physical plant and industrial control and monitoring systems is absolutely impossible to overstate. It is also impossible to overstate the strength of certainty that I feel when I say that this is, generally speaking, the future of control systems. The economics are just too good once the fundamentals get worked out. Dirt cheap commodity hardware pre-equipped with an OS everyone already knows how to write code for speaking universal protocols — all out of the box.

This is happening. It will be an awesome market upset and advancement of the state of the art if it is done well. If it is inadvertently subverted by the side effects of a moral panic, however, it could easily wind up making the backbone of our infrastructure control systems strategically vulnerable to everyone from governments to criminals looking for a technical ransom payoff.

Moral panics are designed to be uncomfortable to bear, but the subject of today’s moral panics are all issues that boil down to simple moral degeneracy. These are the kinds of issues your parents set you straight on as a kid, not the kind of issues the government has any hope of influencing in a positive way. These are issues that can only ever be handled by families and neighborhoods performing their traditional role of instilling moral values in their members and enforcing those values with a combination of instruction, room-to-grow plus mistakes forgiven, wielding the natural tools of social discomfort to encourage morally conformant behavior, and acting on genuine personal concern and love that a government can never hope to replace.

Trading real families and parents for government regulators and a penal system is a bad tradeoff. Coupling that with the economic chaos that would follow The Great Infrastructure Crack might just do us in for good.

Zomp/zx: Yet Another Repository System

December 12th, 2017

I’ve been working on a from-source repo system for Erlang on and off for the last few months, contributing time to it pretty much whenever real-life is not interfering. I’m getting close to making a release. Now that my main data bits are worked out, the rest isn’t all that hard. I need to figure out what I want to say in an announcement.

The problem is that I’m really horrible at announcements and this system does things in a pretty different way to other repository systems out there, so I’m not sure what things are going to be important about it to users (worth putting into an announcement) and what things are going to be important to only me because I’m the one who wrote it (and am therefore obsessed with its externally inconsequential internals). What is internally interesting about a project is almost never what is externally interesting about it. Marketing; QED. So I need to sort that out, and writing sometimes helps me sort that kind of thing out.

I’m making this deliberately half-baked, disorganized, over-long post public because Joe Armstrong gave me some food for thought the other day. I had written him my thoughts on a subject posted to a mailing list but sent the message in private. I made my message to him off-list for two reasons: first, I wasn’t comfortable with my way of expressing the idea just yet; and second, I am busy with real-life stuff and side projects, including the repo system, and don’t want to get sucked into online chatter that might amount to nothing more than bikeshedding. (I’m a world-class bikeshedder!) Joe wrote me back asking why I made the reply private, I told him my reasons, and he made me change my mind. He hopes that more people will publish their ideas all the time, good or bad, fully baked or still soggy — because that’s the only way we can ever find any other interesting ideas these days is by searching for them, usually in text, on the net somewhere. It isn’t like we can’t go back and revise, but whether or not we do go back and clean up our literary messes, the availability of core ideas and exposure of thought processes are more important than polish. He’s been on a big drive to make sure that he posts most of his thoughts to public mailing lists or blogs so that his ideas get at least indexed and archived. On reflection I agree with him.

So here I am, trying to publicly organize my thoughts on my repository system.

I should start with the goals of the system.

This system is intended to smooth over a few points of pain experienced when trying to get a new Erlang project off the ground, and in particular avert the path of pain peculiar to Erlang newcomers when they encounter the “how to set up a project” problem. Erlang’s tooling is great but a bit crufty (deeply featured, but confusing to interface with) and not at all what the kool kids expect these days. And anyway I’m really just trying to scratch my own itch here.

At the moment we have two de facto standards for publishing Erlang systems: erlang.mk and Rebar. I like both of these, especially erlang.mk, but they do one thing that annoys me and never seems to quite fit my need: they build Erlang releases.

Erlang releases are great. They cut all the cruft of a release out and pack everything needed to actually run a system into a single blob of digits that you can move, in a single shot, to a new target system — including the Erlang runtime itself. Awesome! Self-contained deployment and it never misses. This has been an Erlang feature since before people even realized that they needed repeatable deployment infrastructure outside of the classic “let’s build a monolithic, static binary executable” approach. (Erlang is perpetually ahead of its time, even by today’s standards. I look at the poor kids stubbing their toes with Docker and language du jour and just shake my head — though part of that is because many shops are using Docker to solve concurrency issues that they haven’t even become cognizant of, thinking that they are experiencing “scaling” problems but missing the point entirely.)

Erlang releases are awesome when the deployment target is an embedded system, but not so awesome if the target is a full-blown operating system, VM, container, or virtual environment fully stocked with gobs of memory and storage and flush with system utilities and resources. Erlang releases sort of kitchen-sink the deployment itself. What if you want to run several different Erlang programs, all delivered as releases, all depending on the same library? You’ve got tons of copies of that library. Which is OK, but still sort of weird, because you also have tons of copies of the runtime (among other things). Each release is self-contained and lean, but in aggregate this is a bit odd.

Erlang releases make sense when you’re deploying to a phone switch or a sensor device in the middle of nowhere and the runtime is basically acting as its own operating system. Erlang releases are, in that context, analogous to putting a Gentoo stage 3 binary image on a system to leapfrog most of the toolchain process. Very cool when you’re in that situation, but a bit tinker-tacky when you’re just trying to run, say, a client program written in Erlang or test a web front-end for something that uses YAWS or Cowboy.

So that’s the siloed-kitchen-sink issue. The other issue is that newcomers are perpetually confused about releases. This makes teaching elementary Erlang hard. In my view we should really focus on escript for beginner code — just let the new guy run something out of a single file the way he is used to doing when learning a new language instead of showing him pages of really slick code, then some interpreter stuff, and then leaping straight from that to a complex and advanced packaging setup necessarily tailored for conducting embedded deployments to slim hardware devices. Seriously. WTF. Escripts give beginners all the power of Erlang necessary for exploring the more interesting bits of code and refactoring needed to learn sequential Erlang with the major advantage of being able to interface with the system the same way programmers from other environments are used to dealing with langauge runtimes like Bash, AWK, Python, Ruby, Perl, etc.

But what about that gap between scripts and full-blown production deployments for embedded hardware?

Erlang has… nothing.

That’s right! There is no agreed-upon way to deploy or even run Erlang code in the same manner a Python coder would expect to execute a python program. There is no virtualenv type system, there is no standard answer to the question “if I’m in the project directory and type ./do_thingy it will just work, right?” The answer is always “Well, it depends…” and what actually winds up happening is that people either roll a whole release just to crank a trivial amount of code up or (quite often) implement an ad hoc way to get the same effect in a lighter-weight way. (erlang.mk shines here, actually.)

Erlang does provide a number of ways to make a system run locally from source of .beam files — and has actually quite reasonable built-in resources for this — but nothing has been built around these tools that also deals with external dependencies, argument passing in a standard way, or any of the other little things you really need if you want to claim a complete solution. Hence all the ad hoc solutions that “work on my machine” but certainly aren’t something you expect your users to use (not with broad success, anyway).

This wouldn’t be such a big problem if it weren’t for the fact that not having any standard way to “just run a program” also means that there really isn’t any standard way to deal with client side code in Erlang. This is a big annoyance for me because much of what I do is client-side code. In Erlang.

In fact, it totally boggles my mind that client-side Erlang isn’t more common, especially considering that AMD is already fielding zillion-core processors for desktops, yet most languages are fundamentally single-threaded. That doesn’t mean you can’t do concurrency and parallelism in other languages, but most problems are not parallel in nature to begin with (parallel problems are easy to write solutions to in any language) while most real-world problems are concurrent. But concurrent systems are hard to write in almost every language. Concurrent problems are the bulk of the interesting problems we’re still not very good at solving with computers. AMD is moving to make the tools available to make much more interesting concurrent processing tools available on the client side (which means Intel will soon start pouring it gajillions worth of blood diamond money into a similar effort), but most languages and environments have no good way to make use of that on the client side. (Do you see why I hear Lady Fortune knocking?)

Browsers? Oh yeah. That’s a great plan. Have you noticed that most sites slowly move toward the “Single Page App” design over time (read as: the web sucks, so now we write full-but-crippled client-programs and deliver them over the web), invest heavily in do-sneaky-things-without-telling-you JavaScript and try to hog every core your system has if you allow it the slightest permission to do so? No. In the age of bitcoin miners embedded in nearly every ad this is not the direction I think we should be envisioning things going.

I want to take better advantage of the cores users have available, and that doesn’t necessarily mean make more efficient use of every cycle as much as it means to make scheduling across processes more efficient to reduce latency throughout the system overall. That’s something users care about quite a lot. This is the problem Erlang has already solved in a way no other runtime out there has. So I want to capitalize on it.

And yet, there is still not standardish way of dealing with code from source, running it locally, declaring or resolving dependencies, or even launching a client-side program at all.

So… how am I approaching it?

I have a project called “zomp” which is a repository system. It is a distributed repository system, so not everything has to be held in one place. Code in the zomp universe is held in little semantic silos called “realms”. Each realm can have whatever packages the owner (sysop) wants it to have. Each realm must have one server node somewhere that is its “prime” — the node in charge of that realm. That node is where system operator tasks for that realm take place, packagers and maintainers submit code for inclusion, where the package index is built, where the canonical copy of everything is stored. Other nodes configured to see that realm connect to the prime node and receive a copy of the current indexes and are tested for availability and published as available resources for querying indexes or downloading packages.

When too many subordinate nodes connect to a prime the prime will redirect a new node to a subordinate, when a subordinate gets “full” of subordinates itself, it picks a subordinate for new redirects itself, etc. so each realm winds up forming a resource tree of mirror nodes that connect back to the realm prime by a single path. A single node might be prime for several realms, or other nodes may act as prime for different realms — and any node can be configured to become a part of any number of realm trees.

That’s the high-level code division.

The zomp constellation is interfaced with via the “zx” program (short for “zomp explorer”, or “zomp exchanger”, or “Zomp eXtreem!”, or homage to the Sinclair ZX-81, or whatever else might lend itself to the letters “zx” that you might want to make up — I actually forget what it originally stood for, but it is remarkably convenient to type so it’s staying that way)

zx is configured to have visibility on zomp realms the same way a zomp node is (in fact, they use the same configuration files and it isn’t weird to temporarily host a zomp node on your desktop the same way you might host a torrent node for a while — the only extra effort is that you do have to open a port, zomp doesn’t (yet) do hole punching magic).

You can tell zx to run a program using the highly counter-intuitive command:

zx run Realm-ProgramName[-Version]

It breaks the program name down into:

  • Realm (optional, defaulting to the main realm of public FOSS packages called “otpr”)
  • Name (necessary — sort of the whole point)
  • Version (which is optional and can also be partial: “1.0.3” vs just “1.0” or “1”, defaulting to the latest in a series or latest overall)

With those components it then contacts any zomp node it knows provides the needed realm, resolves the latest version number of the requested program, downloads and unpacks it, checks and downloads any missing dependencies, builds the program, and launches it. (And if it doesn’t know any active mirrors it asks the prime node and is seeded with known mirror nodes in addition to getting its query answered.)

The packages are kept in a local cache stored at the user level, not the system level (sort of like how browsers keep their JS and page caches) — though if you want to daemonize zomp and run it as a permanent service (if you run a realm prime, for example) then you would want to create an unprivileged system user specifically for the purpose. If you specify a fully-qualified “realm-name-version” for execution and the packages already exist and are built, zx just launches the code directly (which is the majority case, so no delay there — fast startup).

All zomp nodes carry a complete index of their configured realms and can answer queries with very little overhead, but only the prime node has a copy of all the packages for that realm

 

Zomp realms are write-only. There is no facility for removing a package from a realm entirely, only for upgrading the versions of packages whenever necessary. (Removal is, of course, possible, but requires manual intervention by the sysop.)

When a zx client or zomp node asks an upstream node for a package and the upstream node does not have a copy it will query its upstream until the request reaches a node that does have a copy. Once found a “found” notice goes back down to the client telling it how many hops away the package is, and new “hops away” notices are sent as the package is passed downstream toward the original requestor (avoiding timeouts and allowing the user to get some feedback about what is going on). The package is cached at each node along the way, so subsequent requests for that same package will be handled immediately without any more relay downloading.

Because the tree of nodes is expected to be relatively ephemeral and in a constant state of flux, the tendency is for package stores on mirror nodes to be populated by only the latest, most popular packages. This prevents the annoying problem with old realms having gobs of packages that nobody uses but mirror hosts being burdened with maintaining them all anyway.

But why not just keep the latest of everything and ditch old packages?

Ever heard of “version shear”? Yeah. Me too. It sucks. That’s why.

There are no “up to” or “greater than” or “abstract version 3” type dependency declarations in zomp package metadata. As a package maintainer you must explicitly declare the complete version of each dependency in your system. In the case of diamond-shaped dependencies (where two packages in your system depend on slightly different versions of the same package) the burden is on the packagers to declare a version that works for a given release of that package. There are no dependency trees for this reason. If your package depends on X, and X depends on Y and Z then your package must be defined as depending on X, Y and Z — and fully specify the versions involved.

Semver is strictly enforced, by the way. That is, all release numbers are “Major.Minor.Patch”. And that’s it. No more, no less. This is one of the primary criteria for inclusion into a public realm and central to the way both zx and zomp interpret package semantics. If an upstream project has some other numbering scheme the packager will need to create a semver standard of his own. And actually, this turns out to not be very hard in practice. There is one weird side-effect of full, static dependency version declarations and semver: updating dependencies results in incrementing your package’s patch number, so even if you don’t change anything in a program for a long time, a program with many dependencies under heavy development may wind up on version 2.3.257 without much change other than the {deps, PackageIDs}. line in the package meta file.

zx helps make you aware of these situations, so solving them has not been particularly difficult in practice.

Why do things this way?

The “static dependencies forever and ever, amen” decision is a tradeoff between the important feature of fully repeatable builds Erlang releases are famous for (to the point of bug-compatibility between deployment sites — which is critical in production) and the flexibility users and developers have come to expect from source repository systems like pip, pypi, CPAN, etc. Because each realm is write-only there is no danger that a package will be superceded and disappear. The way trickle-down caching works for mirror zomp nodes does not unduly burden the subordinate realm mirrors, and the local caching behavior of zx itself at launch time tends to make all of this mostly delay-free for zx clients and still gives them the option to always run “latest available version” if they want.

And on the note of “latest version”…

Client-side programs are not expected to be run too terribly long at a time. People shut desktop programs down, restart computers, update their kernels, etc. So even if a client program runs a long time (on the order of web, email, IRC, certain games, crypto wallets/miners, torrent nodes, Freenode, Tor, etc) it will still have a chance to restart every few days or weeks to check for a new version (if invoked in a way that omits the version number so that it always queries the latest version).

But what about for long-running server-side type programs? When zx starts a script checks the initial environment and then starts the erlang runtime with zx as its target application, passing it the package ID of the desired program to run and its arguments as arguments. That last sentence was odd. An example is helpful:

zx run foo-bar arg1 arg2 arg3

zx invokes the launching script (a Bash script on Linux, BSD and OSX, a batch file on Windows — so actually the command is zx.bash or zx.cmd)  with the arguments run foo-bar arg1 arg2 arg3. zx receives the instruction “run” and then breaks “foo-bar” into {Realm, Name} = {"foo", "bar"}. Everything after that is passed in as strings which wind up being the input arguments to the program being run: “foo-bar”.

zx registers a process called zx_daemon which remains resident in the runtime and waits for a subscription request or zomp query. Any Erlang program written with the intention of being used with zx can send a message to zx_daemon and ask it to maintain a connection to the program’s parent realm and enroll for update notifications. If the target program itself is the subject of a realm index update then it will get a message letting it know what has changed. The program can respond any way the author wants to such a notification.

In this way it is possible to write a client-side or server-side application that can enroll to become aware of updates to itself without any extra infrastructure and a minimal amount of code. In some programs I’ve used this to cause a pop up notification to appear to desktop users so they know that a new version has become available and they should restart the program (the way Firefox does on Windows). It could also be used to initiate a restart on its own, or whatever else you might come up with.

There are several benefits to developers of using this system as well.

As a developer I can start a new project by doing zx init app [Realm-Name] or zx init lib [Realm-Name] in an existing project root directory and a zomp.meta file will be generated for it, or a new project template directory will be created (populated with a functioning sample skeleton project). I can do zx dailyze and zx will make sure a generally relevant PLT exists or is built (if not up to date) and used to check the typespecs of the project and its dependencies. zx create package [Path] will create a zomp package, sign it, and populate the metadata for it. zomp keygen will generate the kind of keys necessary to interact with a zomp server. zomp submit PackageFilePath will submit a package for review.

And so on.. It is a lot easier to do most things now, and that’s the main point.

(There are commands for reviewing, approving, or rejecting package submissions, adding packagers and maintainers to package projects, adding dependencies to projects, X.Y.Z version incrementing, etc. as well.)

This is about 90% of the way I want it to be, but that means about 90% of the effort remains (pessimistically assuming the 90/10 rule, because life sucks and nobody cares). Most of that is probably going to be finagling some network lunacy, but a lot of the effort is going to be in putting polish to it.

Zomp/zx is based on a similar project I wrote for use within Tsuriai a few years ago that has much sparser features but does basically the same thing: eases packaging and repeatable deployment from source to client systems. I would never release that version publicly because it has a lot of “works for me!” level functionality, but very little polish and requires manually diddling quite a few settings files in error-prone ways (which is fine because it was just us diddling them).

My intention here is to Cadillac this out a bit so that newcomers can slide into the new language and just focus on that language after learning a minimum of tooling commands or environmental details. I think zx init app foo-bar and zx runlocal are a low enough bar for entry.

The Great Blockchain Race

December 12th, 2017

There is a big hustle going on right now over blockchain-based systems, most notably digital cryptocurrencies. It is as if the public just became aware of the word “blockchain”, saw that Bitcoin posted some crazy value gains, and decided “Oh? It went up? That means it is going to be a safe bet that it will go up forever!” and just hopped in with both feet.

Despite blockchain’s inherent scalability problems…

Despite the totally insane energy cost behind every single transaction going forward…

This has, of course, attracted the attention of The Sneakies. The Sneakies are people who realize that running a confidence game on a single person is moderately difficult, but running one on a large population that doesn’t really have the time or interest to dig into the details is quite easy — especially if you have a piece of cake in one hand, and even easier if they are panicked about something at the same time. Fear and hope are a powerful combination when aligned.

Since about 2014 an interesting proliferation of digital currencies (most being cryptocurrencies, but some even being created by banking consortia — har har!) has occurred. Some try to attract attention by spreading FUD about Bitcoin (not that the things they say about Bitcoin are inaccurate, but the same criticisms usually apply to the newly proposed currency as well), some try to attract attention using a “proof-of-work” system analogous to the original Bitcoin algorithm (“Get in now on the ground-floor!”), some try to leverage pre-existing FUD about Trump or the Euro or whatever. Most use a subtle combination and target a specific demographic (Antifa sympathizers, Randite Objectivist libertarians, Neo-Commies, Neo-Nazis, retirees and other “near-deads”, veterans, even Neo-Pagans).

Catching a trend? This is how trends that become confidence scams start to look.

Are cryptocurrencies the future of lightweight value exchange? Yeah, probably something like that. But we already have something more concrete backed by violence: actual currencies that can be electronically divided, transferred and calculated at a much lower cost to energy.

So what will happen? The early miners are punching out now — because while the run has been great and Bitcoin & co will be worth more than $0 even after the market correction, nobody knows when the correction will come. Full disclosure: I’m holding some Bitcoin. Mostly stuff I mined a few years ago. The value is sort of preposterous at the moment. Will I cash in? Maybe — but who knows what sort of pain that might cause me with tax services? It might not even be worth it unless I’m prepared to be shady about things.

But the scammers are starting to cash in, and it won’t be too much longer before one of two things happen:

  1. Scary but predictable: The Bitcoin “whales” cash in and the market collapses, causing a race to the bottom (like a short-call on everyone who has been betting against the Yen, Dollar, Pound and gold)
  2. Crap your pants scary and unpredictable: A quantum breakthrough or algorithmic development makes the entire blockchain transparent and manipulable — POOF!

I’m not saying these sort of efforts are a bad idea, just that they are unrefined and this is unexplored territory.

Also, as a parting thought… Every piece of software used for running crypto wallets, miners, etc. right now is rushed into production with little to no validation or security testing whatsoever. Maybe that isn’t the best way to safeguard something many non-techhies are hoping to be The Next Big Thing. Many of these platforms require Oracle’s Java, for example, and cannot even run on IBM’s JVM or the OpenJDK. Maybe that’s also not a good plan. That’s like having all your eggs in one big basket inside another basket of baskets. Whoops.

18 U.S. Code § 793 – Gathering, transmitting or losing defense information

December 7th, 2017

Quite a few high-profile instances of leaks, breaches, infractions, cracks and “extreme carelessness in the handling of” classified information have been in the news over the last few years, and while folks like to talk a lot of fluff about whether this or that instance was truly vile or truly virtuous, I’ve never actually seen anyone reference the underlying rules regarding defense information.

So here it is: 18 U.S. Code § 793

Cornell Law has the text posted here as well.