Category Archives: Society

Should be obvious…

Zomp/zx: Yet Another Repository System

I’ve been working on a from-source repo system for Erlang on and off for the last few months, contributing time to it pretty much whenever real-life is not interfering. I’m getting close to making a release. Now that my main data bits are worked out, the rest isn’t all that hard. I need to figure out what I want to say in an announcement.

The problem is that I’m really horrible at announcements and this system does things in a pretty different way to other repository systems out there, so I’m not sure what things are going to be important about it to users (worth putting into an announcement) and what things are going to be important to only me because I’m the one who wrote it (and am therefore obsessed with its externally inconsequential internals). What is internally interesting about a project is almost never what is externally interesting about it. Marketing; QED. So I need to sort that out, and writing sometimes helps me sort that kind of thing out.

I’m making this deliberately half-baked, disorganized, over-long post public because Joe Armstrong gave me some food for thought the other day. I had written him my thoughts on a subject posted to a mailing list but sent the message in private. I made my message to him off-list for two reasons: first, I wasn’t comfortable with my way of expressing the idea just yet; and second, I am busy with real-life stuff and side projects, including the repo system, and don’t want to get sucked into online chatter that might amount to nothing more than bikeshedding. (I’m a world-class bikeshedder!) Joe wrote me back asking why I made the reply private, I told him my reasons, and he made me change my mind. He hopes that more people will publish their ideas all the time, good or bad, fully baked or still soggy — because that’s the only way we can ever find any other interesting ideas these days is by searching for them, usually in text, on the net somewhere. It isn’t like we can’t go back and revise, but whether or not we do go back and clean up our literary messes, the availability of core ideas and exposure of thought processes are more important than polish. He’s been on a big drive to make sure that he posts most of his thoughts to public mailing lists or blogs so that his ideas get at least indexed and archived. On reflection I agree with him.

So here I am, trying to publicly organize my thoughts on my repository system.

I should start with the goals of the system.

This system is intended to smooth over a few points of pain experienced when trying to get a new Erlang project off the ground, and in particular avert the path of pain peculiar to Erlang newcomers when they encounter the “how to set up a project” problem. Erlang’s tooling is great but a bit crufty (deeply featured, but confusing to interface with) and not at all what the kool kids expect these days. And anyway I’m really just trying to scratch my own itch here.

At the moment we have two de facto standards for publishing Erlang systems: and Rebar. I like both of these, especially, but they do one thing that annoys me and never seems to quite fit my need: they build Erlang releases.

Erlang releases are great. They cut all the cruft of a release out and pack everything needed to actually run a system into a single blob of digits that you can move, in a single shot, to a new target system — including the Erlang runtime itself. Awesome! Self-contained deployment and it never misses. This has been an Erlang feature since before people even realized that they needed repeatable deployment infrastructure outside of the classic “let’s build a monolithic, static binary executable” approach. (Erlang is perpetually ahead of its time, even by today’s standards. I look at the poor kids stubbing their toes with Docker and language du jour and just shake my head — though part of that is because many shops are using Docker to solve concurrency issues that they haven’t even become cognizant of, thinking that they are experiencing “scaling” problems but missing the point entirely.)

Erlang releases are awesome when the deployment target is an embedded system, but not so awesome if the target is a full-blown operating system, VM, container, or virtual environment fully stocked with gobs of memory and storage and flush with system utilities and resources. Erlang releases sort of kitchen-sink the deployment itself. What if you want to run several different Erlang programs, all delivered as releases, all depending on the same library? You’ve got tons of copies of that library. Which is OK, but still sort of weird, because you also have tons of copies of the runtime (among other things). Each release is self-contained and lean, but in aggregate this is a bit odd.

Erlang releases make sense when you’re deploying to a phone switch or a sensor device in the middle of nowhere and the runtime is basically acting as its own operating system. Erlang releases are, in that context, analogous to putting a Gentoo stage 3 binary image on a system to leapfrog most of the toolchain process. Very cool when you’re in that situation, but a bit tinker-tacky when you’re just trying to run, say, a client program written in Erlang or test a web front-end for something that uses YAWS or Cowboy.

So that’s the siloed-kitchen-sink issue. The other issue is that newcomers are perpetually confused about releases. This makes teaching elementary Erlang hard. In my view we should really focus on escript for beginner code — just let the new guy run something out of a single file the way he is used to doing when learning a new language instead of showing him pages of really slick code, then some interpreter stuff, and then leaping straight from that to a complex and advanced packaging setup necessarily tailored for conducting embedded deployments to slim hardware devices. Seriously. WTF. Escripts give beginners all the power of Erlang necessary for exploring the more interesting bits of code and refactoring needed to learn sequential Erlang with the major advantage of being able to interface with the system the same way programmers from other environments are used to dealing with langauge runtimes like Bash, AWK, Python, Ruby, Perl, etc.

But what about that gap between scripts and full-blown production deployments for embedded hardware?

Erlang has… nothing.

That’s right! There is no agreed-upon way to deploy or even run Erlang code in the same manner a Python coder would expect to execute a python program. There is no virtualenv type system, there is no standard answer to the question “if I’m in the project directory and type ./do_thingy it will just work, right?” The answer is always “Well, it depends…” and what actually winds up happening is that people either roll a whole release just to crank a trivial amount of code up or (quite often) implement an ad hoc way to get the same effect in a lighter-weight way. ( shines here, actually.)

Erlang does provide a number of ways to make a system run locally from source of .beam files — and has actually quite reasonable built-in resources for this — but nothing has been built around these tools that also deals with external dependencies, argument passing in a standard way, or any of the other little things you really need if you want to claim a complete solution. Hence all the ad hoc solutions that “work on my machine” but certainly aren’t something you expect your users to use (not with broad success, anyway).

This wouldn’t be such a big problem if it weren’t for the fact that not having any standard way to “just run a program” also means that there really isn’t any standard way to deal with client side code in Erlang. This is a big annoyance for me because much of what I do is client-side code. In Erlang.

In fact, it totally boggles my mind that client-side Erlang isn’t more common, especially considering that AMD is already fielding zillion-core processors for desktops, yet most languages are fundamentally single-threaded. That doesn’t mean you can’t do concurrency and parallelism in other languages, but most problems are not parallel in nature to begin with (parallel problems are easy to write solutions to in any language) while most real-world problems are concurrent. But concurrent systems are hard to write in almost every language. Concurrent problems are the bulk of the interesting problems we’re still not very good at solving with computers. AMD is moving to make the tools available to make much more interesting concurrent processing tools available on the client side (which means Intel will soon start pouring it gajillions worth of blood diamond money into a similar effort), but most languages and environments have no good way to make use of that on the client side. (Do you see why I hear Lady Fortune knocking?)

Browsers? Oh yeah. That’s a great plan. Have you noticed that most sites slowly move toward the “Single Page App” design over time (read as: the web sucks, so now we write full-but-crippled client-programs and deliver them over the web), invest heavily in do-sneaky-things-without-telling-you JavaScript and try to hog every core your system has if you allow it the slightest permission to do so? No. In the age of bitcoin miners embedded in nearly every ad this is not the direction I think we should be envisioning things going.

I want to take better advantage of the cores users have available, and that doesn’t necessarily mean make more efficient use of every cycle as much as it means to make scheduling across processes more efficient to reduce latency throughout the system overall. That’s something users care about quite a lot. This is the problem Erlang has already solved in a way no other runtime out there has. So I want to capitalize on it.

And yet, there is still not standardish way of dealing with code from source, running it locally, declaring or resolving dependencies, or even launching a client-side program at all.

So… how am I approaching it?

I have a project called “zomp” which is a repository system. It is a distributed repository system, so not everything has to be held in one place. Code in the zomp universe is held in little semantic silos called “realms”. Each realm can have whatever packages the owner (sysop) wants it to have. Each realm must have one server node somewhere that is its “prime” — the node in charge of that realm. That node is where system operator tasks for that realm take place, packagers and maintainers submit code for inclusion, where the package index is built, where the canonical copy of everything is stored. Other nodes configured to see that realm connect to the prime node and receive a copy of the current indexes and are tested for availability and published as available resources for querying indexes or downloading packages.

When too many subordinate nodes connect to a prime the prime will redirect a new node to a subordinate, when a subordinate gets “full” of subordinates itself, it picks a subordinate for new redirects itself, etc. so each realm winds up forming a resource tree of mirror nodes that connect back to the realm prime by a single path. A single node might be prime for several realms, or other nodes may act as prime for different realms — and any node can be configured to become a part of any number of realm trees.

That’s the high-level code division.

The zomp constellation is interfaced with via the “zx” program (short for “zomp explorer”, or “zomp exchanger”, or “Zomp eXtreem!”, or homage to the Sinclair ZX-81, or whatever else might lend itself to the letters “zx” that you might want to make up — I actually forget what it originally stood for, but it is remarkably convenient to type so it’s staying that way)

zx is configured to have visibility on zomp realms the same way a zomp node is (in fact, they use the same configuration files and it isn’t weird to temporarily host a zomp node on your desktop the same way you might host a torrent node for a while — the only extra effort is that you do have to open a port, zomp doesn’t (yet) do hole punching magic).

You can tell zx to run a program using the highly counter-intuitive command:

zx run Realm-ProgramName[-Version]

It breaks the program name down into:

  • Realm (optional, defaulting to the main realm of public FOSS packages called “otpr”)
  • Name (necessary — sort of the whole point)
  • Version (which is optional and can also be partial: “1.0.3” vs just “1.0” or “1”, defaulting to the latest in a series or latest overall)

With those components it then contacts any zomp node it knows provides the needed realm, resolves the latest version number of the requested program, downloads and unpacks it, checks and downloads any missing dependencies, builds the program, and launches it. (And if it doesn’t know any active mirrors it asks the prime node and is seeded with known mirror nodes in addition to getting its query answered.)

The packages are kept in a local cache stored at the user level, not the system level (sort of like how browsers keep their JS and page caches) — though if you want to daemonize zomp and run it as a permanent service (if you run a realm prime, for example) then you would want to create an unprivileged system user specifically for the purpose. If you specify a fully-qualified “realm-name-version” for execution and the packages already exist and are built, zx just launches the code directly (which is the majority case, so no delay there — fast startup).

All zomp nodes carry a complete index of their configured realms and can answer queries with very little overhead, but only the prime node has a copy of all the packages for that realm


Zomp realms are write-only. There is no facility for removing a package from a realm entirely, only for upgrading the versions of packages whenever necessary. (Removal is, of course, possible, but requires manual intervention by the sysop.)

When a zx client or zomp node asks an upstream node for a package and the upstream node does not have a copy it will query its upstream until the request reaches a node that does have a copy. Once found a “found” notice goes back down to the client telling it how many hops away the package is, and new “hops away” notices are sent as the package is passed downstream toward the original requestor (avoiding timeouts and allowing the user to get some feedback about what is going on). The package is cached at each node along the way, so subsequent requests for that same package will be handled immediately without any more relay downloading.

Because the tree of nodes is expected to be relatively ephemeral and in a constant state of flux, the tendency is for package stores on mirror nodes to be populated by only the latest, most popular packages. This prevents the annoying problem with old realms having gobs of packages that nobody uses but mirror hosts being burdened with maintaining them all anyway.

But why not just keep the latest of everything and ditch old packages?

Ever heard of “version shear”? Yeah. Me too. It sucks. That’s why.

There are no “up to” or “greater than” or “abstract version 3” type dependency declarations in zomp package metadata. As a package maintainer you must explicitly declare the complete version of each dependency in your system. In the case of diamond-shaped dependencies (where two packages in your system depend on slightly different versions of the same package) the burden is on the packagers to declare a version that works for a given release of that package. There are no dependency trees for this reason. If your package depends on X, and X depends on Y and Z then your package must be defined as depending on X, Y and Z — and fully specify the versions involved.

Semver is strictly enforced, by the way. That is, all release numbers are “Major.Minor.Patch”. And that’s it. No more, no less. This is one of the primary criteria for inclusion into a public realm and central to the way both zx and zomp interpret package semantics. If an upstream project has some other numbering scheme the packager will need to create a semver standard of his own. And actually, this turns out to not be very hard in practice. There is one weird side-effect of full, static dependency version declarations and semver: updating dependencies results in incrementing your package’s patch number, so even if you don’t change anything in a program for a long time, a program with many dependencies under heavy development may wind up on version 2.3.257 without much change other than the {deps, PackageIDs}. line in the package meta file.

zx helps make you aware of these situations, so solving them has not been particularly difficult in practice.

Why do things this way?

The “static dependencies forever and ever, amen” decision is a tradeoff between the important feature of fully repeatable builds Erlang releases are famous for (to the point of bug-compatibility between deployment sites — which is critical in production) and the flexibility users and developers have come to expect from source repository systems like pip, pypi, CPAN, etc. Because each realm is write-only there is no danger that a package will be superceded and disappear. The way trickle-down caching works for mirror zomp nodes does not unduly burden the subordinate realm mirrors, and the local caching behavior of zx itself at launch time tends to make all of this mostly delay-free for zx clients and still gives them the option to always run “latest available version” if they want.

And on the note of “latest version”…

Client-side programs are not expected to be run too terribly long at a time. People shut desktop programs down, restart computers, update their kernels, etc. So even if a client program runs a long time (on the order of web, email, IRC, certain games, crypto wallets/miners, torrent nodes, Freenode, Tor, etc) it will still have a chance to restart every few days or weeks to check for a new version (if invoked in a way that omits the version number so that it always queries the latest version).

But what about for long-running server-side type programs? When zx starts a script checks the initial environment and then starts the erlang runtime with zx as its target application, passing it the package ID of the desired program to run and its arguments as arguments. That last sentence was odd. An example is helpful:

zx run foo-bar arg1 arg2 arg3

zx invokes the launching script (a Bash script on Linux, BSD and OSX, a batch file on Windows — so actually the command is zx.bash or zx.cmd)  with the arguments run foo-bar arg1 arg2 arg3. zx receives the instruction “run” and then breaks “foo-bar” into {Realm, Name} = {"foo", "bar"}. Everything after that is passed in as strings which wind up being the input arguments to the program being run: “foo-bar”.

zx registers a process called zx_daemon which remains resident in the runtime and waits for a subscription request or zomp query. Any Erlang program written with the intention of being used with zx can send a message to zx_daemon and ask it to maintain a connection to the program’s parent realm and enroll for update notifications. If the target program itself is the subject of a realm index update then it will get a message letting it know what has changed. The program can respond any way the author wants to such a notification.

In this way it is possible to write a client-side or server-side application that can enroll to become aware of updates to itself without any extra infrastructure and a minimal amount of code. In some programs I’ve used this to cause a pop up notification to appear to desktop users so they know that a new version has become available and they should restart the program (the way Firefox does on Windows). It could also be used to initiate a restart on its own, or whatever else you might come up with.

There are several benefits to developers of using this system as well.

As a developer I can start a new project by doing zx init app [Realm-Name] or zx init lib [Realm-Name] in an existing project root directory and a zomp.meta file will be generated for it, or a new project template directory will be created (populated with a functioning sample skeleton project). I can do zx dailyze and zx will make sure a generally relevant PLT exists or is built (if not up to date) and used to check the typespecs of the project and its dependencies. zx create package [Path] will create a zomp package, sign it, and populate the metadata for it. zomp keygen will generate the kind of keys necessary to interact with a zomp server. zomp submit PackageFilePath will submit a package for review.

And so on.. It is a lot easier to do most things now, and that’s the main point.

(There are commands for reviewing, approving, or rejecting package submissions, adding packagers and maintainers to package projects, adding dependencies to projects, X.Y.Z version incrementing, etc. as well.)

This is about 90% of the way I want it to be, but that means about 90% of the effort remains (pessimistically assuming the 90/10 rule, because life sucks and nobody cares). Most of that is probably going to be finagling some network lunacy, but a lot of the effort is going to be in putting polish to it.

Zomp/zx is based on a similar project I wrote for use within Tsuriai a few years ago that has much sparser features but does basically the same thing: eases packaging and repeatable deployment from source to client systems. I would never release that version publicly because it has a lot of “works for me!” level functionality, but very little polish and requires manually diddling quite a few settings files in error-prone ways (which is fine because it was just us diddling them).

My intention here is to Cadillac this out a bit so that newcomers can slide into the new language and just focus on that language after learning a minimum of tooling commands or environmental details. I think zx init app foo-bar and zx runlocal are a low enough bar for entry.

The Great Blockchain Race

There is a big hustle going on right now over blockchain-based systems, most notably digital cryptocurrencies. It is as if the public just became aware of the word “blockchain”, saw that Bitcoin posted some crazy value gains, and decided “Oh? It went up? That means it is going to be a safe bet that it will go up forever!” and just hopped in with both feet.

Despite blockchain’s inherent scalability problems…

Despite the totally insane energy cost behind every single transaction going forward…

This has, of course, attracted the attention of The Sneakies. The Sneakies are people who realize that running a confidence game on a single person is moderately difficult, but running one on a large population that doesn’t really have the time or interest to dig into the details is quite easy — especially if you have a piece of cake in one hand, and even easier if they are panicked about something at the same time. Fear and hope are a powerful combination when aligned.

Since about 2014 an interesting proliferation of digital currencies (most being cryptocurrencies, but some even being created by banking consortia — har har!) has occurred. Some try to attract attention by spreading FUD about Bitcoin (not that the things they say about Bitcoin are inaccurate, but the same criticisms usually apply to the newly proposed currency as well), some try to attract attention using a “proof-of-work” system analogous to the original Bitcoin algorithm (“Get in now on the ground-floor!”), some try to leverage pre-existing FUD about Trump or the Euro or whatever. Most use a subtle combination and target a specific demographic (Antifa sympathizers, Randite Objectivist libertarians, Neo-Commies, Neo-Nazis, retirees and other “near-deads”, veterans, even Neo-Pagans).

Catching a trend? This is how trends that become confidence scams start to look.

Are cryptocurrencies the future of lightweight value exchange? Yeah, probably something like that. But we already have something more concrete backed by violence: actual currencies that can be electronically divided, transferred and calculated at a much lower cost to energy.

So what will happen? The early miners are punching out now — because while the run has been great and Bitcoin & co will be worth more than $0 even after the market correction, nobody knows when the correction will come. Full disclosure: I’m holding some Bitcoin. Mostly stuff I mined a few years ago. The value is sort of preposterous at the moment. Will I cash in? Maybe — but who knows what sort of pain that might cause me with tax services? It might not even be worth it unless I’m prepared to be shady about things.

But the scammers are starting to cash in, and it won’t be too much longer before one of two things happen:

  1. Scary but predictable: The Bitcoin “whales” cash in and the market collapses, causing a race to the bottom (like a short-call on everyone who has been betting against the Yen, Dollar, Pound and gold)
  2. Crap your pants scary and unpredictable: A quantum breakthrough or algorithmic development makes the entire blockchain transparent and manipulable — POOF!

I’m not saying these sort of efforts are a bad idea, just that they are unrefined and this is unexplored territory.

Also, as a parting thought… Every piece of software used for running crypto wallets, miners, etc. right now is rushed into production with little to no validation or security testing whatsoever. Maybe that isn’t the best way to safeguard something many non-techhies are hoping to be The Next Big Thing. Many of these platforms require Oracle’s Java, for example, and cannot even run on IBM’s JVM or the OpenJDK. Maybe that’s also not a good plan. That’s like having all your eggs in one big basket inside another basket of baskets. Whoops.

18 U.S. Code § 793 – Gathering, transmitting or losing defense information

Quite a few high-profile instances of leaks, breaches, infractions, cracks and “extreme carelessness in the handling of” classified information have been in the news over the last few years, and while folks like to talk a lot of fluff about whether this or that instance was truly vile or truly virtuous, I’ve never actually seen anyone reference the underlying rules regarding defense information.

So here it is: 18 U.S. Code § 793

Cornell Law has the text posted here as well.

Las Vegas shooting prediction: Most casualties were not due to gunshot wounds

Looking over the data for large stampedes and crowd crush events at concerts and sporting events, and comparing this to what I know personally from a career spent mostly handling various weapons in a tactical environment, I expect that we will discover fairly soon that the vast majority of casualties during the Las Vegas shooting — both injuries and fatalities — were actually due to stampede, and not anything to do with gunshot wounds at all.

Of course, in the confusion this issue has become politicized to an absolutely ridiculous degree by various anti-gun factions, and much of the US and European media is loathe to report anything other than anti-gun statistics for the moment, so we are seeing language tailored to evoke images of hundreds of people with actual gunshot wounds and zero people with stampede injuries.

For example: “Shooter in Las Vegas [blah blah blah] over 500 wounded.” This makes the reader or listener immediately envision 500 people actually wounded, as in due to violent trauma — and deliberate violent trauma at that. Which in this case would be exclusively due to gunshot wounds. But we have never seen a breakdown of causes of bodily harm by type, and this data will take a while to assemble.

By the time we do see these stats most people will not really be interested because immigration in Europe or stubborn people in Madrid/Barcelona or NFL SJW activity or whatever else will steal the spotlight and public attention before then. In other words, people will be distracted with another issue-of-the-day by then and forget that the new factoids they see relate to a previous event they felt very strongly about at the time it occurred.

Watch for this one.

Asian Governments Making Social Moves Together

I expect Asian governments to manifest a low-key but characteristically firm and absolute (and often official) position against Islam. Actually, I don’t expect it, I’m watching it happen and just now recognizing a fairly uniform trend. Something is going on in Asia with regard to this, and I don’t know quite what it is, but there is no doubt that doors are closing all across Asia for Muslims in general.

I think the timing is not a coincidence — the nature of Islamic threats are changing, becoming more diffuse, and taking on a different character just as a new generation of indoctrination is beginning across the West and Asia.

  • Myanmar has found something much more compelling than mere domestic political expediency to engage in its current operations (ISIS returners, as are turning up in Malaysia, Indonesia and the Philippines, is one possibility).
  • China has begun confiscating the Koran and categorized it as a book containing extremist political sentiment.
  • Thailand is readying a firm move against the southern Muslim rebels — and at the same time ISIS returners are very effectively influencing the young generation throughout the old Pattani region.
  • Saudis and other donors are standing up madrasas throughout Malaysia and Indonesia, and the Malaysian government is both unable to stop the trend while at the same time higher-ups in Putrajaya are strangely blind to the problem while also complaining about it.
  • The Philippines is obviously on a “you’re with us or against us” path politically and socially. And a certain of portion of the younger Muslim generation today is much more willing to take that as a challenge instead of an offer to pledge fealty (or at least negotiate terms).
  • Japanese are, at least anecdotally, becoming increasingly uneasy with the idea of accepting any Muslims, even as guest workers. The striking thing there is that ten years ago (well after 9/11) the topic of religion would never have been mentioned discussing this issue socially, but now it is brought up. This change over the last year or two coincides with the first mosque in Kyoto trying to promote itself via online ads and Japanese demonstrating an instant and strong aversion to the very concept of proselytization. They are now in “wait and see” mode socially — to watch and see how things turn out in Europe.
  • South Koreans seem to be on the same page as the Japanese — the attitude toward Islam having soured considerably over the last five years or so. Once again, this is anecdotal, but the subject has come up more than once, and many South Koreans keep up with news of attacks in France, Sweden and the UK.
  • Indonesia is seeing the rise of extra-judicial Islamic enforcement gangs.
  • Malaysia is seeing a similar rise in extra-judicial Islamic enforcement gangs, but the effect is somewhat muted by considerable repression by the special police and more active engagement with the group leaders.
  • Returners, returners, returners. ISIS veterans are flooding into various part of Asia, fresh off a tour in Syria, North Africa, Iraq or Afghanistan with ISIS and keeping in touch with one another. Of course, nobody feels comfortable with that. Unlike in Europe, though, well-known jihadis are not left to their own devices and most go missing somewhere in transit — but it is clear and evident that many are still returning and building new lines of communication and influence locally.

Any one of these issues, from official government actions to simple social reactions, would be grounds for certain groups to rally large responses — Islamic groups as well as Western-based political groups with strong anti-Asian nationalist agendas (something I’ve always found very odd). But the only thing making the news is Myanmar right now, and that’s a pretty hopeless fight to try to pick in terms of political pressure. Myanmar is about as pliable as North Korea as long as China is on their side, and China is indeed on their side with regard to this detail.

I do not see a future where Asian governments will feel compelled to do anything other than increase their resistance to an increased domestic Muslim presence. I fully expect that religious questions will be incorporated on visa applications to places like China eventually (not that repression of religion is anything new there).

I have no idea how any of this is going to turn out, but I find this trend notable and the timing troubling. I don’t know exactly what is triggering this much activity just now (why not a decade ago?), but something is clearly going on. It could be the outcome of some government assessments, or simply a change in the domestic social outlook, or both — but something is going on with this. And, of course, it is impossible to say “they are wrong”. It is just what they are doing and I’m just pointing it out.


I just realized that it is futile to drop hints to my wife about, say, a snack before dinner. The master play is to talk about snacks with my kids before dinner, and they will always find a way to deliver the goods.


Evidence of real power: badass snacks.

Its a Small, Small World

A question on the Worldbuilding SE site about military contractors / mercenaries / “evil henchmen” / whatever caught my attention and I responded. A discussion started between myself and another guy who has contracted before in the same places, but on the tech side of the business instead of PSD. I’m preserving the conversation here because it illustrates the point I was trying to make in my answer, but doesn’t really fit within the answer and will certainly be deleted by the mods:

@zxq9 I’m impressed by your answer. I worked as a contractor as well, and ran a bar at night in one country (Middle East). Lots of the guys I worked with were former Ranger / SF. I did tech instead of PSD, but some jobs we were responsible for our own security, travel, etc. All the guys I saw ended up going back to the States (or the Philippines…) after contracts; nobody really hung out in theater after jobs. Are those little communities you’ve referred to really that common? Because I’ve yet to see people hang around after the fact.
hathead 15 mins ago

@hathead If you were ever in Baghdad you might remember that along the main E-W road between the traffic circle and Triple Canopy’s base Olympia was a mish-mashed neighborhood with a few shops and restaurants, and a couple of smaller contract company offices working out of houses. (Near the “Hot Tomato” restaurant that was always weird.) That neighborhood is exactly the sort of spot I’m talking about. Kabul has something similar. I think most of that may have been invisible to you if you didn’t work PSD or base def (and therefore have your own wheels) — but its there. Similar in Kampala.
zxq9 10 mins ago

@zxq9 – I remember the neighborhood but didn’t spend a lot of time out there. People bought alcohol there. I worked on Bes-maya, I guess you know where that is. My glory days were in Kabul, though. There we all lived on the economy and would pop into the embassy / base to do work, or speed out to Bagram. Kabul was always weird- you can get whatever you needed and half the guys at the bar were wanna-be journalists. Tourists even popped in sometimes. This neighborhood you’re talking about in Baghdad – it seemed like it was within the confines of base (checkpoint controlled). Was it not?
hathead 7 mins ago

@hathead I also forgot to mention… There are contracts in the southern Philippines, Indo, Malay, Cambodia and Thailand , too — and there are two SF association houses (the front half of which are really bars, like Garfield’s in AC and Tilac II in Pattaya). The neighborhoods near there are full of older SF guys who married locally and never left, and sometimes its easy to source logistics or meet the right folks to get a crew together locally. Pace of life is a bit slower there, but the shape of things is fairly similar — but not everyone wants to have anything to do with contracting.
zxq9 6 mins ago

@hathead The neighborhood wasn’t, but right next to it was the former finance minister’s house and apartments for his wives — which was turned into a business development center. The edge of it was right on the NW edge of the circle and had walls + a guard force from TC (which I was in charge of for 6 months once). That place gave the impression the neighborhood was controlled, but it was just that one facility. The volume of alcohol trade there always surprised me! A lot better than the “parts cleaner” thing the Poles had going on down in al Kut.
zxq9 3 mins ago   edit

@zxq9 – half the ex-Army guys I worked with had houses in the Philippines. You probably know some of the same guys. It’s not easy getting work as a tech with a PMC but I managed to find it with a big name (closed now, but you’d know them) and it was a lot of fun. Hanging out in Turkey now. It’s not hot enough yet that a worthwhile expat community has established. Yet.
hathead 2 mins ago

@zxq9 – The Europeans had fine alcohol, it was just a matter of finding it. They had a bar on Liberty that was wild. Panties stapled to the ceiling. I only saw it once; kept out of that sort of thing mostly. Did brew my own beer, though.
hathead 1 min ago

@hathead Likely do know some of the same folks. They’ll put this thread in chat or delete it pretty soon. My url is the same as my alias here: . Drop me a line. Turkey is an interesting place right now, too!
zxq9 46 secs ago

@zxq9 – I’ll drop you a line, and delete my previous chats if I’m still able.

The Yuan: Stealing from Piers to pay 保罗

So, indeed the Yuan was made into a reserve currency and the link-by-failure is already being established. It is interesting, though, that the bulk of the value transfer involved is coming from the Euro, not the Dollar or Yen. Linked by the IMF formally or not, though, if either the Euro or the Yuan fail over the mid-term the other will as well. The Chinese and Eurozone economies are intimately linked already, but were linked more by success than failure until now. That the failure of either is a very real possibility is too terrifying for the financial press to discuss, I think, and it is a political landmine public figures are trying very hard to avoid mentioning. It even seems that the made-up nature of Chinese government economic estimates isn’t even in the news much these days. You would think that little detail might enter into the discussion about adding a new currency to the IMF’s reserve currency group.

There is no longer a strong relationship among basic aspects of value assignment, legal ownership, practical control (that is, “real ownership”), vested business interest in terms of the performance value of concerns, and available goods and services in either the Eurozone or Chinese markets. While there are no recipes for economic success, there are several recipes for disaster (ask an economist about this — their responses tend to be as enlightening and humorous as they are depressing on reflection). A lack of correlation between various forms of utility values and assigned values is one of the disaster recipes. There is no easy way to fix this other than a kinetic re-establishment of property rights, and that means there is nothing left to do in the current situation than hope that when they do fail, they fail cleanly. But historically there is no such thing as a clean failure (in theory, of course, all sorts of lovely solutions exist).

Dropping an anvil on the overloaded camel’s back in the Eurozone or China would be rather easy at the moment, as both economies are in precarious situations. In fact, inducing a major market collapse would be so easy right now that failure is almost certain to come as the result of a deliberate action from an external player than by mere circumstance. The more players who realize this is true the more likely such an action becomes: why let a failure happen to you when you can be the one making it happen if the event is inevitable?

In describing the European and Chinese economic situations a financial analyst friend of mine used the phrase “poised to fail” (along with a lot of depressed-looking facepalming). When someone says that to a geopolitical analyst, though, ears perk up. There is always opportunity to be found in crisis, and sometimes when crisis is inevitable the best play is to be the cause of it yourself, because then you are the only one truly prepared. Consider the economic fallout of Russia’s invasion of Georgia in August 2008. A similar performance is absolutely not out of the question, nor is having some “terrorists” conveniently demonstrate the peaceful nature of some religion all over a ship in the Strait of Malacca at a perfectly horrible moment.

2007-2010 SPY chart

Rhetoric forces us to pretend that the August invasion of Georgia did not trigger a reassessment of risk in Eastern European carry trade loans, and instead believe that the already liquidated American subprime loans acted as a magical “contagion” that unfairly crushed the Eurozone. As if the European economies were not profoundly overleveraged and primed to implode.

There really isn’t anything to do about what is going on with the Yuan, really. This course was set about 20 years ago (yes, all the way back in 1995 — after the Cold War, after the first post-Tienanmen Square Five Year Plan was in action; as China started on its “Money is Good” -> “Expansion Above All” -> “Don’t Stop the Train” -> “WTO Rules? Screw the rules, I have money!” chain of policies). The general trend will continue, as none of the players seems to have any inkling of how to change the rules of the game — and the trend is of the end of a decades-long political and financial cycle. The way these stories end is never happy.

1984-2015 Money Base chart

Anyone who thinks that events since 2008 have been business as usual and that geopolitics plays no part in this because “its just a market hiccup” is deluded.


Every end is a new beginning, and that’s what is really worth focusing on. That may sound like small comfort (and it is), but if you already know things are going to get worse before they get better, then at least you won’t find yourself sleeping in a bed of broken dreams. It is too soon to tell which way this Jenga tower is going to topple, but we are nearing the end of this round of the game.

China: Yuan Will Be a Reserve Currency, Come What May

EDIT: Indeed, it has been made a reserve currency, or at least it looks like announcements have already been made to pave the way.

The IMF is considering adding the Yuan to the group of reserve currencies. That would put it alongside the U.S. Dollar, the Japanese Yen, the English Pound, and the Euro in terms of “officially perceived” stash-your-value-here viability. As far as actual criteria for inclusion go, the Canadian Dollar, Australian Dollar, and very likely the Russian Ruble are probably actually closer to being genuine reserve currency material than the Chinese Yuan.

But… politics.

China is much closer to a total financial collapse and internal civil disruption* than recovery and stability in its current form. Long-term, of course, China will still be right where it is and the people there will still be Chinese (but there will eventually be far fewer of them, at least for a few generations). A Chinese collapse right now would be a major disaster for everyone. The commodity markets are depressed more than they have been for several decades (in relative terms, actually, I’m not sure that we actually have a post-WWII precedent for what is happening), energy is cheap, credit is massively overleveraged, and yet people aren’t buying enough stuff to keep the wheels spinning.

What does that have to do with the Yuan becoming a reserve currency? It does three things:

  • Gives China access to an external aggregate value device to prop up the yuan if necessary (links their economy to everyone else’s by failure, similar to the way subsidies can do this within a national economy). This effect is actually more a hoped-for psychological effect on the market than a tangible superpower China is being granted by the rays of a yellow sun.
  • Makes the Yuan a necessary holding for anyone trying to carry a balanced basket of reserve currencies (temporarily spikes demand for the Yuan).
  • Promotes an impression of stability in the Yuan (well-founded or not).

Why would the West agree to this? (And I say “the West” because, let’s face it, Washington and London are pretty much the ones who will be deciding.) Because if China were to fail right now it would be a severe annoyance for the U.S. and a complete disaster for Europe and Russia. Nobody really knows what the fallout of that would be, but it wouldn’t be pretty.

The Yuan will be made a reserve currency, whether it makes sense or not, and whether it actually fits in the reserve currency club by the standards and rules the IMF itself has laid out. These are scary times and nobody has any good levers to pull to “fix the economy” so national governments and central banks are pulling at straws because there is simply nothing left to try. All the control rods have been yanked out and tossed already, or shoved in and locked tightly; all the red buttons have been mashed; all the hyperbolic rhetorical devices have been so over-used at this point that the only thing that might actually influence market participants is a frank exposition about the truth rather than more “we’ll do whatever it takes!” and other gung-ho, “it’ll work this time” and “this is the lastest of the last rounds of QE, and this time it will really be the most effectivest of effective measures… I promise!” blather.

[* China is due for two painful corrections which will likely occur together, as they are linked. The first is a political correction; China’s geography does not lend itself to a central command economy. The second is a property-claims correction; when basic goods cannot be had at any price it means the entire system is so out of whack due to government interventions that only a hard reset can fix things. This will likely take the form of a civil war, but who knows. It could be gradual decline toward state failure followed by a logical and non-violent nation-wide roundtable discussion, or even a bloodless revolution coupled with a voluntary capitulation of material holdings by the power elites. But seriously, this has never happened in history and there is no reason to expect China’s inevitable transitions to occur independently of one another, or for either to be non-violent.]

2015-2016 Energy Price Drop Will Disentangle Russia, Not Crush It

Energy prices are dropping. Commodity prices are dropping, actually, in general. Production is slowing. People are finally realizing that China is as flimsy as government construction. The Euro is indeed linking Eurozone economies by failure instead of success, and doing so in a way that limits their market rights. The EU is inherently unstable. Cold War II is finally no longer a secret (though many folks are still oblivious to it).

Long story short: a lot of stuff is going on!

This is one of those “interesting” periods in history — the kind that Chinese proverbs use as a curse. Or, rather, we are re-entering a normal period in history, one where there are more than two poles to the world, and the Cold War alignment stresses are not purely polarized — which means more interesting plays for middle empires (like France), and a very strong possibility that empires that are currently viewed as either permanently vanquished (Japan and Thailand) or part of the new world standard (China) are likely to either find a way to rise again, or endure complete collapse prior to changing form entirely upon resurrection.

Japan might realize the space play it could make by diverting public largess toward space instead of beautifully designing dead-end mountain roads. China will very likely endure a civil war, but it could just as easily be won by the standing government which changes form after it wins as it could be reformed as a republic under a totally different political concept. France my find a way to leverage its African empire to provide an energy alternative to Germany and thereby insulate it from Moscow’s control at the same time it forces it into a subordinate relationship (winning a Napoleonic victory without fighting a war — the way the Germans thought they were “re”winning WWI without fighting a war by imposing the Euro as the new European currency under their former central bank, renamed as the ECB). The Turks may find a way to leverage their water control position over Iraq and work to put ISIS in control of Baghdad as part of a bid to force them to normalize by giving them something that can be taken away. Tehran and Washington are very likely to become close allies. etc.

The world is changing.

Many folks feel guilty pleasure at watching Russian financial numbers decline and the ruble fall as energy prices sink worldwide. Folks think “ah, this is finally it, once Russia’s economy suffers enough, Moscow will have to agree to work with Washington and stop bullying the East so much”.

Well, that last part isn’t going to happen. Not because of energy prices, anyway. There are two reasons for this: Russia doesn’t need money the same way other countries do, and in some regions alternatives to Russian energy are impossible to obtain at any price.

The first point is that Russia is a raw-materials exporter, and also maintains a considerable high-tech domestic manufacturing capability. The reason we don’t see more Russian products in the world, though, is because Russia lack much heavy shipping infrastructure. In particular, its ability to push products to ocean ports is severely limited, so it will never make sense to produce finished goods in the interior of Russia, ship them by rail or truck overland to deep water ports (across hostile political lines, no less — if you think pre-Civil War inter-state tariffs were insane within the United States, imagine what they look like in the middle of a Cold War-style mutual embargo and tariff festival), and from there to the world. China is a much easier shortcut. On the other hand, it will always make sense to ship raw materials from the interior of Ukraine or Russia (which are effectively controlled by the same political decision-makers — even more obviously now than ten years ago), because raw materials can only be had at their sources.

A big part of Russia’s power comes not from being able to throw money around, but by being able to make client states become dependent on material subsidies from Russia. If the Russian’s are subsidizing gas at a certain price in Germany, then the existing infrastructure will be built with that in mind. That lowers the difficulty of obtaining and routing that energy source. That means it lowers the cost of extending that infrastructure and thus deepening systemic dependency on that source over time. That means Russia winds up with a lever of control. As long as nothing bad happens Russia will keep the gas flowing. Once things go their way or favors are refused gas lines might “suffer breakdowns” and prices might arbitrarily increase. It doesn’t matter if the global market price for natural gas is X if that gas is physically impossible to obtain in any significant quantity when you are talking about powering an entire national economy’s energy needs. The local spot price of the gas can be whatever Russia makes up — and if Russia wants to cause pain it can simple experience a series of conveniently timed technical difficulties.

The Russians of today can, as they have for the past several waves of their history, substitute labor for capital when necessary. The methods by which this is accomplished change a bit every generation — conforming to the expectations of the peasantry (and for all practical purposes Russia is still a country of peasants and royalty). A drop in the ruble is annoying, it prevents Moscow from keeping the charade of open engagement with Western economies alive, but does not fundamentally change the power relationship between the West and East, and certainly does not change the geopolitical calculus over the long-term.