Bad Art: Natsumi’s Facial Lines

I wanted to take a shot at drawing facial lines, and since my girlfriend’s face had nice lines, it was a perfect subject. I never got good at this style, but I really like the effect it has. Not nearly as cartoony or アニメ as the way I usually do stuff.

I don’t have any touched up versions of these, these are the only images I have. (I wonder if there are sketch books just sitting in my old kit bags somewhere…?)

Natumi's face, direct view.

Natumi’s face, direct view.

Comparing both of these now, I think I like the second one more — but that’s probably just because of the expression.

Natsumi's face, slightly oblique.

Natsumi’s face, slightly oblique.

Bad Art: The Tia — Less Hair

I took a picture of a friend of mine after she shaved her head close, and decided to sketch it to see if I could. The sketch is a bit of a disaster, but the image is still funny to me. This is my only photo of it.

The Tia, after a haircut and demonstrating her typical concern for the impression she makes on others.

The Tia, after a haircut and demonstrating her typical concern for the impression she makes on others.

Bad Art: The Tia

Special circumstances tend to attract special company, at least that’s the way my weird life tends to work out. I met a phenomenal person in Iraq who had a striking profile. I decided to sketch her a few times. Here is the result of one attempt.

The images progress from raw photos of the sketch to a more processed look.

Raw photo of The Tia sketch.

Raw photo of The Tia sketch.

After a little outline removal and cutout work…

Sketch image after from outline removal.

Sketch image after from outline removal.

After a little more gimp work to selectively blend and smudge the paper-lines away (I didn’t have anything other than Iraqi college-ruled paper available)…

The sketch after some smudging and shading work.

The sketch after some smudging and shading work.

The final product still shows the effect of the lines on the paper near the bottom right of the image — but I either didn’t notice or didn’t care. Meh. I enjoyed this anyway.

Here are some screenshots I apparently took while doing the processing work…

Screenshot Screenshot-1 Screenshot-2 Screenshot-3

That was actually pretty enjoyable, almost monk-like work.

Bad Art: Welcome Back From Leave

I made a few sketches a few years ago while bored out of my mind in Iraq. This is one of them. First is the raw image version, the second is after I cleaned it up with gimp. I seem to have either never finished the cleanup, or just lost the finished version. But whatever.

Convoy receiving a welcome-back present.

Raw sketch image

Slightly processed version.

Slightly processed version.

The are signed using my nickname because I was still connected with the military at that time and there was a lot of fuss over whether or not exposing identities was a big bad thing or not. I chose to avoid the issue by never producing anything under my legal name for quite a while, and mostly put stuff on other sites (like this deviantart account to prevent finger pointing).

If I remember correctly, I was drawing this off of an AAR video of an incident that happened just as I started my second contract out there — bad luck. The reason I chose to draw this, though, was I wanted to try to draw an explosion in pencil, and had seen some really awesome explosion sketches on some sketch art sites. I think the explosion turned out OK, but the rest of the sketch isn’t particularly interesting. Meh. I’m pretty bad at drawing fences and trees (at least as background).

2015-2016 Energy Price Drop Will Disentangle Russia, Not Crush It

Energy prices are dropping. Commodity prices are dropping, actually, in general. Production is slowing. People are finally realizing that China is as flimsy as government construction. The Euro is indeed linking Eurozone economies by failure instead of success, and doing so in a way that limits their market rights. The EU is inherently unstable. Cold War II is finally no longer a secret (though many folks are still oblivious to it).

Long story short: a lot of stuff is going on!

This is one of those “interesting” periods in history — the kind that Chinese proverbs use as a curse. Or, rather, we are re-entering a normal period in history, one where there are more than two poles to the world, and the Cold War alignment stresses are not purely polarized — which means more interesting plays for middle empires (like France), and a very strong possibility that empires that are currently viewed as either permanently vanquished (Japan and Thailand) or part of the new world standard (China) are likely to either find a way to rise again, or endure complete collapse prior to changing form entirely upon resurrection.

Japan might realize the space play it could make by diverting public largess toward space instead of beautifully designing dead-end mountain roads. China will very likely endure a civil war, but it could just as easily be won by the standing government which changes form after it wins as it could be reformed as a republic under a totally different political concept. France my find a way to leverage its African empire to provide an energy alternative to Germany and thereby insulate it from Moscow’s control at the same time it forces it into a subordinate relationship (winning a Napoleonic victory without fighting a war — the way the Germans thought they were “re”winning WWI without fighting a war by imposing the Euro as the new European currency under their former central bank, renamed as the ECB). The Turks may find a way to leverage their water control position over Iraq and work to put ISIS in control of Baghdad as part of a bid to force them to normalize by giving them something that can be taken away. Tehran and Washington are very likely to become close allies. etc.

The world is changing.

Many folks feel guilty pleasure at watching Russian financial numbers decline and the ruble fall as energy prices sink worldwide. Folks think “ah, this is finally it, once Russia’s economy suffers enough, Moscow will have to agree to work with Washington and stop bullying the East so much”.

Well, that last part isn’t going to happen. Not because of energy prices, anyway. There are two reasons for this: Russia doesn’t need money the same way other countries do, and in some regions alternatives to Russian energy are impossible to obtain at any price.

The first point is that Russia is a raw-materials exporter, and also maintains a considerable high-tech domestic manufacturing capability. The reason we don’t see more Russian products in the world, though, is because Russia lack much heavy shipping infrastructure. In particular, its ability to push products to ocean ports is severely limited, so it will never make sense to produce finished goods in the interior of Russia, ship them by rail or truck overland to deep water ports (across hostile political lines, no less — if you think pre-Civil War inter-state tariffs were insane within the United States, imagine what they look like in the middle of a Cold War-style mutual embargo and tariff festival), and from there to the world. China is a much easier shortcut. On the other hand, it will always make sense to ship raw materials from the interior of Ukraine or Russia (which are effectively controlled by the same political decision-makers — even more obviously now than ten years ago), because raw materials can only be had at their sources.

A big part of Russia’s power comes not from being able to throw money around, but by being able to make client states become dependent on material subsidies from Russia. If the Russian’s are subsidizing gas at a certain price in Germany, then the existing infrastructure will be built with that in mind. That lowers the difficulty of obtaining and routing that energy source. That means it lowers the cost of extending that infrastructure and thus deepening systemic dependency on that source over time. That means Russia winds up with a lever of control. As long as nothing bad happens Russia will keep the gas flowing. Once things go their way or favors are refused gas lines might “suffer breakdowns” and prices might arbitrarily increase. It doesn’t matter if the global market price for natural gas is X if that gas is physically impossible to obtain in any significant quantity when you are talking about powering an entire national economy’s energy needs. The local spot price of the gas can be whatever Russia makes up — and if Russia wants to cause pain it can simple experience a series of conveniently timed technical difficulties.

The Russians of today can, as they have for the past several waves of their history, substitute labor for capital when necessary. The methods by which this is accomplished change a bit every generation — conforming to the expectations of the peasantry (and for all practical purposes Russia is still a country of peasants and royalty). A drop in the ruble is annoying, it prevents Moscow from keeping the charade of open engagement with Western economies alive, but does not fundamentally change the power relationship between the West and East, and certainly does not change the geopolitical calculus over the long-term.

France in Syria: Still Not Here to Save the World

Folks have been really excited about France getting into war-mode on Syria after the Paris attacks. People were even momentarily excited about Turkey’s decision to shoot down a Russian plane the other day. Now Paris and Moscow are maybe working together on hitting ISIS? While the Washington is doing the same thing?!?

Man, its like a giant peace party!

Wait… um… what? No. War is not the same thing as peace. Let’s be careful to remember that. Let’s also be careful to remember that “peace”, taken without any qualifications, is a meaningless, impossible, and downright harmful goal.

Some folks are very hopeful that France’s involvement in the Syrian conflict signals a Great Change in The Way Things Work. They hope that Paris will somehow “bring Moscow into the fold” because “now Paris understands terrorism”. Other even hope that now Paris will bring NATO together under a single purpose (other than simply being an anti-Russian alliance). Lovely hopes, but not really the way things are going to work out.

I don’t mean that Paris doesn’t understand terrorism, they totally understand it — to the point that Paris is expert at both resisting it and employing it where it makes sense. They understand it so well that they know that uniting NATO “against terrorism” would make about as much sense as uniting NATO against Middle Eastern kidnapping.

I’m not saying the French are evil, mind you, I am saying they are savvy. They get the way the game works. They have been on the ball, racking up a string of strategic victories in Africa over the last decade. They’ve re-established their “middle empire” (the “middle” being between Washington and Moscow) while Washington has been too preoccupied with chasing brown guys to notice. This indicates that while they know how to play terrorism for votes in domestic politics (they are too smart to care much about what outsiders think of them) they also know that making “terrorism” the target of a major military operation is totally ridiculous.

Terrorism is a tactic not an identity. You can’t target terrorism any more than you can target long-hand division or yoga. Terrorism is the “civil disruption” phase that a political movement goes through whenever a legitimate political course of action is not available. Consider the evolution of the PLO or Hezbollah. Unless you are over 40, you likely won’t even remember that those are the groups that were arch-terrorists before. Now they are political parties. Yasser Arafat, the PLO’s Dr. Evil himself, received a Nobel Peace Prize (not that the Peace Prize means anything). Hezbollah is a fantastically profitable global franchise operation now, only partially focused (by some measures) on imposing a political outcome in Lebanon (their purpose for existing is the subject of eternally flexible rhetoric — which means the real purpose for Hezbollah’s existence is simply the survival of Hezbollah at this point).

But what about this cooperation thing? France getting into Syria must require some coordination with Russia and the US, right? And NATO? Turkey is in NATO, the US is in NATO, France is (again, that is) in NATO… so what gives?

Coordination will be necessary to prevent more “friendly” (?) fire incidents, but its more the kind of coordination that seeks to prevent midair collisions as polits jockey for superior position against one other while they run their sorties against ground targets. Remember, Turkey just shot down a Russian plane — anyone in the sky above Syria right now is considering everything else in the sky and absolutely everyone on the ground to be a threat. Sounds weird? Well, it is. But that’s reality for you. You couldn’t make up a plot for a book more convoluted than the way the real world works.

France’s goals are still France’s goals. They are not American goals. Sure, a lot of Americans and French and non-French Europeans see things the same way for the moment — but that’s a common view held of mutually held anger at a third party than anything else. The immigration wave and xenophobia that is going to increasingly fuel will continue to drive a common view over the short term (not in the least because the nightmares fueled by fear of rampant Middle Eastern and African immigration are not without foundation, particularly when coupled with domestic population decline).

France’s goal is to maintain its middle empire and use it to force Berlin into a subordinate relationship with Paris. This goal has held steady since the creation of the Euro, and France has demonstrated an amazing amount of fortitude and clarity of direction in the realization of that goal — even more amazing considering the contentious nature of their electoral politics since the Soviet collapse. Germany being in NATO with France, being home to the ECB, being “friends” with France, etc. doesn’t really matter — the reason France and Germany have been enemies so long is still based on geography, and that still forces France and Germany to regard one another in terms of capacity instead of intent. France has the upper hand in military terms, and will the economy likely to crash the only lever Germany has (dependent entirely on imported energy) is likely to disappear, or fall under the indirect control of Paris anyway if France can create an energy alternative for Germany that isn’t Russian gas (which is why France worked to isolate Germany even more from practical alternatives by destroying Libya and demonstrating their practical ability to dominate North Africa (ENI’s gas fields) and the Mediterranean).

Russia’s goals are still Russia’s goals. Well, in Russia’s case it is even more clear that Russia’s goals are actually Moscow’s goals. That is also not going to change, and despite a lot of poorly disguised epicaricacy on behalf of Western powers, Russia’s financial problems based on dropping energy prices are actually more likely to make Russia turn into a deliberately confrontational, economically detached player than a compliant ally of the West. Russia is looking out for what its own survival in what it necessarily views a dark and dangerous world. The West world will soon find itself with even fewer levers to control Russia for the forseeable future.

So France “bringing Russia into the fold”? France “getting NATO on the same page” to lead a charge against dastardly terrorist types? Nope. That’s just as naive as hoping that Russia was either sincere about squashing ISIS or helping Assad (either goal would at least speed a non-ISIS resolution to the Syrian conflict — and life under Assad wasn’t nearly as screwed up as life in a civil war…). Syria is, for the moment, a useful problem for France. That doesn’t mean that French politicians won’t accidentally start believing their own rhetoric (the way the Americans did after the invasion of Iraq was over… whoops!), but unlike 2001, the world today is full of threats that are obviously more important than chasing brown guys. In view of the Cold War II / WWIII type issues at stake right now, if the Europeans get serious about “solving” terrorism they are much more likely to resort to historically typical European solutions such as mass deportation at spear-point, mass military impressment, mass concentration, or mass execution than believing that an air campaign is going to make anything change (well, maybe carpet bombing would have some effect…).

France is a lot more likely to play Syria partly to drive a wedge between Russia and NATO (particularly Germany and Turkey), partly to demonstrate to Russia that France is willing to deal (and has something worth dealing), partly to show Washington where the red lines are (without spoiling the relationship with AFRICOM), and partly for the domestic electoral lulz. Killing ISIS guys is always good press and all it costs (right now) is printing more money and a general disregard for collateral damage (which is, ironically, why the Americans are always going to be utterly ineffective — they are absolutely afraid to hurt anyone, and religious bad guys are very good at hiding in plain sight, right behind rows of school children). Aside from this there is a vast array of geopolitical opportunities open in Syria right now, because of how the Syrian play augments the Russian play in Armenia (to pressure Turkey and keep Georgia as an effective vassal). Syria has become an interesting stage upon which Cold War II politics is playing out — this act of it, anyway.

How the Internet of Things Will Change the World: Not by Much

Are you ready for the enormous, revolutionary, ground-shattering changes coming with the IoT?

If you said “yes” and by “yes” you meant you were prepared for breathtaking changes, you are a naive child wading in a murky pool of lampreys, your will putty in the hands of the same charlatans who brought you terms like “cloud computing” which still has yet to be defined in any concrete technical sense.

If you said “yes” and by “yes” you meant that you felt that the more things change the more they stay the same — then you are indeed prepared.

Cold War II, civil war in China, the breakup of the EU, abolishment of American drug laws, the DEA and an end to the Mexican civil war all at once — those are the kinds of things that will have a measurable impact on life. The so-called “internet of things” concept as heard in internet marketing is… well, not at all what the guy who coined the term “Internet of Things” meant.

We already have an internet of things. Has it cured cancer yet? Nope. But if we put RFID in every part of our bodies we will certainly be even more exposed to the will of outside actors. Not that the public has demonstrated that it cares about complete loss of its privacy, especially when “Google style conveniences in exchange for your life’s data” can be backed up by the rhetoric of fear necessitated by government “anti-terrorism” funding. (Yes, I mock this, and yes, I was a Green Beret in the US Army for 6 years — the direction that rhetoric is headed is toward government empowerment, and the government is exactly the least well equipped element of society to deal with terrorism.)

Want to see an internet of things? Tesla cars receive system updates across the network now, and can turn in performance data to help the maker improve on their designs and software. Open water jetski robots can follow automated routes and report hydrographic and bathyrithmic data back to a data processing facility to chart change over time. I was working on a (now defunct, but promising) design project to develop spotting scopes that were intelligent enough to peer data amongst one another within an operational space and change “spotter calls” into more generally interesting “shot requests” and aggregate shot providers in the area to engage targets based on type, effect and following damage reports. Whenever any peers had a network connection they could aggregate data externally.

Dude, we’re already there.

What we lack is standards. Oh, wait, nevermind… we actually have tens of thousands of those. What we lack is standards that people actually can use, that aren’t harder to learn and comply with than the handling of the basic user problems at hand are. These problems will mostly never be solved, not really. Truly understandable data must have a semantic foundation, and semantics are essentially arbitrary in most ways that matter in data processing. That means data must either be tagged in a non-trivial way or must be placed into a schema where relationships are what have meanings.

Note that “tagged in a non-trivial way” above means taking tagging systems to such extremes that they become their own ontologies. Think about that. It should make your face turn pale. That’s at least as difficult as developing an arbitrary formal language. (In case you didn’t notice, an “arbitrary formal” language is a oxymoron — though consortia and governments alike love nothing more than funding committee efforts to formalize the syntax of futile efforts in this area). Writing even trivial software using such a tagging system would require that programmers at every step of the system learn this arbitrary formal language of tagging before they do much of anything, and that’s a lot harder overall than just continuing on with our pile-of-ad-hoc-systems approach. Schema-based systems, while having some different tradeoffs (computationally natural descriptions of data as a “shape”, for example, is a really big win in practical application), ultimately suffer from the same complexity explosion at some level. In particular, applying a particular schema designed in the context of one problem domain will very often not fit in the context of another problem domain — and fully normalizing all data ever, ever would eventually require an infinite (and ever growing) number of relational definitions. Yech.

So… Internet of things? Yeah. We’re already living it. Don’t get too excited and don’t give into the hype. Just because you technically can read data from remote sensors or activate your house’s appliances with your phone (hint: you already can) doesn’t mean this is something you will want to do, or that the venture capitalists of the world will peel their lips off the adsearch cock for long enough to realize that there are more interesting things they could be funding than bounce-under ads and invisible iframe reclick-to-click javascript tech.

Rest easy. Humanity’s material circumstances will continue to get incrementally better (save the occasional dip due to predictably stupid things we do to ourselves) until The Singularity when we are all either suddenly eliminated due to obsolescence, or drive ourselves into a new mode of existence-as-slavery to whatever Google turns into (when all data is network accessible, privacy does not exist, all data is the private IP of a single aggregate, the rights of conscious uploaded entities don’t exist, the definition of “life” is still “way way way after birth”, and continued conscious existence equates to paying a service charge — that’s not really life). None of this is particularly dependent upon the hype surrounding the “Internet of Things”.

Almost Always Stupid: Parallel Quicksort

So you’ve been thinking about applying your awesome parallel programming skills to some problem that is just complex enough to impress the noobs, but just trivial enough to code in a few lines? Cool. Knock yourself out.

But stop trying to parallelize quicksort.

Posting articles, tutorials, SO questions, and blog posts with the term “parallel quicksort” in the title only serves to encourage excitable, well-meaning, naive people to copypasta whatever code they find there and screw up what would have otherwise been straightforward, easy-to-read, relatively performant beginner-style code.

A naive quicksort implementation is pretty darn fast in the common cases found in real programs. There are edge cases where an intelligently chosen pivot can be a significant optimization — but the act of choosing the pivot intelligently is not free, which means before committing to this you have to know a lot about the context in which that particular sort will be used so you can be confident that it won’t be an overall pessimization.

A naive parallel quicksort implementation can only ever be slower than a naive serialized quicksort. Why? Think through the serial algorithm: each branch depends on the results of the subsequent branches. When we parallelize that the same dependencies exist but you’ve added the overhead of spawning and messaging to the rollup of values at the end. Even in Erlang where spawning and messaging (on a single node) are amazingly lighter weight and faster than in other environments, spawning and messaging are still more overhead in terms of time and memory than simple comparison. So a naive parallelized quicksort, while perhaps impressive because you get to show that you’re spawning and messaging everywhere, is never going to be as fast as a normal naive quicksort.

What about an intelligent parallel quicksort? Ah. Now here we have something potentially interesting. This can indeed be a significant performance win, even bigger than the case of an intelligently implemented quicksort in a guaranteed case of lopsided values where you know where the optimal pivot is located (consider for a moment how utterly tailored to the situation the choice must be, and how utterly familiar with things about that specific use case that are not quicksort one must be to know that any of this is true).

Let’s assume everything is going your way and you can already have a lot of information about the list you need to sort — enough that the overhead of finding smart pivots is outweighed by the overall running time of the sort — it can indeed be possible to write a much faster parallel sort. But not by forking more workers than you have schedulers.

The naive parallel quicksort forks/spawns every branch. That’s insane. A smart parallel quicksort will only fork until it has occupied the available scheduling resources of the system (cores, threads, schedulers, whatever the environment has). Why? Because forking any further will only incur scheduling overhead that amounts to busywork the platform has to do just to keep up with multiprocess accounting. That means an intelligent parallel quicksort will have to know not only how to pick an optimum pivot, it will also have to know how to stop forking once it has maxed out the scheduling resources of the system, and it will have to pull one more not-so-simple trick: it will have to know enough about the data being sorted to continue picking smart pivots until it is finished forking (to be faster than a simply paritioned two-threaded implementation).

That is a significant amount of arcane information needed about the input, and a not entirely obvious pile of code needed to both continue to pick pivots, communicate among threads and stop forking once the optimum number of threads for the environment has been reached. I know how to find out how many schedulers are available in an Erlang node, and I know how to find out how many cores are available on a Linux or BSD system in C. I don’t know how to find this out in Go or Java. I’m not really sure how to do it in Python, either, other than just making a subprocess call — but oops, see, once I do that the code isn’t really Python, it is “Python that works on platform X” (and not to mention, most sorts will complete in less time than a call via subprocess will… defeating the point). That’s a programmer pessimization. Spending a huge amount of time to write code that by definition probably can’t be made generic and reused later is not a very intelligent use of time (neither is writing this blog post, for that matter).

What else happens, though, when we parallelize quicksort? When we parallelize anything? Nothing is free, so there must be a cost. What are some cases where I want to sort things, and of those what cases are good candidates for parallel quicksort? This list isn’t exhaustive (sorting is pretty common) but thinking through the tradeoffs in just a few common situations may be illustrative. Remember: optimizations are usually tradeoffs. Once a routine has been optimized for the general case the remaining options are about shifting burdens away from dear resources and onto cheap ones (or, in the happiest case, not doing anything).

  1. Sorting a list before passing it to a function that requires a sorted list.
  2. Before an efficient update/merge of two lists (a special case of the above).
  3. Ordering client-side data for presentation.
  4. Organizing log data before flush (can be thought of  as a variation of the above).
  5. Building indexes.
  6. Organizing variable length search data (usually text search; a variant of the above).

In any of the cases above is it likely that I might have a set of data that is so huge that I might need to optimize the sort operation? Maybe in cases #1, #2, #4, #5, and very often in #6. So that’s entirely possible.

In any cases above is it likely that I will know enough to be able to build a customized parallel quicksort that is smart enough to pick smart pivots? Hrm… that is a lot trickier. Assuming you already might know something extra about the nature of the input it is maybe possible that this could be the case in #4, or #5 — but that is not the general case. You’d have to know an awful lot to get this right. The most likely candidate here is actually just #6.

Of the remaining cases (and for the vast majority of actual code, that is really just going to be #6), are there any situations where I want to slow down the whole system just to make a single sort go faster? Is the system doing anything else that might be important? Will this degrade overall performance when run in a cron job? Is perhaps the whole point of having lots of cores that maybe we don’t want the entire system to be affected by a single long-running operation? Consider a database that needs to build a text search index and pass it somewhere else based on whatever has changed in the last 12 hours (or an adsearch index builder that needs to sort billions of items of campaign change data). Is it acceptable for the whole system to experience a performance degradation? I don’t know — only you can answer that question. It might be a big win, it might not. I’ve only encountered a single case* where this was true in actual production code.

Once. Ever.

Now go do a search for “parallel quicksort” and skim over the results with new eyes. You almost never need this. Most articles on parallel quicksort are exactly as useful as the articles on parallel Fibonacci functions (or any Fibonacci functions, really). Sure, some of the code examples are fun to read and muse about, but they don’t get you any closer to solving whatever your actual user problems are.

[* Actually, it was adsearch campaign data, which is why I mentioned it.]

Turkey Responds to Russia: “No”

As I mentioned when Russian airstrikes in Syria began, the airstrikes have nothing to do with Assad and everything to do with keeping Washington distracted, maintaining the status quo in Syria (that is, prolonging the conflict), and pressuring Turkey (as an expansion on the already decades-old play of keeping Armenia at odds with Turkey and Azerbaijan).

The Russians did what militaries so often do when they want to present a pressuring posture and forced the issue by violating a political target’s airspace while in the course of some other operation (consider the US Navy’s recently deliberate disregard of what the Chinese claim are their “territorial waters” in the South China Sea — though the issue there is almost exactly reversed: the Chinese are the aggressors in the sense that they are laying claim to broad swathes of ocean over which Beijing has never had any practical control). Turkey decided to take the opportunity to send a message to both Moscow and Washington by shooting down a Russian jet.

The important message Ankara is sending is that they will not cooperate on any terms with Moscow, that Ankara still considers itself a Western-ally, and — perhaps most interestingly — forcing the public dialog to become, at least temporarily, about the geopolitical game that is going on instead of the incidental and petty distraction of Assad and ISIS that has been filling the news. ISIS has used terror tactics to get in the news lately (Paris made a big splash, after all), and now Turkey has used a similar technique, though not terrorism by any stretch, to change the focus of public reporting for at least a few days.

If Washington was waiting for a green light in the region before surprising everyone with a sudden shift from Arab to Persian support, this was it. The best move right now would be for Obama to show up in Tehran tomorrow, and Washington to flip sides overnight, both with regard to Tehran/Riyadh and ISIS/Assad. By getting on the Persian side of things Russia has nowhere to go, loses its lever in Iran, and has to (for the first time in two decades) react to Washington instead of being the initiator. The Israelis and Egyptians will play ball — they have before and they will again (and judging by Bibi’s deft use of hyperbolic rhetoric over the last few years, he’s ready to make a deal that let’s Tel Aviv relax), and Turkey is all but shouting out loud in plain language that its time to pinch the destabilizing issues at their source.

Whether anyone who is allowed to make a decision is paying attention is anyone’s guess — the last several years of American policy make me wonder if anyone is paying any attention at all… which is probably why Ankara is trying its hardest to force a focus on the strategic issues that underlie the future-changing alignment shifts in the region instead of letting the public dialog remain purely about peripheral issues like ISIS and Assad.

Methodologies in open source development

Prompted by an article on opensource.com about scrum in foss projects. This is an incomplete, impulsive rough draft and requires some major revisions.

Nothing makes me facepalm faster than when I get hired to work on a project and the first thing anyone tells me is “This project is going to be great because we are using the best methodologies, you know, like TDD and Scrum and everything. Totally perfect.”

WTF!? How about telling me about the problem the project solves or why it will be an improvement over whatever came before or talking about the architecture or something? There are only three conclusions I can draw from seeing that the only thing someone can think to say when I first enter a project is a word about methodology:

  1. They have no clue what the project actually does.
  2. There are no project goals.
  3. They are hiring me specifically because they have not yet encountered any other competent developers and they know it, yet (amazingly) they feel confident in dictating whatever they decided are “best practices” to me.

AHHHHHH!

Often this results in me backing out of a job, or leaving if I had tentatively agreed to it already — unless they are just going to pay enough to make watching a project tie itself in knots worth my time.

(Incidentally, I was talking with a guy from Klarna the other day and had the exact opposite experience. They know what problem their platform tackles and why it is a really good idea. So its not hopeless everywhere, just most places. What is really amazing is the guy I was speaking to wasn’t even a developer but understood what their main product is all about. That’s almost a reason to have actual hope for the future of that particular company. I don’t think I’ve written the word “hope” without prefacing it with a word like “unreasonable”, “false”, “sad” or “vain” in an article about software development.)

Today there is a problem with methodologies infecting and then blinding software projects (foss and otherwise). Its like competing flavors of ISIS taking over the ethos of various management and teams while sapping the reason from their adherents. Its not that Scrum or Agile or TDD are necessarily bad — there are a lot of good ideas that can be pulled from them — it is problematic that things like Scrum have become religions. Let’s put that in the big letters:

The problem isn’t this or that particular practice, it is the religion part.
— Me, just now

When it is absolutely unacceptable to question the utility of a methodology or specific coding practice you have a major problem. Selecting a particular set of practices without any regard to the context within which the methods or techniques are to be applied you are simply making uncritical, faith-based decisions. That is like hoping that a man who lives in the clouds will make everything better for you because you move a wooden block back and forth three times a day or kiss the ground a lot.

No, nevermind, actually it is worse than that, because you’ll only lose a few moments of your day moving a little block of wood around and kissing the ground is pretty quick. Neither has to bother anyone else who is trying to be productive right then. It is when you schedule daily or weekly meetings about block-moving techniques, force everyone to take a “vacation/working retreat” (oxymoron much?) to the tune of hundreds of thousands of dollars in company money to attend seminars given by charlatans about ground-kissing, and schedule weekend work time and “fun events” like 24-hour “hackathons” and “weekend sprints” to make up for the time lost in coordinating all the above activities. That’s way worse.

(Note that in the ranty paragraph above I’m not calling all scrum/agile/whatever coaches charlatans. The real coaches I’ve met (who have themselves written successful software) would agree entirely with what I’m writing here and tend to say straight out that chosen practices must match the context of the project. I am calling the seminar circuit and “methodology certification” guys charlatans. Those shitbags have learned that they can make crazy money by telling sweet, long, loud lies to management in a culture desperate for something to point value-blind investors at as a reason to throw good money after bad. Of course, this also implies that quite a bit of tech management is incompetent and most tech investors are just shooting in the dark.)

Without the ability to question a practice it becomes impossible to evaluate its impact on your goals. For example, sometimes TDD makes a lot of sense, especially when the product is a library. Srsly, if you write libraries and you don’t do TDD then you had better have some other word for something that amounts to the same thing. But sometimes tests — especially when they are, by necessity, integration tests of a non-trivial, comprehensive nature — can be a massive distraction, totally unhelpful, and a noticeable net loss to a project (especially project components that are literally untestable before the heat death of the universe).

The choice of a methodology or technique has to be a goal-based decision. Very often this means a business decision but sometimes its a project-goals-and-ethos type decision. Business decisions are the easiest because they are the most straightforward and the goals are relatively simple. It is a bit different in a foss project where adherence to a particular practice might be an underlying goal or a core part of the social value system that surrounds it. But even in a foss project it isn’t so hard to pin down a goal and determine how a particular practice when imposed as a rule will either help, hinder, or confuse the goal.

There is a general incongruency  between the context scrum was designed around and the context within which most foss projects exist. Scrum (and agile in general) is about customer-facing code; specifically in the case where the customer is a paying entity that is inexpert in the software being developed, but the developers are inexpert in the problem domain being solved. That is the primary Agile use case. It works well there — but this does not describe the situation of most foss projects.

Most foss projects are intended to be used by other software developers or techie users (system operators, interface specialists, DBAs, etc.), and are especially focused around infrastructure. In this case Agile simply doesn’t fit. Some documentation policies don’t even fit: there is a lot of blending among the idea that “code should be documented”, “code should be commented”, “comments should adhere to a markup that generates documentation”, “the API documentation is the manual” and “there is a product manual”. These are not at all the same things in certain markets, but in the open source world there are shades of meaning there spanning “the code is the documentation” to “literate code” to “we have a separate documentation project”.

I use foss software almost exclusively for work, but my customers don’t. They have no idea the difference, really, and don’t care over the span of a purchasing decision whether they are getting foss software or proprietary software — they only care that it works to solve their immediate business problem (that closed source sometimes introduces a different, longer-term business problem is not something they are generally prepared to understand the arguments for). In this case I have non-developer users paying for my (incidentally foss) software — and scrum/agile/whatever works very well there as a set of guidelines and practices to draw from when outlining my group’s workflow.

But the infrastructure stuff we do that sits behind all that — scrum/agile/whatever is totally insane. There is no “sit in with the customer to develop use cases” and holding a scrum meeting daily when it comes to network protocols that underlie the whole system, or the crypto suites that render the whole thing safe, or the database schemas that decompose the raw data into meanings relevant on the back-end. That sort of stuff is generally core-competency for foss developers, but it doesn’t fit the scrum or agile methodologies at all.

Only when you are working on a wicker-castle of a web project where concerns are so totally interwoven and mishmashed together that front-end customer-facing concerns become backend concerns does agile make any sense — but in this case it only makes sense because you are shipping a giant ball of mud and your hair is on fire every single day because you’ll never catch up with breakages (welcome to the actual modern paradigm that rules 90% of software projects I’ve seen…).

I’ve poked at a few different sides of development. What I’m really getting at is that there are different worlds of software development, and each world is suited better or worse by different methodologies that have been developed over time. (As an extreme case, consider the lunacy of scrum in most embedded projects.) No methodology I’ve ever seen fits any given project very well, but all of them absolutely work against the interests of a project that exists outside the context from which they originated. That could be because of time constraints (consider the infamous “docs don’t pay” attitude — yet it shouldn’t be dismissed outright because it is actually a valid business concern sometimes), the context of use doesn’t fit, or whatever.

In the same way that we (should) try to pick the best tools (languages, data stores, data structures, network paradigms, etc.) for a given job based on the nature of the task, we should approach selection of development methodology and team workflow based on the nature and context of the project.