Category Archives: Science & Tech

Everything from lampooning Popular Science to picking on Microsoft and Oracle belong here.

On the Meaning of “Carbon Neutral”

I noticed that a few products in my house have begun to proudly proclaim themselves as being “carbon neutral” over the last few months. Apparently this is among the new set of empty phrases marketing people feel are necessary to distinguish their otherwise ordinary commodity products from identical products of comparable quality. It used to be “Made in U.S.A.” or “日本製” (depending on the neighborhood), then it was “low sodium”, then “waterproof”, then “low fat” then “low transfat” then “cholesterol free” then “omega-3″ then something else I probably forgot.

The problem isn’t that any of these things aren’t potentially good selling points, its that they usually don’t apply to the things I see the labels on. For example, I remember seeing an electric wok that said “Made in U.S.A.” on the bottom. I’m not so sure that’s the best thing to concern one’s self with when buying a cooking apparatus that originated in another hemisphere. That’s like buying a tuna steak because the sticker on the package marks it as being “a peanut-free product” or believing that a piece of software is high quality because its written in Java (or even more uselessly, “utilizes Java technology!”).

This reminds me of my sister’s enlightening tale of the truth behind the now heavily regulated terms “organic” and “all natural” as applied to food labels. She did her undergraduate study in genetics and graduate work in nutrition, worked in colon cancer research for a while, started a dietary medicine program at a hospital in Virginia a few years back, and now (after reaching some disillusionment with the state of government-funded research) raises “range fed Angus beef” as a side interest. She is therefore directly governed by some of the more hilarious regulations the FDA has come up with.

Needless to say, her opinion on the value of these buzzwords has much more influence to me than whatever a “medicinal cannabis expert” has to tell me about the benefits of toking up or the local yoga girl at the gym has to tell me about the benefits of yogurt shakes or almond oil or peanut-butter enemas or whatever it happens to be this week (of course, she’s just right about the benefits of sex in exciting places). In short, the regulations governing terms such as “organic” and “natural flavor” (or even the way the term “X% fat free” is permitted to be used) are both economically draining legally apply due to the administrative overhead of regulatory compliance and yet so full of loopholes that there is really no clear distinction between a head of lettuce that is “organic” and one that isn’t so labeled. Essentially the only difference is the price of the market package.

Of course, the real difference is that the lettuce sporting an “organic” sticker on it is almost undoubtedly produced by a large agribusiness firm that can afford the overhead of doing all the pencil-drills necessary to proclaim their lettuce to be “organic”. Either that, or it is quite pricey lettuce only rich folks who feel the need to spend more to sate their moral thirst can afford, grown at an “organic” farm run by one savvy businessman and a flock of altruist peons bent on saving humanity from itself one vegetable at a time. I’m certainly not saying that large agribusiness is bad — ultimately its the only way we’re going to survive over the long-term (and here I’m including post colonization of space) — but that the terms used on packaging are enormously deceptive in nature.

But that’s food. It is a specific example where it is relatively easy to trace both the regulatory documentation and the research literature. Of course, very few people actually track all that down — other than unusual people like my sister who happen to be trained in genetics, directly involved in agriculture, and so habituated to both scientific and regulatory research that they find nothing daunting about navigating the semantic labyrinth the FDA has let agricultural regulation become in the US (and the phrase “let…become” could easily be replaced with “deliberately made of…”). I suppose the problem isn’t that few people track all that down, really; its more a problem that even if my sister were to go to the trouble of explaining what she knows to the average consumer they wouldn’t have the faintest clue what she was getting at. The average consumer is instead faced with an essentially religious (or at least dogmatic) choice of whether to trust someone that has a stack of official paper backing up her credibility, or a government agency and a huge food industry which are both populated by thousands of people who each have every bit as much officious documentation backing up their reputations.

And that brings me back to “carbon neutral”. We still chase the purported value of demonstrably empty terms such as “cloud computing”, demonstrably failed vehicles such as “social networking”, and demonstrably flimsy labels such as “organic” and “all natural”. But we don’t stop there. We are jumping head-first onto the “carbon neutral” bandwagon as well. The point isn’t that we shouldn’t be concerned with the terrestrial environment, but rather that we must at all times guard against political forces that constantly seek to invent new social mores and foist them on us by conjuring meaning into empty phrases like “carbon neutral”. It tricks you not just into buying ordinary thing A over ordinary-but-cheaper-thing B, but also into feeling morally superior. In this it is indistinguishable from other dogmatic rhetoric that engenders an unfounded sense of moral certainty. If we thought convincing people that a man in the sky doesn’t want them to fly airplanes into office buildings was hard, consider how much more difficult it is to convince average people who genuinely want to “do good” that reasonablish sciency words are nothing more than unfounded political siren songs trying to open one more door for the tax man.

So back to the reasonablish sciency phrase “carbon neutral”… what does it mean? This is where I have some semantic issues, mainly because nobody really knows.

Let’s say, for example, that we start a paper mill. We’ll make paper, but only from recycled paper and only using wind energy. This could probably qualify as being entirely “carbon neutral”. But so could the same paper mill if it planted its own trees. But what about the wind generators? They have to come from somewhere. What about the diesel-powered trucks that carry the old paper stock to the recycling mill? What about the initial material itself? Are we being carbon neutral if we don’t go replace as many trees as our recycled stock represents? How about the electricity used by the paper-compactors run by other companies we have no control over? What about our employees’ cars they use to get to work? What about all the flatulence the invite by eating pure vegan meals?

The initial production itself would almost certainly not qualify as being “carbon neutral” — which demonstrates that we have to draw an arbitrary line somewhere from which we can derive a meaning for the term “carbon neutral”. It is almost certain that something, whether directly or indirectly, involved an increase in carbon emissions (and the meaning of “direct” and “indirect” really should be their own battlegrounds here, considering what people think the term “carbon neutral” means) somewhere at some point, otherwise there wouldn’t be people to buy our recycled earth-friendly paper to begin with.

But what are “carbon emissions”? This is, apparently, intended to only refer to releasing carbon into the air. Consider for a moment how monumentally arbitrary that is. There are currently some well-intended, but enormously misguided efforts to “sequester” carbon by burying it in the crust of the Earth. This, of course, represents an enormously heavy emission of carbon into the environment, but we are calling this a “good” emission (actually, we refrain from using the word “emission” because we intend that to be a “bad” word) because it is going into the ground and not the air. Incidentally, it is also not going into something useful like diamond-edge tools or nano insulators or any other beneficial process that is desperate for carbon (which our planet happens to be poor in by cosmological standards).

So where did all this “bad” carbon come from? If you believe the press, its coming from our SUV exhaust, coal-burning plants, Lady GaGa (well, she might be a Democrat, in which case she can’t be bad), and pretty much anything else that humans use to modify local order at the expense of a probable increase in universal entropy.

Where did the carbon come from for the coal, crude, natural gas and bovine flatulence? Probably from the atmosphere and the sea. At least that’s what a biologist will tell you.

And here is a problem for me. Nobody has explained (not just to my satisfaction, but explained at all) where all the billions of tons of carbon necessary to create the forests that created the coal (and possibly crude oil) came from in the first place.

Well, that’s not quite true. In the first place it came from a long-dead stellar formation, some crumbs of which clumped together to form our solar system. That’s the first place. So the second place. Where did the carbon for all this organic activity come from in the second place? Was it distributed evenly in the early Earth? Has it always been a fixed quantity in the atmosphere? Does it boil out of the molten terrestrial substrate and gradually accumulate in the atmosphere and ocean?

If the forests grew in the first place then the carbon was in the air, at least at one point. If it is a fixed amount of atmospheric carbon then the growth of the forests and their subsequent demise and burial beneath sediment represents an absolutely massive sequestration of atmospheric carbon. If it is indeed a fixed amount, then the absolutely huge amounts of flora and fauna represented by these forests were not prevented from thriving in an atmosphere which contained a huge amount more carbon than the atmosphere contains today. If that is true, then either climate change is not affected much by the carbon content of the atmosphere, or a changed climate does not pose much of a threat to organic life on Earth.

Some parts of the fixed atmospheric quantity scenario don’t really add up. Despite a bit of effort I’ve only managed to scratch the surface of the ice core research literature, but a static amount of available atmospheric carbon doesn’t seem to be the story research efforts so far tell. This area of research has been made amazingly difficult to get a straight tack on ever since environmental sciences got overheated with politically-driven grants looking for results that validate political rhetoric instead of grants seeking research into what is still largely an observational field, but it seems fairly clear that there have been fluctuations in atmospheric carbon content that do not directly coincide with either the timing of ice-ages or the timing of mass terrestrial forestation. (The record is much less clear with regard to life in the ocean — and this could obviously be a key element, but it doesn’t seem that many people are looking there, perhaps because the current rhetoric is full of fear of rising sea levels, not full of hope for a marine component to the puzzle of eternal human salvation). That said, there must be some pretty massive non-human sources of atmospheric carbon which have been in operation millions of years before we evolved (as for where trillions of tons of carbon may have gone, I think the huge coal formations may be an indication).

While the idea that a carbon-rich atmosphere providing adequate conditions for thriving terrestrial life might seem odd (at least when compared with the “Your SUV is killing the Earth!” dialog), the idea that the Earth itself has both mechanisms to gradually ratchet up the total amount of carbon in the atmosphere over the eons and to drastically change the climate in spans measured in mere years (not decades, not centuries or millenia) without human or even atmospheric input is pretty scary.

A lot more scary than the idea that driving my kids to school might be damaging in some small way.

But this isn’t the way we are thinking. We are letting marketers and politicians — two groups infamous for being as destructively self-serving as possible — sell us a new buzzword stew, and we, the consumers, are ready to confidently parrot phrases such as “carbon neutral” about as if they mean something. “Oh, Irene, that salad dressing is 100% organic and carbon neutral — you’re such a gourmet!”

We’re clearly having the wool pulled over our eyes, except this time it doesn’t just play to our ego-maniacal craving to live forever (“If you eat gluten-free yogurt and drink positive-ion water you’ll live forever — and have huge tits/a thicker penis/ungraying hair/a tiny waist!”), it engenders a dangerous sense of moral superiority (“I’m doing this for the planet/global socialism/God/The Great Leader!”) which tends to eliminate all possibility of rational thought on a subject which could indeed affect us all.

What if, for example, the Atlantic currents are just panting their last, barely keeping us away from a global mass cooling event? We won’t just be blind to the threat because we’ve blown our research money on politically driven quests to generate the academic support necessary to pursue whatever new pork-barrel projects we come up with over the next decade or two — we will deny the very idea that a threat other than carbon emissions could possibly exist on moral grounds because we’ve already identified the “real enemy” (wealthy people in SUVs who come with the added benefit of being fun to hate). That’s dangerous.

Words mean things. We should remember that.

Keyboards, Machine Guns, and Other Daily Tools

It looks like I’ll be at least occasionally moving between my home in Japan and offices in the US where I may wind up setting up a system for myself to use while I’m there (I’m not a huge laptop fan when it comes to extended work). This brings up an annoying issue: keyboard layouts.

It is difficult to find US-layout keyboards out here, so even though I usually write only a few Japanese-language emails per day its just not practical to use anything but the local flavor. Even if I did have a bunch of US-layout keyboards it would be insanely annoying to have to switch between JP-layout on laptops, server crash carts and customer systems and then switch back to US-layout when I got back to one of my offices. So I’ve gotten accustomed to this layout and it works well for me.

The main keys that do letters and numbers are all in the same place, so it seems like this wouldn’t be a big deal. The problem is the crazy keys that do “top row” and wildcard stuff like bangs, hashes, quotes, backticks, at-marks, brackets, colons, semicolons, parens, etc. All the stuff that is rarely used on, say, a student or blogger’s keyboard is usually worn smooth on a programmer’s keyboard, especially if he switches languages all day. And they are all in radically different places on JP and US layouts.

So… naturally I’ll probably just get a decent one here and keep in the closet over in the US, and whip it out whenever I show up.

But that brings up a point about familiarity and how “good” tools are from the perspective of the one using them. I could easily take the position that US-layout is poo and that JP-layout is superior. Or I could get uber nerd and pretend that some statistical study on finger reach and frequency of blahblahblah matters whatsoever when it comes to programming. It doesn’t, really. That’s to imagine that input is the hard part of programming, and its not — its figuring out what to write in the first place. So its not speed of input, per se, but smoothness of operation. More to the point, its which layout prevents the wetware halting problem: where the human doing the work has to stop what he is doing to figure out something unrelated to the essential task at hand.

But it remains true that some layouts are probably actually worse than others. It follows that other sorts of tools can fall into the realm of “good enough that preference is a matter of taste or familiarity” or in the realm of “actual poorly designed garbage”.

The reminds me of guns. There are several excellent machine gun, rifle and pistol designs employed in militaries across the world. Many of them are decent enough that while some have a slight edge over others in some areas, I’d go to work with pretty much any of them. For instance, the M4 vs. the SCAR. The SCAR is actually better, but the M4 is familiar enough to me and I have enough confidence and skill with it that I just don’t really care which one I wind up getting stuck with.

I don’t have nearly as much faith in the AK-47 as a precision weapon, especially in an environment where quick on/off safety and partial reloading is critical. They are famously resistant to neglect (which is often mistaken for durability), but that’s really a key attribute for a rifle intended for the mindless Commie Horde sweeping en masse across the tundra or untrained insurgent/freedom-fighter/terrorist whose backers need cheap trashy guns with which to arm their cheap trashy goons. Indeed, the AK-47 is in real terms less good than the SCAR or M4 and there is a whole list of rifles and carbines I would consider before going to work with one (but still, its not absolutely awful, just so much less good than the alternatives that I’d avoid it if possible — sort of like Java).

Where this is really striking is with machine guns and pistols. On the pistol side there are a rather large number of designs that actually break down frequently during use. This never happens in a James Bond movie, of course, but in real life it happens at the most inconvenient times. Come to think of it, there is never a convenient time for a pistol to break. Once again, despite the absolute superiority in design of the semi-automatic over the revolver, familiary can overcome the technical deficiencies between the two (with lots of practice) and I would actually prefer to go to work with certain revolvers over certain semi-autos. (This is to say nothing, of course, of the issue of caliber…)

With machine guns, however, the differences in good vs. bad designs are vast. In the nearly any modern military you’re absolutely spoilt. A “bad” gun is one that doesn’t have a TV built into the stock to ease the passage of long turns on security. They are mindlessly easy to load, sight, barrel change, fire, strip, clean, maneuver with, etc. The links disintegrate and can be re-used on unlinked ammo, all sorts of cool toys fit around the thing (which can, sometimes, make them start to suck just from the “too much Star Wars” problem), runaways can have their belt broken, they will eat through just about any garbage that gets caught in the links or even fire bent ammo. They aren’t even unreasonably heavy (and its patently unfair to compare it to the uber lightness of an M4). Its amazing how well these things work. But when they are all you know you start complaining about them, wishing you had a 240 when you’ve been handed an M-60 (because its possible to jam it up if you accidentally load it bolt-forward, or probably lacks a rail system, or you’re an unsufferable weakling complaining because you didn’t get the lightweight bulldog version, or whatever). I’ve had the misfortune of having to go to work with old Soviet machine guns, though, and can attest that they are indeed of almost universally horrible design.

When we say “crew served weapon” in modern armies we mean “the weapon is the centerpiece of the crew” not “this weapon is absolutely unreasonable to assign to any less than three people”. It might have meant that operating the machinery actually took a crew back when tripods included full-sized chairs, ammo came on a horse-drawn cart, and vast amounts of oil and water were consumed in operation. But that was the early 1900’s. We still employ machine guns as crew served weapons beacuse its an advantage to have an AG and actually set up a tripod if you wind up facing off against a for-real infantry force, not because its difficult to wield one. Today a single person can easily maintain and operate a 240, M-60, MAG58, 249, MG42, MG3, or whatever. Not so with, say, the PKM (or heaven forbid the SG-43). An RP-46 is actually better if you come to the field with American-style assumptions that a single person is adequate to handle a machine gun.

The PKM is not really belt fed, its chain fed, and the chain doesn’t disintegrate. Its also extremely strong. Like you can support more than a single person’s weight from a belt and it won’t break. The longer the belt the more bullets, and this seems good, until you realize that it feeds from the wrong side (the right), which prevents a right-handed shooter from feeding the pig himself with his left hand and leaves the indestructible spent chain right in front of the shooter. This means its right underfoot when running after a bit of shooting — which has made be bust my face in the dirt on the top of the gun more than once (not so convenient at interesting moments, and absolutely detrimental to my Cool Point count).

But the failure of design doesn’t stop there. That stupid belt is nearly impossible to reload by hand without wearing gloves and using a lever (box top, table top wrench, whatever) to force the rounds into the thing (yeah you might load 50 rounds by hand, but how about 5000?). They also rust instantly, in accordance with the PKM Belt Rust Time Law — however long its been since you last packed the belt is precisely how long it takes to rust exactly enough to generate a vast amount of busy work without rusting so much that the belt should be discarded. If you try oiling them to prevent that they gum up or actually start growing hair instantly. Its a never ending cycle of trying to keep the belts from making your life suck without giving up and throwing them all away. Which is why the Soviets conveniently invented a reloading machine. Which itself sucks. I can’t even begin to explain the inadequacy of this stupid machine, but it actually is the only way to maintain even a marginally reasonable reload rate for belts — but there is no way you could do this under fire, or on Tuesday (the machine jams spectacularly on random days, Tuesday tending to be the worst day for this for some magical reason).

I haven’t even begun to mention the inadequacy of the ammo crates. The standard ammo crates are insanely stupid. Actually, this isn’t a gripe reserved just for 7.62 ammo, its true for all commie ammo I’ve ever seen. The ammo cans aren’t like the infinitely reusable, universally useful, hermetically sealed, flip-top boxes found in non-backward armies. They are actually cans. Like giant soup cans, but without a pull-tab — not even a sardine-key. They come with a can opener. A huge one (but only one per crate, not one per can). You read that right, a can opener. You know, the lever-kind where you hook the grabby part onto the crimp at the top edge of the can and pull to lever the pointy part down until it makes a tiny puncture, then slide over a touch and repeat until you’ve prized and ripped a gash large enough to do your business. Let that sink in. We’re talking about an ammo can. Like with bullets that people need to do their job, hopefully sometime this year. But once you’re inside the fun just doesn’t stop — no way. The thousand or so rounds inside are in boxes of 5 or 6 or so. The can that you worked so hard to open isn’t full of pre-loaded belts. That would deprive someone of a government job somewhere and that’s just not Progressive. So inside there are dozens and dozens of tiny, crappy, flimsy little cardboard boxes, each containing a few rounds. And the rounds are individually wrapped in tissue paper.

You just can’t make this trash up. Its amazing. How on earth could such a horrible, stupid, backward constellation of designs emerge from one of the two nations to reach the Moon before the end of the 20th century?

A guy I worked with a few years ago called Mule had a theory that this was, in fact, an excellent design for a machine gun system in a Socialist military. Nobody can use it alone, so you can’t get a wild hair up your ass and get all revolutionary — you need to convince at least a platoon to get crazy with you. You employ a gazillion people not only in the loving production and hand gift-wrapping of each one of the billions of rounds of machine gun ammunition throughout the nation, you employ another gazillion or so to open and load the belts. Its the ultimate low employment figure fixer — at least until the state digests enough of itself that this becomes suddenly unsustainable, of course.

Mule’s theory was that this machine gun design — from the actual shittiness of the gun itself to the complete circus of activity which necessarily surrounds its production, maintenance and use — is a brilliant design from the perspective of the State, not the soldier, and that the aims of the two are at odds is simple the natural result of a socialist system. Mule was one of the most insightful people I’ve ever met (and I’m not being rhetorical — he really was a hidden genius).

Thinking about what he said has made me re-evaluate some of my assumptions of bad designs. Perhaps the designs are excellent — not for the end user, but for whoever is in charge of the end user. And that brings me back to thinking about just why the Java programming language is so bad, yet so prolific. Java is the PKM of the programming world. Its everywhere, it sucks, it is good for (some, Stalin-like) bosses, and the whole circus surrounding its existence just won’t ever go away. And sometimes those of us who know in painstaking detail why a 240 (or nearly anything else in common use) is better are still stuck using it to get real work done.

Development Speed VS Quality

I’ve been working under some pretty insane time constraints lately. Two things jump out at me upon review of my work:

  1. I can, when cornered, churn out thousands upon thousands of lines of functioning code in a flash.
  2. The code works, but it is not particularly insightful or brilliant — it merely works.

The first bit is sort of cool: Typo Monster and Syntax Bear are no longer a part of my life (at least not in my Big Five). Nice.

On the other hand, the second bit isn’t so nice. It means that insightful development requires far more time than people tend to imagine. This has been a point of discussion for decades in engineering and especially software development, but its hard to fully appreciate until you get a chance to compare your own code, written once under duress, with code written for a similar problem domain at a pace that allows more time for grokness.

Now that I think of it, its not the pace exactly which is the problem, it is the way that certain forms of deadline and environmental pressures can re-tune the time budget in a way that extends typing/coding/input time at the expense of grok time.

This is particularly pronounced when it comes to data modeling — no matter if this means in the sense of managed state, a relational database schema, a no-schema blobbulation, a serialization scheme of some sort, or an object/class model.

This is a particularly touchy thing, since rearranging a data model of any sort entails major surgery when more than one thing relies on the way data is stored, but refactoring functions is relatively lightweight (so long as you understand what the intended mapping was to begin with).

I suspect that the vast majority of production code is written under deadline and that most of it relies on expedient and trashy data models. I further suspect that most corporate settings urge programmers to “be programming” and officially establish that “programming” means “typing” instead of “understanding”.

It is shocking to revisit something written in a hurry and simplify it in a flash of insight, and then contemplate what just happened. The surprise is all the more resonant once you fully realize how much less time is required to input insightful code than rushed, verbose-but-functional trash.

Looking to my left and right, it is not particularly evident that much production code benefits from a culture of sacred Grok Time.

More on a new Data Language

I’ve given more thought to the new data language I’m working on, and finally decided on a name as well.

The new baby shall be called RyuQ — a reference to where I live respelled to accommodate the mandatory “Q” in any shiny new query language. I’ve settled on a modified form of S-expressions, calling them SP-expressions to distinguish between real S-expressions and my mutated version. The (still infantile) language description is peppered with examples, so you can see pretty quickly if SP-expressions feel comfortable or more alien relative to SQL.

I have to say, the further I go on this the more I wonder what was going through the minds on the committee that created SQL. So many things are just so obvious when sticking to an algebraic notation rather than trying to form some new thing that is neither relational algebra nor relational calculus.

Until I have a complete implementation worked into a fork of Postgres I won’t be able to do anything but toy with this language — but the more thought I give it the less I am satisfied with the work I am currently compelled to do in SQL.

Fedora: A Study in Featuritis

Its a creeping featurism! No, its a feeping creaturism! No, its an infestation of Feature Faeries! No, its Fedora!

I’ve been passively watching this thread (link to thread list) on the Fedora development list and I just can’t take anymore. I can’t bring myself to add to the symphony, either, because it won’t do any good — people with big money have already funded people with big egos to push forward with the castration of Fedora, come what may. So I’m writing a blog post way out here in the wilds of the unread part of the internet instead, mostly to satisfy my own urge to scream. Even if alone in the woods. Into a pillow. Inside a soundproof vault.

I already wrote an article about the current efforts to neuter Unix, so I won’t completely rehash all of that here. But its worth noting that the post about de-Nixing *nix generated a lot more support than hatred. When I write about political topics I usually get more hate mail than support, so this was unique. “But Unix isn’t politics” you might naively say — but let’s face it, the effort to completely re-shape Unix is nothing but politics; there is very little genuinely new or novel tech going on there (assloads of agitation, no change in temperature). In fact, that has ever been the Unix Paradox — that most major developments are political, not technical in nature.

As an example, in a response to the thread linked above, Konstantin Ryabitsev said:

So, in other words, all our existing log analysis tools have to be modified if they are to be of any use in Fedora 18?

In a word, yes. But what is really happening is that we will have to replace all existing *nix admins or at a minimum replace all of their training and habits. Most of the major movement within Fedora from about a year ago is an attempt to un-nix everything about Linux as we know it, and especially as we knew it as a descendant in the Unix tradition. If things keep going the way they are OS X might wind up being more “traditional” than Fedora in short order (never thought I’d write that sentence — so that’s why they say “never say ‘never'”).

Log files won’t even be really plain text anymore? And not “just” HTML, either, but almost definitely some new illegible form of XML by the time this is over — after all, the tendency toward laughably obfuscated XML is almost impossible to resist once angle brackets have made their way into any format for any reason. Apparently having log files sorted in Postgres wasn’t good enough.

How well will this sit with embedded systems, existing utilities, or better, embedded admins? It won’t, and they aren’t all going to get remade. Can you imagine hearing phrases like this and not being disgusted/amused/amazed: “Wait, let me fire up a browser to check what happened in the router board that only has a serial terminal connection can’t find its network devices”; or even better, “Let me fire up a browser to check what happened in this engine’s piston timing module”?

Unless Fedora derived systems completely take over all server and mobile spaces (and hence gain the “foist on the public by fiat” advantage Windows has enjoyed in spite of itself) this evolutionary branch is going to become marginalized and dumped by the community because the whole advantage of being a *nix admin was that you didn’t have to retrain everything every release like with Windows — now that’s out the window (oops, bad pun).

There was a time when you could pretty well know what knowledge was going to be eternal (and probably be universal across systems, or nearly so) and what knowledge was going to change a bit per release. That was always one of the biggest cultural differences between Unix and everything else. But those days are gone, at least within Fedoraland.

The original goals for systemd (at least the ones that allegedly sold FESCO on it) were to permit parallel service boot (biggest point of noise by the lead developer initially, with a special subset of this noise focused around the idea of Fedora “going mobile” (advanced sleep-states VS insta-boot, etc.)) and sane descendant process tracking (second most noise and a solid idea), with a little “easy to multi-seat” on the side to pacify everyone else (though I’ve seen about zero evidence of this actually getting anywhere yet). Now systemd goals and features have grown to cover everything to include logging. The response from the systemd team would likely be”but how can it not include logging?!?” Of course, that sort of reasoning is how you get monolithic chunk projects that spread like cancer. Its ironic to me that when systemd was introduced HAL was held up as such a perfect example of what not to do when writing a sub-system specifically because it became such an octopus — but at least HAL stayed within its govern-device-thingies bounds. I have no idea where the zone of responsibility for systemd starts and the kernel or userland begins anymore. That’s quite an achievement.

And there has been no end to resistance to systemd, and not just because of init script changeover and breakages. There have been endless disputes about the philosophy underlying its basic design. But don’t let that stop anybody and make them think. Not so dissimilar to the Gnome3/Unity flop.

I no longer see a future where this distro and its commercially important derivative is the juggernaut in Linux IT — particularly since it really won’t be Linux as we understand it, it will be some other operating system running atop the same kernel.

Come to think of it, changing the kernel would go over better than making all these service and subsystem changes — because administrators and users would at least still know what was going on for the most part and with a change in kernel the type of things that likely would be different (services) would be expected and even well-received if they represented clear improvements over whatever had preceded them.

Consider how similar administering Debian/Hurd is to administering Debian/Linux, or Arch/Hurd is to administering Arch/Linux. And how similar AIX and HP/UX are to administering, say, RHEL 6. We’re making such invasive changes through systemd that a change of kernel from a monolothic to a microkernel is actually more sensible — after all, most of the “wrangle services atop a kernel a new way” ideas are already managed a more robust way as part of the kernel design, not as an intermediate wonder-how-it’ll-work-this-week subsystem.

Maybe that is simpler. But it doesn’t matter, because this is about deliberately divisive techno politicking on one side (in the vain hope that “if our wacko system dominates the market, we’ll own the training market by default even if Scientific Linux and CentOS still dominate in raw numbers!”), and ego masturbation on the other (“I’ll be such a rebel if I shake up the Unix community by repeatedly deriding so-called ‘Unix traditions‘ as outdated superstitions and generally giving the Unix community the bird!”) on the other.

Here’s a guide to predicting the most likely outcomes:

  • To read the future history* of how these efforts work out as a business tactic, check the history of Unix from the mid-1980’s to early 2000’s and see how well “diversification” in the interest of carving out corporate empires works. I find it strikingly suitable that political abuse of language has found its way into this effort — conscious efforts at diversification (defined as branching away from every other existing system, even your own previous releases) is always performed under the label of “standardization” or “conformance to existing influences and trends”. Har har. Joke’s on you, though, Fedora. (*Yeah, its already written, so you can just read this one. Easy.)
  • To predict the future history of a snubbed Unix community, consider that the Unix community is so used to getting flipped the bird by commercial interests that lose their way that it banded together to write Linux and the entire GNU constellation from scratch. Consider also that the original UNIX was started by developers who were snubbed and felt ill at ease with another, related system whose principal flaw was (ironically) none other than the same featuritis the Linux community is now enduring.

I don’t see any future where Fedora succeeds in any of its logarithmically expanding goals as driven by Red Hat. And with that, I don’t see a bright future for Red Hat beyond v7 if they don’t get this and other priorities sorted**. As a developer who wishes for the love of everything holy that I could just focus on developing consumer business applications, I’m honestly sad to say that I’m having to look for a new “main platform” to develop for, because this goose looks about cooked.

** (sound still doesn’t work reliably — Ekiga is broken out of the box, Skype is owned by Microsoft now — Fedora/Red Hat don’t have a prayer at getting on mobile (miracles aside) — nobody is working on anything solid to stand a business on once the “cloud” dream bubble pops — virtualization is already way overinvested in and done better elsewhere already anyway — easy-to-fix media issues aren’t being looked at — a new init system makes everything above worse, not better, and is distracting and requires admins to completely retrain besides…)

Fred Brooks: Design Requirements

This overlooked resource is a talk that Fred Brooks gave to a very small audience June 4, 2007 — the first day of the WSOM Design Requirements Workshop. He does not discuss requirements or elicitation of requirements (which is what everybody else talked about) but rather discusses design as a process and how that is inseparable from the process of divining requirements. His primary points are:

  • You can’t know what you want to build until you already have something concrete. Everything is a throw-away until it is “good enough”.
  • Since you can’t know what you want to build, you don’t know the requirements, even after you start work. This indicates that the process of design and the process of divining requirements cannot be treated separately the way that academic literature treats the subjects.
  • The design process sometimes occurs in a more complex environment than in past years because of the late 20th century concept of “collaborative design”. Further, this is not the best design model for every project.

Unlike many other Fred Brooks talks there is a bit of audience interruption — sometimes with valid points, sometimes not — and you can observe both Fred’s unscripted response as well as others in the audience responding to audience comments and questions. It is interesting from a human perspective for this reason as well.

On Looping and Recursion

This post was prompted by a discussion about various programming languages on the Scientific Linux Forum. The discussion is in the member’s sub-forum, so I can’t link it here very effectively.

wearetheborg wrote:

I don’t understand the statement “…develop a sense time being slices along the t-axis (similar to thinking transactionally) instead of dealing in terms of state.” Can you elaborate on this? I have been envisioning state as the state of the various variables and objects.

That is the way most people think of state. This is normal in procedural and structured programming languages (everything from assembler to Fortran to C to Java). It doesn’t have to always work that way, though.

Consider a web request. HTTP is stateless by design. We’ve backhacked the bejeezus out of it to try making it have state (cookies, AJAX, etc) but in the end we’re always fighting against that protocol instead of just accepting that documents aren’t application interfaces. But remember the original Perl CGI stuff? That was all based on the way databases treat time — as transaction points along the t-axis. This is very different than inventing the notion of ongoing state. So a user fills out a form, and when he submits it there is a database transaction that completely succeeds or completely fails. Each request for a page view generates a new page via a function whose input is the request URL (and whatever detailed data lies within the GET string), which then calls another function whose input is a complete response from the database containing a snapshot of data from a point in time, and whose output is the web page requested.

There is no necessity for state here. The input to the function is the request URL string. The function breaks that down and returns a complete output (which could be an error message). But there is nothing ambiguous about this and executing the same functions with the same inputs would always return the exact same output, every time. There are no objects carrying state between requests. There are no variables that live beyond the request -> response lifetime.

“But timestamps and things might change and if someone updates a page then a request before and after would be different!” you might say. And this is true (and indeed the whole point of such CGI scripts) but all that is external to the functions themselves. The content of the pages, including the timestamps, page templates and record data, are input to the functions, not persistent state variables that are evolving with time. The functions don’t care what the inputs are, they only need to do their job — the data is a completely separate concern (remembering this will minimize fights with your DBA, by the way… if you know an asshole of a DBA, consider that he’s probably not trying to be an asshole, but rather trying to help you simplify your life despite yourself).

All this functions-that-don’t-carry-state business means is that we don’t need variables, we need symbolic assignment. Further, its OK if that symbolic assignment is immutable, or even if that assignment never happens and is merely functions passing their results (or themselves) along to one another.

This is the foundation for transactional thinking and it treats time as slices or points along the t-axis instead of a stateful concept that is ongoing. Its also the foundation of quite a bit of functional programming. There are no variables internal to the functions themselves that have any influence on the output of the program. Every case of a given input results in an exactly defined output, every time. This simplifies programming in general and debugging in particular.

Its not the best for everything, of course. In simulations and games you need a concept of state to cover the spans between transactions or time-slicing periods. For example, in a game you are fighting a mob. The fight isn’t instant, it is a process of its own and the immediate gameplay is based on the state carried in the various objects that make up your character, the mob, equipped items, world objects, etc. But it doesn’t really matter in a grander sense whether or not each strike or point of time in the fight is actually transacted to the data persistence layer — if the server crashed just then you’d probably prefer to not log in back to the middle of a boss fight while your screen is still loading, or half your party isn’t logging in at the same time with them, etc.

So the fight is its own discreet subsystem which is intimately concerned with state and OOP design principles and is therefore completely expendable — it matters not if you won or lost that particular fight if the server crashes in the middle of it. The overall world, however, is designed based on transactional concepts of time slices — it does matter what the last non-combat status and position snapshot of your character was, and you probably care quite a bit about certain potentially monumental events like when your character binds himself to some epic loot, level up or apply a new skill point (that better go in the database, right?).

The vast majority of user applications aren’t games or simulations, though. Almost everything on the web is text string manipulation and basic arithmetic. Almost everything in business applications development amounts to the same. So we don’t need stateful functions and objects, in fact we get confused by them (the exception here being the interface itself — objects make awesome widgets and windows). If I make an object to represent, say, a customer invoice, and I’m doing calculations on that invoice or within it using methods, to really bug test that object I have to test every method under every possible condition, which means every permutation of state possible. If that object is a subclass of anther object, or interacts with another object in the system, like say a customer record object or a discount coupon object, I have to test both against each other in every combination of possible states. That’s impossible in the lifetime of this universe, so nobody does it. The closest we come is testing a tiny subset of the most likely cases out of a tiny subset of what’s possible, and then we are constantly surprised at lingering bugs users report in modules later because they passed all the tests (or even dumber, we aren’t surprised at all, which says something about how blindly we stick to methodology in the face of contrary evidence).

But we don’t need a stateful concept for, say, a customer invoice. We need a snapshot of what it looked like before, and what we want it to look like next. This “next” result (which is just the output of a function), once confirmed (transacted in the database) becomes the next snapshot and you discard the intermediate concept of “state” entirely. Line item calculations and changes should be functions that operate per line on input data from that line. Invoice sums, averages, discounts, etc. should be functions concerned only with relevant input data as well, namely the aggregate result of each line item.

None of this involves a stateful requirement and shouldn’t involve stateful objects because that state is an unnecessary complication to the system. It also causes problematic architectural questions (is an invoice an object? is a line item an object? are they both objects? what do they inherit from? do they have a common ancestor? how do we make relational data sorta fit with our object model?) What are all these class declarations and the attendant statefulness actually doing for you? Nothing but permitting you to write yourself into a hole with mistaken side-effects (oops, that method wasn’t checking to clear the “discount” boolean in this object before it does self.foo()).

Even more interesting, sticking with the web example above and moving it forward to 2012 where everyone is mad for Django and Rails, these ORM-based frameworks almost never maintain persistent state objects between calls. When they do it often causes a peculiar type of unique-per-thread bug. So what is all this business about class/model declarations in the first place, since objects are first and foremost data structures with behaviors? These class declarations are trying very hard to be CREATE TABLE and ALTER TABLE SQL commands but without the benefit of actually being SQL commands — in other words, they are a weak form of data dictionary that sacrifice both the clean presentation of S-expressions or YAML trees and the full power of your favorite database’s native language.

Dealing with the database on its own terms and letting your functions be stateless makes it enormously easier to test your system because each function has a knowable output for each knowable input. It does mean that you must have a small constellation of functions to do the job, but you were going to write at least as many methods against your objects anyway (and often duplicate, or very nearly duplicate methods across non-sibling objects that do very nearly the same thing — or even worse, inherit several layers deep in a ripped fishing net pattern, which makes future changes to parent classes a horror). If you are dealing with data from a database trying to force your relational data into objects is poor thinking in most cases anyway, not least because none of that object architecture gets you one iota closer to accomplishing your task, despite all the beautiful design work that has to go into making relational data work with OOP. Unless you’re writing a simulation (as in, a MUD, an MMORPG or a flight simulator) there is no benefit here.

The easiest way to envision slices of time for people who got into programming later than 1994 or so is usually to recall the way that database-to-webpage functions worked in the CGI days as referenced above. They don’t maintain state, and if they do it is just as static variables which exist long enough to dump content into a webpage substitution template before being deallocated (and if written differently this wouldn’t always be necessary, either — the following are not the same: return (x + 1);, return x++;, x = x + 1; return x;). The only thing that matters to such functions is the input, not any pre-existing state. You can, of course, include cookies and SSL tokens and Kerberos tickets and whatnot in the mix — they merely constitute further input, not stateful persistence as far as the function itself is concerned.

There are some consequences to this for functional programs. For one thing, most loops wind up working best as recursively defined functions. In a imperative OOP language like Java this is horrible because each iteration requires instantiating a new object with its own state and therefore becomes a wild resource hog and executes really slow. In a functional language, however, tail-recursive functions often perform better than the alternative loop would. The reason for this is that (most) functional languages are optimized for tail-recursion — this means that a recursive function that calls itself as the last step in its evaluation doesn’t need to return state to its previous iterations; it is dependent only on its input. The machine can load the function one time and jump to its start on each iteration with a new value (whatever the arguments are) in the register without even needing to hit the cache, mess with iterator variables, check, evaluate or reform state, etc. (Note that this is possible in languages like C as well, but usually requires use of the “evil” goto statement. In higher level languages, however, this sort of thing is usually completely off limits.)

Let’s look at a stateful countdown program of the type you’d probably write on the first day of class (small caveat here: I’ve never had the privilege to attend a class, but I’m assuming this is the sort of thing that must go on the first day):

#include <stdio.h>

int main()
{
    int x = 10;

    while (x > 0)
    {
        printf("%d\n", x);
        x--;
    }

    printf("Blastoff!\n");
    return 0;
}

Of course, this could be done in a for() loop or whatever, but the principle is the same. Looping is king in C, and for a good reason. Here is an equivalent program in Guile (a Scheme interpreter that’s just one “yum install guile” away if you want to play with it):

(define (countdown x)
  (begin
    (display x)
    (newline)
    (if (> x 1)
      (countdown (1- x))
      (display "Blastoff!\n"))))

(countdown 10)

In this program there is no loop, there is a call to itself with an argument equal to the initial argument decremented by 1, but we are not actually dealing with variables at all here.

In fact, the expression (1- x) does not change the value of x at all, because it is merely an expression which returns “the result of ‘decrement x by 1′” (an equivalent would be (- x 1), which may or may not be equal in efficiency depending on the lisp environment) and not an assignment statement. (There are assignment statements in most functional languages, and they are primarily (ab)used by folks coming to functional languages from non-functional ones. Not saying you never want variables, but more often than not its more sensible to deal with expressions that say what you mean exactly once than stateful variables).

Being a toy example I’m using a “begin” statement to guarantee execution order in the interest of formatting text output. This sort of thing isn’t present in most functions, since whatever is required for the return value will be executed. Here we are generating an output side effect. We could eliminate the “begin” in the interest of being “more functional” while still emitting something to stdout as a side effect, but might be harder to read for someone coming from an imperative background:

(define (countdown x)
  (display (string-append (number->string x) "\n"))
  (if (> x 1)
    (countdown (1- x))
    (display "Blastoff!\n")))

If displaying an integer argument with a newline appended (or more generally, any argument value with a newline appended) was a common requirement (like for log files) the line beginning with display would become its own function, like display\n or something and be called more naturally as (display\n x).

This is a very simple example of how stateless recursion works in place of loops. You can implement loops in functional languages, which is usually a bad idea, or you can implement tail-recursive functions in most imperative languages, which is also usually a bad idea (especially when it involved objects, unless you start inlining assembler in your C with the appropriate jumps/gotos… which isn’t worth the trouble compared to a loop). Having demonstrated the equivalence of loops and recursion for most purposes, it bears mentioning that within the lisp community (and probably others) using tail-recursive functions in place of while()/for() loops is so common that very often when someone says “loop” what they mean is a recursive loop, not an iterative loop.

I say this to illustrate that when comparing languages its not about whether you can do something in language X or not, but whether a certain style of thinking matches the way the compiler or execution environment work. Step outside of the pocket that the language or runtime has built for you and you wind up very quickly in a nasty, panicky, twisty place where unmaintainable hacked up speed optimizations start feeling necessary. The futility of such speed optimizations is usually evidence that you’ve existed the pocket of your language, and is why old timers often are heard saying things like “if you need all those speed hacks, you need to reconsider your design overall” — its not because there is a speedier way to make that one method or function work, per se, but more typically what they are getting at is that you are thinking against the paradigms or assumptions that your language or environment was designed around.

In cases not involving simulation I find that carrying lots of state in variables, structs and the like complicates even simple things unnecessarily and makes testing confusing by comparison to stateless functions. OOP is even worse in this regard because you’ve got functionality intertwined with your state vehicles, and inheritance makes that situation even more convoluted. When dealing with lots of state stack traces can’t give you a clear picture of what happens every time with any given input, only what happened this time, because understanding what happened when you’re dealing with state is not as simple as looking at the arguments, you have to consider the state of the environment and objects within it at the moment of execution. This gets to the heart of why I recommend thinking of time as slices or points or snapshots instead of as continuous state, and is a very important idea to understand if you want to grok functional programming.

Racing to remove the last Nix

This post was prompted by a discussion on ScientificLinuxForum. The subject of this post diverts significantly from the original discussion, so I’ve placed it here instead. The thread was initially about the release of RHEL 6.3, but discussions there have a tendency to wander, particularly since many are worried we are in the last days of free computing with the advent of UEFI lock-down, DRM-Everything and new laws which prevent the digital equivalent of changing your own oil, but this post just doesn’t belong in the thread and may be of interest to a more general audience.

Unix philosophy is shifting. We can see it everywhere. Not too long ago on a Fedora development list an exchange equivalent to:

“I wanna do X, Y, and Z this new way I just made up and everyone says its bad. Why?”
“It breaks with everything Unix has done for 40 years that is known to work.”
“I don’t care what Unix has done. I want to make it work this way instead.”
“Its insecure.”
“ummm… oh…”
“Besides, it introduces binary settings so the user can’t adjust and fix them manually if the system goes to poo. So users can’t write scripts to change their settings without going through an API you’ve yet to even consider writing causing more work for everyone, and at the same time security is going to suffer. Besides, telling someone to reinstall their OS because one file got corrupted is not acceptable by our way of thinking.”
“uhhhhh… oooh?”

Let me be clear, there is the world of Unixy operating systems, there is the Unix design philosophy, and then there is the Unix religion. Part of the fun in a flame war is detailing how your opponent is a proponent of whatever part of the spectrum would most undermine their position at the time (usually the religion accusation is thrown, unless someone makes a dive straight to Nazism). The problem with dividing the world of grown-up operating systems into three stripes that way, though, is that it misses why a religion evolved in the first place.

Religion is all about belief, in particular a belief in what is safe and reliable. If I don’t offend God I’m more likely to get into Heaven — that’s safe and reliable. If I don’t give every process the ability to write arbitrarily then I’m less likely to have problems — that’s safe and reliable. Whatever God is up to I’m not really sure, he hasn’t let me in on it all, but that restrictions to write access prevent things like a rogue process (malicious, buggy or deliciously buggy) from DoS’ing the system by filling it up with garbage is something I can understand.

But not everyone can understand that, just like I can’t understand God. That’s why we have guidelines. er, religions. The fun part about the Unix religion is that its got a cultish flair, but the most functional part about it is that its effects can be measured and generally proved (heuristically or logically if not formally) to be better or worse for system performance and service provision.

It is good to question “why” and be a rebel every so often, but you’ve got to have a point to your asking and you’ve got to be prepared to hear things you may not have expected — like the response “Its insecure” which may be followed by an ego-demolishing demonstration. But people don’t like having their egos demolished and they certainly hate studying up on new-things-that-are-actually-old and yet still adore the question “why” because it sounds so revolutionary and forward-thinking.

But IT people are educated, right? They are good at dealing with detailed situations and evaluating courses of action before committing to this or that plan, right? Its all about design, right?

I’m here to tell you that we’ve got problems.

We are absorbing, over time, less talented and grossly inexperienced developers across all of techdom. It started with tossing C in favor of Java, and now even that in favor of Ruby in some places because its like “easier Java… and… hey, Rails!”. (This isn’t to say that Ruby is a bad language, but certainly that it shouldn’t be the only one you know or even the first one you learn.) Almost no universities treat hard math or electrical engineering courses as a prerequisite for computer science any more. In fact, the whole concept of putting hard classes first to wash out the stupid or unmotivated has nearly evaporated. This is not just in CS courses, but the dive has been particularly steep there. These days, as ironic as it may seem, the average programmer coming from school knows next to nothing about what is happening within the actual machine whereas a hobbyist or engineer coming from another field who is fascinated with machine computation understands quite a bit about such things.

Part of it probably has a lot to do with motivation. A large percentage of university students are on a conscious quest for paper, not knowledge, and want to get rich by copying what is now an old idea. That is, they all dream of building the next Facebook (sorry, can’t happen, Facebook will version up; at best you might get hired by them, loser). On the other hand every hobbyist or out-field engineer who spends personal time studying sticky problems in computer science on their own time is genuinely interested in the discipline itself.

It is interesting to me that most of my self-taught friends have either worked or are working through the MIT open coursework on SICP, K&R, Postgres source tours, and a variety of other fairly difficult beginner and advanced material (and remember their reference points remarkably well), while most of the CS graduates I know are more interested in just chasing whatever the latest web framework is and can’t explain what, say, the C preprocessor does. Neither group spends much time writing low-level code, but the self-educated group tends to have some understanding at that level and genuinely appreciates opportunities to learn more while many of the provably educated folks don’t know much, and don’t care to know much, about what is happening within their machines. (That said, I would relish the chance to go to back to school — but since I know I’ll never have the chance I’ve just got to read the best sources I can find and have my own insights.)

This has had a lot of different effects. In the past as a community we had a problem with the Not Invented Here syndrome (aka NIH — yes, its got its own acronym (and sometimes there are good reasons to make NIH a policy)) and sometimes deliberate reinventing of the wheel. Now we have the even worse problems of Never Heard of That Before and Let’s Jam Square Pegs Where They Don’t Belong (like, try to coerce the Web into being an applications development framework instead of being a document linking and publication service, for example).

A lot of concerns have been raised over the last few years about the direction that Unix has been headed in (or more specifically, a few very popular consumer-oriented distributions of Linux which represent the majority of Unix in desktop and tablet use today). There are issues ranging from attempts to move settings files from plain text to binary formats, efforts to make the desktop into one giant web page, efforts to make the system behave more Windows-like (give anyone the privileges to install whatever packages they want into unrestricted environments (protip: toy with the last two words here — there is a solution…)), and many other instances which scream of misinterpreting something that is near to someone’s experience (“easy”) as being less complex (“simple”). Some of these are just surface issues, others are not. But most grind against the Unix philosophy, and for good reason.

Most of these un-Unixy efforts come from the “new” class of developer. These are people who grew up on Windows and seem determined to emulate whatever they saw there, but within Unix. Often they think that the way to get a Unix to feel like Windows is to muck with the subsystems. Sometimes this is because they think that they know better, sometimes this is because they realize that the real solutions lie in making a better window manager but since that is hard subsystems are the easier route (and this feels more hackish), but most often it is simply because they don’t understand why things work they way they do and lack the experience to properly interpret what is in front of them. What results are thoughts like “Ah, I wish that as an unprivileged user I could install things via binary bundle installers, like off downloads.com in Windows, without remembering a stupid password or using some stupid package manager and get whatever I want. I can’t remember my password anyway because I have the desktop set to auto-login. That would put me in charge as a user!” Of course, they think this without ever realizing that this situation in Windows is what puts East European and Chinese government crackers in charge of Windows worldwide.

This gets down to the core of operating system maintenance, and any system administrator on any operating system knows that, but the newcomer who wants to implement this “feature” doesn’t. What they think is “Linux permissions are preventing me from doing that? Linux permissions must be wrong. Let’s just do away with that.” and they go on to write an “extension” which isn’t an extension at all, but rather a huge security flaw in the system. And they do it deliberately. When others say “that’s a bad idea” they say “prove it” and accusations of religious fundamentalism soon follow.

But there could have been a better solution here. For example, group permissions were invented just for this purpose. There is (still) a wheel group in every Linux I’ve seen. There’s even still  a sys group. But I’ve seen them actually used properly once or twice, ever — instead we have another triangular wheel which has been beaten round over the years called sudo and a whole octopus of dangly thingies called PAM and SE domains and… and… and… (do we really want one more?)

Anyway, {groups, [insert favorite permissions system]}  aren’t a perfect solution but they go a long way to doing things right in a simple manner without a lot of mucking about with subsystem changes. Back in the Old Days users had the same concerns, and these systems were thought out way back then. But people don’t go back and research this sort of thing. Learning old, good idea is hard. Not really to do, but to sit still and think long enough to understand is hard for a lot of people. There is a wealth of knowledge scattered throughout the man pages, info docs and about a bajillion websites, open source books, mailing list archives, newsgroup archives, design documents, formal treatments, O’Reilly books, etc. (what?!? books?!? How old fashioned! I’m not reading a word further!) but few people take the time to discover these resources, much less actually use them.

SELinux is another good idea someone had. But its not immediately obvious to newcomers so most folks just turn it off because that’s what someone else said to do. This is totally unnecessary but its what a lot of people do. It also gets very little development attention on Ubuntu, the most Windows-like Linux distro, because that distro has the highest percentage of uneducated ex-Windows users. You know what most exploits are written for? SELinux disabled Ubuntu boxes running a variety of closed-source software (Adobe products are pretty high on the list, but there are others) and unsecured web services (PHP + MySQL (i.e. hacked up Drupal installations) top the list, but to be fair they are the most prolific also). An example of the misconceptions rampant in the Ubuntu community is that running something in a chroot makes something “secure” because it is colloquially called a “chroot jail“. When told that chroot doesn’t really have anything to do with security and that a process can escape from a chroot environment if it wants to they get confused or, even funnier/sadder, want to argue. They can’t imagine that subsystems like mock depend on chroot for reasons other than security.

Why on earth would anyone disable a tool like SELinux if they are going to digitally whore their system out all over the internet by exposing the sensitive bits the way PHP programs do? Because they just don’t know. Before turning it off, no Apache screen. After turning it off, feathers! Before turning off SELinux and installing Flash no pr0nz on the screen just a black box that said something was broken on pornhub.com. After turning it off, tits! The immediate effect of turning it off is simple to understand; the long-term effect of turning it off is hard to understand; learning the system itself requires grokking a new concept and that’s hard. That’s why. And even better, the truly uninformed think that setenforce 0 is some slick haX0r trick because its on the command line… oooh.

So, simply put, pixels.

Pixels is the interest these days. Not performance, not sane subsystems, not security, not anything else. The the proper arrangement of pixels. Pixels can put tits on the screen, security subsystems and text configuration files can’t do that — at least, the connection between the two is impossible to manage for the average ex-Windows user.

The new users coming to Linux trying to Dozify it are doing so in the pure interest of pixels and nothing more. They don’t know much about information theory, relational data theory or any of the other things that people used to be compelled to study (“nuh uh! I learnt how to make Drupal show words on the screen, so I know about RDBMSs!”). Many mistake the information in a howto on a blog for systems knowledge, and most will never actually make the leap from knowledge to wisdom. They tinker with Linux but most of that tinkering doesn’t involve exploration as much as it involves trying to reshape it in the image of an OS they claim to be escaping. They can tinker with Linux because you just can, and you can’t with OS X or Windows.

You can make Linux your own. This is the right reason to get involved, whether your motivation is primarily pixels or whatever, any reason is a good reason to be interested in new development. But you can’t roll in assuming you know everything already.

And that’s the core problem. Folks show up in Linux land thinking they know everything, willing to break over 40 years of tremendous computing success and tradition. Some people even going so far as to arrive with prior intent to break things just for the social shock value. But ultimately its all in the interest of pixels.

But we don’t have to compromise the underlying OS and subsystems to get great desktop performance, run games, get wacky interactive features that aren’t available anywhere else, do multimedia (legally via Fluendo or via more natural means), or even just put tits on the screen. In fact all those things were possible (even easy) about a decade ago on Linux, but few people knew enough about the different components to integrate them effectively. What we need is developers who are educated enough about those separate systems to develop competently within and atop them without creating n00beriffic, monolithic junk designs that spread dependencies like cancer across the entire system.

The original triad of RPM, Yum and PackageKit was a great example of how to do it right — not perfect, but very nearly. They were linearly dependent, and the dependencies were exclusively top-down, accepting for necessary core system libraries/runtimes (the presence of Python, openssh and Bash, for example, is not an unreasonable expectation even on a pretty darn slim system).

But then someone comes along and wants to make PackageKit able to notify you with an audio alert when there is something worth updating — and instead of developing a modular, non-entangled extension that is linearly dependent on PackageKit, and not knowing well enough how to design such things nor willing to take the time to read PackageKit and grok it first, the developer decides to just “add a tiny feature to PackageKit” — which winds up making it grow what at first appears to be a single, tiny dependency: PulseAudio.

So now PackageKit depends on a whole slew of things via PulseAudio that the new feature developer didn’t realize, and over time those things grow circular dependencies which trace back to the feature in PackageKit which provided such a cute little audio notifier. This type of story gets even more fun when the system becomes so entangled that though each component comes from wildly differing projects no individual piece can be installed without all the others. At that point it matters not whether a dependency is officially up, down or sideways relative to any other piece, they all become indirectly dependent on everything else.

HAL got sort of like that, but not through external package dependencies — its dependencies got convoluted on the inside within its own code structure, which is just a different manifestation of the same brand of digital cancer. Actually, gcc is is need of some love to avoid the same fate, as is the Linux kernel itself (fortunately the corrosion of both gcc and the kernel is slower than HAL for pretty good reasons). This sort of decay is what prompts Microsoft to ditch their entire code base and start over every so often — they can’t bear to look at their own steaming pile after a while because it gets really, really hard and that means really, really expensive.

In the story about PackageKit above I’m compressing things a bit and audio alerts is not the way PackageKit got to be both such a tarbaby and grow so much hair at the same time (and it is still cleanly detachable from yum and everything below) — but it is a micro example of how this happens, and it happens everywhere that new developers write junk add-on features without realizing that they are junk. A different sort of problem crops up when people don’t realize that what they are writing isn’t the operating system but rather something that lives among it its various flora, and that it should do one thing well and that’s it.

For example I’m a huge fan of KDE — I think when configured properly it can be the ultimate desktop interface (and isn’t too shabby as a tablet one, either) — but there is no good reason that it should require execmem access. Firefox is the same way. So is Adobe Flash. None of these programs actually require access to protected memory — they can run whatever processes they need to within their own space without any issue — but they get written this way anyway and so this need is foisted on the system arbitrarily by a must-have application. Why? Because the folks writing them forgot that they aren’t writing the OS, they are writing an application that lives in a space provided by the OS, and they are being bad guests. Don’t even get me started on Chrome. (Some people read an agenda into why Flash and Chrome are the way they are — I don’t know about this, but the case is intriguing.)

Some distros are handling these changes better than others. The ones with strong guidelines like Fedora, Arch and Gentoo are faring best. The ones which are much further on the “do whatever” side are suffering a bit more in sanity. Unfortunately, though, over the last couple of years a few of the guidelines in Fedora have been changing — and sometimes not just changing a little because of votes, but changing because things like Firefox, systemd, PulseAudio, PackageKit, etc. are requiring such changes be made in order to function (they haven’t gone as far as reversing library bungling rules completely to let Chrome into the distro, but its a possibility).

To be polite, this is an interesting case of it being easier to re-write the manual than to fix the software. To be more blunt, this is a guideline reversal by fiat instead of vote. There is clear pressure from obviously well-monied quarters to push things like systemd, Gnome3, filesystem changes  and a bunch of other things that either break Fedora away from Linux or break Linux away from what Unices have always been. (To be fair, the filesystem changes are mostly an admission of how things work in practice and an opportunistic stab at cleaning up /etc. Some of the other changes are not so simply or innocently explained, however.)

This is problematic for a whole long list of technical reasons, but what is says about the business situation is a bit disconcerting: the people with the money are throwing it at people who don’t grok Unix. The worst part is that the breaking of Linux in an effort to commit such userland changes is completely unnecessary.

Aside from a very few hardware drivers, we could freeze the kernel at 2.6, freeze most of the subsystems, and focus on userland changes and produce a better result. We’re racing “forward” but I don’t see us in a fundamentally different place than we were about ten years ago on core system capabilities. This is a critical problem with a system like Windows, because customers pay through the nose for new versions that do exactly what the old stuff did. If you’re a business you have a responsibility to ask yourself what you can do today with your computers that you couldn’t do back in the 90’s. The idea here is that the OS isn’t what users are really interested in, they are interested in applications. Its harder to write cool applications without decent services being provided, but they are two distinctly different sets of functionality that do not have any business getting mixed together.

In fact, Linux has always been a cleanly layered cake and should stay that way. Linux userland lives atop all that subsystems goo. If we dig within the subsystem goo itself we find distinct layers there are well that have no business being intertwined. It is entirely possible to write a new window manager that does crazy, amazing things that were unimagined by anyone else before without touching a single line of kernel code, messing with the init system, or growing giant, sticky dependency tentacles everywhere. (Besides, any nerd knows what an abundance of tentacles leads to…)

The most alarming issue over the longer-term is that everyone is breaking Linux differently. If there was a roadmap I would understand. Sometimes its just time to say goodbye to whatever you cling to and get on the bus. But at the moment every project and every developer seems to be doing their own thing to an unprecedented degree. There has been some rumbling that a few things emanating from RH in the form of Fedora changes are deliberate breaks with Unix tradition and even the Linux standard, and that perhaps this is in an effort to deliberately engender incompatibility with other distros. That sounds silly in an open source world, but the truth of the business matter with infrastructure components is (and to be clear, platform equates to infrastructure today) that while you can’t lock out small competitors emerging or users doing what they want, without enormous funding no newcomer can make a dent in the direction of the open source ecosystem without very deep pockets.

Consider the cost of supporting just three good developers and their families for two years in a way that they feel comfortable about their career prospects after that two years. This is not a ton of money, but I don’t see a long line of people waiting to plop a few hundred thousand down on a new open source business idea until after its already been developed (the height of irony). There are a few thousand people willing to plop down a few million each on someone selling them the next already worn-out social networking scheme, though. This is because its easy to pitch a big glossy brochure of lies to suckers using buzzwords targeting an established market but difficult to pitch creation of a new market because that requires teaching a new idea; as noted above, people hate having to work to grasp new ideas.

Very few people can understand the business argument for keeping Linux as a Unixy system and how that can promote long-term stability while still achieving a distro that really can do it all — be the best server OS and maintain tight security by default while retaining the ever-critical ability to put tits on home user’s screens. Just as with developers where effort and time isn’t the problem but rather understanding, with investors the problem isn’t a lack of batteries but rather a lack of comprehension of the shape of the computing space.

Ultimately, there is no reason we have to pick between having a kickass server, a kickass desktop, a kickass tablet or a kickass phone OS, even within the same distro or family. Implementing a sound computing stack first and giving userland wizards something stable to work atop of is paramount. Breaking everything to pieces and trying to make, say, the network subsystem for “user” desktops work differently than servers or phones is beyond missing the point.

Recent business moves are reminiscent of the dark days of Unix in the 80’s and early 90’s. The lack of a direction and deliberate backbiting and sidedealing with organizations which were consciously hostile to the sector in the interest of short-term gain set back not just Unix, but serious computing on small systems for decades. This is, not to mention, that it guaranteed that the general population became acquainted with pretty shoddy systems and were wide open to deliberate miseducation about the role of computers in a work environment.

Its funny/scary to think that office workers spend more hours a day touching and interacting with computers than carpenters spend interacting with their tools, but understand their tools almost none at all whereas the carpenter holds a wealth of detailed knowledge about his field and the mechanics of it. And before you turn your pasty white, environmentally aware, vegan nose up at carpenters with the assumption that their work is simple or easy to learn, let me tell you from direct experience that it is not. “Well, a hammer is simpler than a computer and therefore easier to understand.” That is true about a hammer, but what about the job the carpenter is doing or his other tools, or more to the point, the way his various tools and skills interact to enable his job as a whole? Typing is pretty simple, too, but the scope of your job probably is not as simple as typing. Designing or even just building one house is a very complex task, and yet it is easier to find a carpenter competent at utilizing his tools to build a house than an office worker competent at utilizing his tools to build a solution within what is first and foremost an information management problem domain.

That construction crewmen with a few years on the job hold a larger store of technical knowledge to facilitate their trade than white-collar office workers with a few years on the job do to facilitate theirs is something that never seems to occur to people these days. When it does  it doesn’t occur to the average person something is seriously wrong with that situation. Nothing seems out of place whether the person perceiving this is an office worker, a carpenter or a guy working at a hot dog stand. We have just accepted as a global society that nobody other than “computer people” understands computing the same way that Medieval Europeans had just accepted that nobody other than nobility, scribes and priests could understand literacy.

It is frightening to me that a huge number of college educated developers seem to know less about how systems work than many Linux system administrators do unless we’re strictly walking Web frameworks. This equates to exactly zero durable knowledge since the current incarnation of the Web is built exclusively from flavor-of-the-week components. That’s all to the benefit of the few top players in IT and to the detriment of the user, if not actually according to a creepy plan somewhere. There probably was never a plan that was coherent and all thought up at once, of course, but things have clearly been pushed further in that direction by those in the industry who have caught on since the opportunity has presented itself. The “push” begins with encouraging shallow educational standards in fundamentally deep fields. Its sort of like like digital cancer farming.

Over in my little corner of the universe I’m trying hard to earn enough to push back against this trend, but my company is tiny at the moment and I’m sure I’ll never meet an investor (at least not until long after I really could use one). In fact, I doubt any exist who would really want to listen to a story about “infrastructure” because that admits that general computing is an appliance-like industry and not an explosive growth sector (well it is, but not in ways that are hyped just now). Besides, tech startups are soooo late 90’s.

Despite how “boring” keeping a stable system upon which to build cool stuff is, our customers love our services and they are willing to pay out the nose for custom solutions to real business problems — and this is SMBs who have never had the chance to get custom anything because they aren’t huge companies. Basically all the money that used to go to licensing now goes to things that actually save them money by reducing total human work time instead of merely relocating it from, say, a typewriter or word processor to a typewriter emulation program (like Writer or Word). This diversion of money from the same-old-crap to my company is great, but its slow going.

For starting from literally nothing (I left the Army not too long ago) this sounds like I’ve got one of those classic Good Things going.

But there is a problem looming. We’re spending all our time on custom development when we should be spending at least half of that time on cleaning up our upstream (Fedora, Vine and a smattering of specific upstream projects) to get that great benefit of having both awesome userland experiences and not squandering the last Nix left. If we can’t stick to a relatively sane computing stack a lot of things aren’t going to work out well over the long-term. Not that we or anyone else is doomed, but as a community we are certainly going to be spending a lot of time in digital hamster wheel fixing all the crap that the new generation of inexperienced developers is working overtime to break today.

As for my company, I’d like to hop off this ride. I know we’re going to have to change tack at some point because the general community is headed to stupid land as fast as it can go. The catch is, though, answering the question of whether or not I can generate enough gravity in-house to support a safe split or re-center around something different. Should I take over Vine by just hiring all the devs full-time? Revolve to HURD? Taking over Arch or Gentoo might be a bit much, but its got some smart folks who seem to grok Unix (and aren’t so big that they’ve grown the Ubuntu disease yet).  Or can I do what I really want to do: pour enough effort into sanifying Fedora and diversifying its dev community that I can use that as a direct upstream for SMB desktops without worry? (And I know this would benefit Red Hat directly, but who cares — they aren’t even looking at the market I’m in, so this doesn’t hurt anybody, least of all me. Actually, just for once we could generate the kind of synergistic relationship that open source promised in the first place. Whoa! Remember that idea?!?)

Things aren’t completely retarded yet, but they are getting that way. This is a problem deeper than a few distros getting wacky and attracting a disproportionate number of Windows refugees. It is evident in that I am having to cut my hiring requirement to “smart people who get shit done” — however I can get them. I have to train them completely in-house in the Dark Arts, usually by myself and via surrogate example, because there are simply no fresh graduates who know what I need them to know or think the way I need them to think. It is impossible to find qualified people from school these days. I’ve got a lot of work to do to make computing as sensible as it should be in 2012.

I might catch up to where I think we should be in 2012 by around 2030. Meh.

Gravity: Not what it does, what it causes

In addition to the LibreCAD thing, OS support stuff, etc. I’m also working on an ERP solution for my clients. This solution has an enormous number of obvious advantages over the way they are using software right now, but it requires me as an individual to understand how their business works better than any individual in that company does (or at least it seems that way after talking with all the different section leaders over there). My thinking about their problems and how to model them accurately in an ERP system leads me back to the problems that can be solved in my own company by a similar system, which leads me to the idea of generalization of functions and rules. This is, of course, the goal of good software design, but without spending some time reflecting on the nature of problems, the nature of data, and the nature of computing it is impossible to identify the parts that can be correctly be said to be general to all problems of a similar type, and what elements remain that make the specific problem at hand unique and identify it as that specific problem and not the same problem also found somewhere else.

This is, in a sense, what must be done when designing general functions, or correct object design, or deciding what utilities and handy tools should be considered to be “system utilities” and what other are just niche applications or personal tools. The concept of classification implies description, and at a certain level specifying a problem implies the ready resolution of the same problem (pretty neat!). But many times we get the identification of the problem wrong. More correctly, we inadequetely or incorrectly specify a problem and then develop whatever solution naturally follows from this mistaken version of the problem as we (wrongly) understand it.

As I was driving home in the rain today I was thinking about this — both the nature of the specific problems my ERP system needs to solve for the customer and the nature of problem classification itself. This led to a thought on how the precise, yet incorrect understanding of problems can lead to silly things like the widely misquoted statement “mathematics/physics states that bees can’t fly.” But quite clearly they do — which means neither mathematics nor physics is what says bees can’t fly, but rather an inaccurate mathematical model of flight predicts that bees can’t fly. But the misquote above is the more popular concept (its more fun, anyway, because it leaves the door open to magical thinking and the world of foolish mystery). The problem with this thinking is not just that it misleads people into thinking that math and physics don’t work — it also personifies math and physics (as in, creates the idea that “they” are beings of some sort who would attempt to prevent bees from flying as if the “can’t” in the misquote relates to the idea of permission) in a way that is weird and leads to more wrong thinking later. That idea led me down a mental journey into physics and I recalled an article I read recently about M-theory, gravity and General Relativity — and, specifically, the parts in the article that relate to the idea that gravity might be repulsive at great distances.

So… Gravity, at great distances, is it repulsive? Does this make sense? Or is there, perhaps, a misconception of the problem space here? There quite definitely is a miconception of the problem — that is evident in our inability to describe gravity in a mathematically consistent way that reconciles relativity with quantum physics. But what sort of misconception? I’m not part of the physics community, but from the way articles written for the layman put things (which is highly suspect) it seems as though people are personifying gravity a bit much. In other words, they are looking for what gravity “does” and from that trying to derive an accurate model of how gravity does that instead of thinking about what gravitiy “is” and then following the path of consequences to its existence.

The four basic forces (weak atomic, strong atomic, electromagnetic and gravity) are fairly well established. Interactions of things (space/matter/energy) those forces have to explain all phenomena — and so far pretty much do, which indicates that this is likely a correctish concept of the way things are. There doesn’t seem to be room for a fifth basic force, though there may be room for more things or types of things with which they might interact or ways they might interact (that is, unthought of dimensions, unobserved particles, etc, but not new forces themselves).

So… Gravity. It a sense it is a side effect of what happens when you concentrate mass in a single place. We say it “curves” space, though the way I tend to picture this in my mind is more of compression that bending, because bending can only happen to things that are truly unbounded, and space seems to be bounded by itself. The most common demonstration is to take a taught, suspended sheet and place something heavy on it, and then say “like this, it bends the surface down” and then the path of a marble on the sheet when rolled across tends towards the heavy thing. But this is a massive oversimplification.

If we take the suspended sheet as a 2D object then the downward direction that it bends to when something is placed on it represents a third dimension for that thing to bend “to” — hence it is bendable because it is unbounded in a new direction. The situation with space and gravity doesn’t seem to be the same because while we are fairly certain there are far more than 3 simple dimensions, we’re not being told to imagine that space itself bends in a 4th extra direction due to the influence of gravity/the presence of mass.

Another problem is the reason for the bending. Space is being directly influenced by the presence of matter via gravity, wheras the sheet is being influenced by something pressing on it. In other words, to get something to bend in an extra direction/new dimension it must be pushed, not contracted. So space under the influence of gravity behaves more the way that a wet cotton sheet contracts towards a spot that warm, dry air is applied to while the wet remainder stays lose and stretched out than the way that a sheet with something heavy on gets forced down in a single spot by the heavy thing.

And another problem with this sheet example is the rolling of the marble in an attempt to explain how things get drawn toward “gravity wells” in the same way the marble gets drawn to the lower points of the sheet. In the case of gravity the path of something under the influence of inertia is to continue moving in a straight line. But the straightness of that line is through space and gravity has contracted space into a smaller area than it normally would have been (or at least it appears so) and so the “straight” line is now curved relative to things that aren’t as local to the mass as the moving thing is. With the sheet example the path of the marble is actually longer than the original path, so this is a mis-example.

So this explanation and concepts derived from it are wrong. Now let’s return to the 2D sheet, because the number of dimensions really isn’t important here. If we were to draw a straight grid on it (or just a bunch of uniformly even or uniformly random dots), get it wet and then apply a hairdryer to a single part of it, we would start to see a subtle warping of the lines on the sheet, though over the whole sheet the size and general shape of things would remain the same. Now if we traced a line from one side to the other we would continue on that line just fine, but our path would bend toward the point we applied the hairdryer (interestingly, using a bounded space/area the path bends, but the medium itself does not, it just contracts in an area).

A more extreme example (and the one that came to mine while driving) was the shrink wrap we used to use when I was a teenager working at a video store. We would put items for sale into a polymer bag, and then blow hot air on the bag to make it shrink down. Being michievious kids we would sometimes experiment on down times with the stuff, and found that you could really make some weird things happen by blasting select spots of a large, flat sheet of the wrap material when spread out against the wall or floor. We were forcing local contractions on a self-bound 2D plane when we did this on material that was stretched out flat.

What does this have to do with gravity and localized attraction vs distant repulsion? Everything. If we blow hot air at points opposite one another on the same stretched out sheet the wrap material in between the two sheets get stretched tigher. Anything point that is closer to one point than another is pulled away from the center and toward the opposite point — relatively speaking this means that a point that is distant enough from one spot is traveling away from it. And this happens despite the fact that our actual influence on the sheet is constrictive in nature — all pull and no push. If space behaves in anything approaching this, then gravity can easily have a secondary effect of causing points beyond a certain distance from one another to grow further apart and yet not have any properties of repulsion at all. This increasing distance of points beyond a certain distance also does not require that the sheet continues to expand overall, either. That the universe itself likely is expanding just confuses the issue a bit more.

To a tiny observer on a whirling rock out in deep cold space this effect could definitely look forbiddingly like an unfriendly “push” effect to gravity. If that observer were prone to personify elements of existence (in other words, assign blame for effects on them) then it would be very natural to blame the observed possible increasing rate of expansion of the universe on a property of gravity rather than on an indirect effect or condition that it causes. One effect per force makes more sense than having a magical force that somehow exhibits one behavior in one place and yet another in another place.

Of course, the mental idea above that space doesn’t “bend” is going to probably bother people who carry with them a mental model of space as a bendy thing, and of blackholes as places where space “folds back on itself” when contraction is really the issue. The mental picture of a black hole just gets all screwy then — but this is probably more correct, not less. Anyway, with teeny tiny dimensions apparently being all over yet so small in scale, yet so universal because they represent entire dimensional planes that have been prevented from much direct interaction with our normal Euclidian(ish) selves, it seems likely that perhaps the folded up space stuff that makes up matter and energy might just be manifestations of tiny black holes compressed in directions that are just not a part of our 3 spacial dimensions, and all those black holes bubbling in and out of existence could have a lot to do with the basics of why/how all subatomics are unstable, yet predictably so. But that is a whole different discussion.

I am completely unqualified to make any statements about physics anyway, but these were the thoughts that went through my mind as I drove home in the rain. Unfortunately I’ll probably never have the time to really study physics, so the common crap written in the press for the layman (this includes most “science magazines” as well) are all I’ll likely ever get a chance to read and be mislead into dumb mental models like the ones above by.

Google’s Hearing and Insertions

Google’s CEO testified before Congress the other day during an antitrust hearing. The basic issue is whether Google is attempting to use its de facto monopoly on search to develop or even in some cases force a monopoly on other services which are not stated anywhere in their charter. The monpoly on earch is legal. Nobody was ever forced to use Google for searching, and until very recently there weren’t any decent alternatives anyway. Providing a great service and capturing a huge customer base is perfectly legal. The issue here is whether Google is using its search monopoly as a gateway to pitching its own services in other areas to generate monopolies over general data services and thereby extend its monopoly to everything.


[Google obviously does plug its own services as if they were search results -- and plugging the Chrome browser is one of the most important things the company could do to exert direct control over what information users see and use over the longer term.]

This would not be legal for a few reasons — one of which is that Google would be able to grant itself an unfair advantage. Hordes of unsavvy internet users who don’t know much about how computers or the internet work would never be able to find things without Google because very often in the minds of millions of lay users Google search equates to the gateway to the internet, and things they click on from the main Google search page are, in their mind, already linked to Google. So Google favoring its own services in search equates to users simply never learning about anything other than Google services. The problem with this is that as hordes of lay users gravitate to one or another online services the network effect comes into play, making which ever service takes an early lead overwhelmingly more important in the market than any other. Evidence of this is everywhere, and for good or ill, the fact is that most data service markets predict a monopoly almost out of necessity.

Google is obviously aware of this, and so are consumer protection groups. The creepy thing about this is that Google is not just offering search and online services, it is trying to offer everything online. Including all of your data. So under the Google model (actually, under all cloud models — which are all dangerously stupid) every bit of your computing data — personal photos, music files, blog posts, document files for work, document files for not work… everything — would be hosted on Google servers and saved on Google Inc.’s hard disks and nothing would be stored on your own disk. In other words, nothing would in any practical way be your own property because you won’t have any actual control over anything. And heaven forbid that an earthquake knocks your internet service out or anything else happens that disconnects you from the internet.

If one can’t see the danger here, one simply doesn’t have their thinking cap on. Anyway, this being a dangerously stupid way to handle your personal data is beside the point — the majority of internet users do not understand the issues well enough to know that its not a good idea to not manage their own data storage. But then again, most people don’t even recognize that their entire existence is merely a specific reordering of pre-existing matter, and therefore by definition simply a specific set of data. The information a person generates or intersects with in their life is the sum total of what they are — and this, of course, goes quite beyond being important somewhere on the web and as technology advances over the next few decades will increase in importance as the very nature of who and what we are increasingly mingles with automated data processes.

This is the real goal — extend monopoly to information as a general concept, and thereby generate a monopoly on modern existence (and I’m not simply talking about some ephemeral concept of what it is to be “modern” — in concrete terms we really are just masses of information). If there ever was a brilliant business plan, this is it. And it is a bit scary to think things might go that way. Google’s “Don’t be evil” theme is just words — as I have written elsewhere on this blog about how geopolitics works, power is about capability not about intent. Muslims may adhere to a religion based entirely on absolute social and political dominance of the planet, but being incapable of actually achieving it makes them a geopolitical nuisance over history instead of the driving force of history. On the other hand America’s intention is absolutely not to actually colonize and take over the world, but the fact that it is actually capable of doing this makes lots of people (even some Americans) panic and/or kick and scream about what they perceive as “American Imperialism” even though this is in no way the actual case.

So what about Google? That Google actually is developing the situation to make a drive at information monopoly is one thing. Their intent to not be evil is merely an intent. The capability expressed by a realized information monopoly would be of much more importance to the 1st world than even an American capability to successfully invade Skandinavia, for example, and is therefore something that should be guarded against.