The Intellectual Wilderness There is nothing more useless than doing efficiently that which should not be done at all.

2015.04.10 10:48

XML: Xtensively Mucked-up Lists (or “How A Committee Screwed Up Sexps”)

Filed under: Computing — Tags: , , , , — zxq9 @ 10:48

XML sucks. I believe that it is, to date, the most perfectly horrible corruption of one of the most universal and simple concepts in computer science: sexps.

ZOMG! Someone screwed up sexps!

Let that one sink in. What a thing to say! How in the world would one even propose to screw up such a simple idea? Let’s consider an example…

Can you identify the semantic difference among the following examples?
(Inspired by the sample XML in the Python xml.etree docs)

Verson 1

<country name="Liechtenstein">
  <rank>1</rank>
  <year>2008</year>
  <gdppc>141100</gdppc>
  <neighbor name="Austria" direction="E"/>
  <neighbor name="Switzerland" direction="W"/>
</country>

Version 2

<country>
  <name>Liechtenstein</name>
  <rank>1</rank>
  <year>2008</year>
  <gdppc>141100</gdppc>
  <neighbor>
    <name>Austria</name>
    <direction>E</direction>
  </neighbor>
  <neighbor>
    <name>Switzerland</name>
    <direction>W</direction>
  <neighbor>
</country>

Version 3

<country name="Liechtenstein" rank="1" year="2008" gdppc="141100">
  <neighbor name="Austria" direction="E"/>
  <neighbor name="Switzerland" direction="W"/>
</country>

Version 4

And here there is a deliberate semantic difference, meant to be illustrative of a certain property of trees… which is supposedly the whole point.

<entries>
  <country rank="1" year="2008" gdppc="141100">
    <name>Liechtenstein</name>
    <neighbors>
      <name direction="E">Austria</name>
      <name direction="W">Switzerland</name>
    </neighbors>
  </country>
</entries>

Which one should you choose for your application? Which one is obvious to a parser? From which could you more than likely write a general parsing routine that could pull out data that meant something? Which one could you turn into a program by defining the identifier tags as functions somewhere?

Consider the last two questions carefully. The so-called “big data” people are hilarious, especially when they are XML people. There is a difference between “not a large enough sample to predict anything specific” and “a statistically significant sample from which generalities can be derived”, certainly, but that has a lot more to do with representative sample data (or rather, how representative the sample is) than the sheer number of petabytes you have sitting on disk somewhere. “Big Data” should really be about “Big Meaning”, but we seem to be so obsessed over the medium that we miss the message. Come to think of it, this is a problem that spans the techniverse — it just happens to be particularly obvious and damaging in the realm of data science.

The reason I so hate XML is because the complexity and ambiguity introduced in an effort to make the X in XML mean something has crippled it in terms of clarity. What is a data format if it confuses the semantics of the data? XML is unnecessarily ambiguous to the people who have to parse (or design, document, discuss, edit, etc.) XML schemas, and makes any hope of readily converting some generic data represented as XML into a program that can extract its meaning without going to the extra work of researching a schema — which throws the entire concept of “universality” right out the window.

Its all a lie. A tremendous amount of effort has been wasted over the years producing tools that do nothing more than automate away the mundane annoyances dealing with the stupid way in which the structure is serialized. These efforts have been trumpeted as a major triumph, and yet they don’t tell us anything about the resulting structure, which itself is still more ambiguous than plain old sexps would have been. Its not just that its a stupid angle-bracket notation when serialized (that’s annoying, but forgiveable: most sexps are annoying paren, obnoxious semantic whitespace, or confusing ant-poop delimited — there just is no escape from the tyranny of ASCII). XML structure is broken and ambiguous, no matter what representation it takes as characters in a file.

2014.11.12 21:23

Why OTP? Why “pure” and not “raw” Erlang?

I’ve been working on my little instructional project for the last few days and today finally got around to putting a very minimal, but working, chat system into the ErlMUD “scaffolding” code. (The commit with original comment is here. Note the date. By the time this post makes its way into Google things will probably be a lot different.)

I commented the commit on GitHub, but felt it was significant enough to reproduce here (lightly edited and linked). The state of the “raw Erlang” ErlMUD codebase as of this commit is significant because it clearly demonstrates the need for many Erlang community conventions, and even more significantly why OTP was written in the first place. Not only does it demonstrate the need for them, the non-trivial nature of the problem being handled has incidentally given rise to some very clear patterns which are already recognizable as proto-OTP usage patterns (without the important detail of having written any behaviors just yet). Here is the commit comment:

Originally chanman had been written to monitor, but not link or trap exits of channel processes [example]. At first glance this appears acceptable, after all the chanman doesn’t have any need to restart channels since they are supposed to die when they hit zero participants, and upon death the participant count winds up being zero.

But this assumes that the chanman itself will never die. This is always a faulty assumption. As a user it might be mildly inconvenient to suddenly be kicked from all channels, but it isn’t unusual for chat services to hiccup and it is easy to re-join whatever died. Resource exhaustion and an inconsistent channel registry is worse. If orphaned channels are left lying about the output of \list can never match reality, and identically named ones can be created in ways that don’t make sense. Even a trivial chat service with a tiny codebase like this can wind up with system partitions and inconsistent states (oh no!).

All channels crashing with the chanman might suck a little, but letting the server get to a corrupted state is unrecoverable without a restart. That requires taking the game and everything else down with it just because the chat service had a hiccup. This is totally unacceptable. Here we have one of the most important examples of why supervision trees matter: they create a direct chain of command, and enforce a no-orphan policy by annihilation. Notice that I have been writing “managers” not “supervisors” so far. This is to force me to (re)discover the utility of separating the concepts of process supervision and resource management (they are not the same thing, as we will see later).

Now that most of the “scaffolding” bits have been written in raw Erlang it is a good time to sit back and check out just how much repetitive code has been popping up all over the place. The repetitions aren’t resulting from some mandatory framework or environment boilerplate — I’m deliberately making an effort to write really “low level” Erlang, so low that there are no system or framework imposed patterns — they are resulting from the basic, natural fact that service workers form constellations of similarly defined processes and supervision trees provide one of the only known ways to guarantee fallback to a known state throughout the entire system without resorting to global restarts.

Another very important thing to notice is how inconsistent my off-the-cuff implementation of several of these patterns has been. Sometimes a loop has a single State variable that wraps the state of a service, sometimes bits are split out, sometimes it was one way to begin with and switched a few commits ago (especially once the argument list grew long enough to annoy me when typing). Some code_change/N functions have flipped back and forth along with this, and that required hand tweaking code that really could have been easier had every loop accepted a single wrapped State (or at least some standard structure that didn’t change every time I added something to the main loop without messing with code_change). Some places I start with a monitor and wind up with a link or vice versa, etc.

While the proper selection of OTP elements is more an art than a science in many cases, having commonly used components of a known utility already grouped together avoids the need for all this dancing about in code to figure out just what I want to do. I suppose the most damning point about all this is that none of the code I’ve been flip-flopping on has been essential to the actual problem I’m trying to solve. I didn’t set out to write a bunch of monitor or link or registry management code. The only message handler I care about is the one that sends a chat message to the right people. Very little of my code has been about solving that particular problem, and instead I consumed a few hours thinking through how I want the system to support itself, and spent very little time actually dealing with the problem I wanted to treat. Of course, writing this sort of thing without the help of any external libraries in any other language or environment I can think of would have been much more difficult, but the commit history today is a very strong case for making an effort to extract the common patterns used and isolate them from the actual problem solving bits.

The final thing to note is something I commented on a few commits ago, which is just how confusing tracing message passage can be when not using module interface functions. The send and receive locations are distant in the code, so checking for where things are sent from and where they are going to is a bit of a trick in the more complex cases (and fortunately none of this has been particularly complex, or I probably would have needed to write interface functions just to get anything done). One of the best things about using interface functions is the ability to glance at them for type information while working on other modules, use tools like Dialyzer (which we won’t get into we get into “pure Erlang” in v0.2), and easily grep or let Emacs or an IDE find calling sites for you. This is nearly impossible with pure ad hoc messaging. Ad hoc messaging is fine when writing a stub or two to test a concept, but anything beyond that starts getting very hard to keep track of, because the locations significant to the message protocol are both scattered about the code (seemingly at random) and can’t be defined by any typing tools.

I think this code proves three things:

  • Raw Erlang is amazingly quick for hacking things together that are more difficult to get right in other languages, even when writing the “robust” bits and scaffolding without the assistance of external libraries or applications. I wrote a robust chat system this afternoon that can be “hot” updated, from scratch, all by hand, with no framework code — that’s sort of amazing. But doing it sucked more than it needed to since I deliberately avoided adhering to most coding standards, but it was still possible and relatively quick. I wouldn’t want to have to maintain this two months from now, though — and that’s the real sticking point if you want to write production code.
  • Code convention recommendations from folks like Joe Armstrong (who actually does a good bit of by-hand, pure Erlang server writing — but is usually rather specific about how he does it), and standard set utilities like OTP exists for an obvious reason. Just look at the mess I’ve created!
  • Deployment clearly requires a better solution than this. We won’t touch on this issue for a while yet, but seriously, how in the hell would you automate deployment of a scattering of files like this?

2014.03.3 12:24

Source or Satire?

Filed under: Computing — Tags: , , , , , , , , , — zxq9 @ 12:24

From time to time I encounter openly discoverable code that is so wild in nature that I can’t help but wonder if the author was writing a machine function or a satirical statement.

Groovy source: ArrayUtil.java

After spending a few days plowing through Java code at the outset of a new Android project I found myself checking around for practical alternatives. In the course of that search (which netted Scala, Groovy and Clojure, in descending order of easy tooling for Android) I stumbled across this gem of a source file in the Groovy codebase. At first I couldn’t really tell if this was a joke about Java’s expressiveness or a functioning bit of code, but then I realized it is actually both — all the more funny because its expressing a cumbersome optimization that will execute on the same JVM either way:

ArrayUtil.java

Breach: A browser as practical satire

Someone from the Erlang world was kind enough to paste a link to Breach — a browser written in node.js. Its so full of meta fail and manifests the very essence of hipster circular logic that… I can only assume it is satire in the same vein as INTERCAL.

breach.cc

IBM SDK for Node.js on System Z

The going question at IBM has, for the last few decades at least, not been “Is it a good idea?” but “Are people deluded enough to pay for this?” This stands in heavy contrast to the countless bouts of genius that have peppered their research and development over the last century.

But IBM is a business, after all, and we’ve all got to eat. IBM was late recognizing that the majority of programmers and other IT professionals had left the world of engineering behind for the greener pastures of pop culture and fansterish tech propaganda, and to play catch up IBM had to innovate. Actually, this is a sort of business genius: IBM realized that tech doesn’t sell as well as bullshit and buzz when it watched Motorola get steamrolled by Intel’s marketing efforts around the original 286.

Intel pitched a bad chip design to tech illiterate execs, deliberately avoiding customers’ engineering departments, and prevailed against Motorola’s vastly superior designs — brilliant, if you don’t mind being a charlatan. Fortunately, Intel has at least occasionally made up for it since then (having been the only ones serious about SSD reliability early on, for example).

IBM has gone one further. I think this must have started as a practical joke at IBM, and then someone realized “Wait! Holy shit! This could be the hipster coup of the century and get us back in the web game!” Har har har. Joke’s on… all of us.

IBM SDK for Node.js v1.1 on System Z

“Process scope variables” in Erlang

I couldn’t have written a more concise satire-by-demonstration on what sucks about bringing the Java-style OOP thinking process as mental/emotional baggage when one starts using Erlang than this short message that appeared on the Erlang-Questions mailing list one day:

> So much time spent for removing one State variable from a few function calls.

much more time will be saved when refactoring from now on.

imagine:
most funs will now have signature:

-spec a(pid()) -> ok.

and most function bodies will look like:
check(Pid),
update(Pid),
return(Pid).

it will be a breeze now.

Like… whoa. That just happened. And it wasn’t sarcastic.

A gloriously sarcastic response can be found here.

2013.08.15 13:44

A Technically Inaccurate Explanation of Monads

Filed under: Computing — Tags: , , , , , — zxq9 @ 13:44

I’ve come across a multitude of technically inaccurate explanations of monads on the web, all intending to explain them in simple terms to FP newbies, most revolving around their use in Haskell. They all suck. This is mine. It also sucks, but I hope it sucks less.

[NOTE: I have no intent here to explain the mathematical concept of monads, the semantic gap between the way the term is used in set theory and in some flavors of functional programming, or any of that high-minded magic. My purpose is to explain what a monad is in terms of a computing utility to a functional programmer. This is a discussion about the sort of tool it is, not the theory behind that tool.]


Imagine using a purely functional language where side effects are verboten and we can’t do IO or anything else useful because of this. The host environment can clearly do IO just fine, otherwise our environment couldn’t do things like read memory locations (or our code) and our program would be useless. So it is obvious the host environment can do things externally that our program itself cannot do directly. This does not prevent us from describing what we wish it would do on behalf of our program, however.

In this way the host environment is sort of like a pantheon of gods and our programs are mere mortal blips of disembodied consciousness — capable of thought, desire and expression but powerless to touch the physical world in any way. All we can do is express our desires as regard the results of our thoughts and appeal to the gods to pity us and take action in accordance with our desires. Whether they take actions on our behalf is entirely up to them.*

With this in mind…
Imagine a function written in SomeLanguage:

int f(x): return x

It takes an integer and returns an integer. So what? The system doesn’t do anything with it so it is practically useless.

Now imagine a wrapper function:

tuple w(int x):
    tuple t = (f(x), "Please put this in STDOUT")
    return t

So w() has taken the output of f() and wrapped it in a tuple that contains the value returned by f() and a prayer to the system gods to “Please put this in STDOUT”. Pretty straight forward. Now the system gods can do whatever they want with it, but our language itself can’t — because talking to STDOUT would be a side effect and that’s strictly forbidden on Planet Functon. What we can do, however, is pray that the gods take mercy on us and act according to our wishes as regard the value that accompanies the prayer.

If this is the best model of a monad you can muster at the moment, then just stop here. This model is sufficient to make the concept of monads useful to you as you write programs that do useful work. (Which is the whole point. Unless you’re an academic, in which case your point is to write a lengthy article in a peer reviewed journal in which your explanation of monads is prefaced with a demonstration of your deep knowledge of set theory and type theory and esoteric discussion of things like monoids and thus make it appear that unfamiliarity with these subjects precludes the ability to use monads in a useful way in functional programs.)

But there are a few unresolved issues above, most of which I won’t address below, but the glaring issue of typing is relevant… (The other glaring issue of the undefined and imaginary syntax of SomeLanguage won’t be addressed at all.)

The type tuple is pretty ambiguous for this use (we could be talking about coordinates or relational values or immutable strings or whatever). How are the gods to know that we have a prayer for them? Should they inspect every tuple, or even every value we dream up, just checking for a “prayer” type? Of course not, they are busy doing immaculate things. We need a more clear signal.

One way to clear this up is to make a function that takes a function, its arguments and a request specification and returns a type prayer whose existence is itself a magical signal to get the attention of the gods. So ultimately a prayer is a type whose definition is a combined value composed of a thingy plus a prayer request regarding that thingy:

prayer p(function z, list l, request r):
    return (z(*l), r())

And then define an action of type request and rewrite w() in a way that returns a prayer:

request q = "Print to STDOUT"

prayer w(int x):
    return p(f, x, q)

Here we have defined a request for the system to perform, passed a reference/pointer/label to the function f() along with its arguments to the prayer maker function p(), which has executed f() along with its argument and returned a bundle of type prayer that contains the original integer return value along with a request to the system gods to do something regarding the bundled value.

Note that in Haskell it is common to define the underlying function (represented here by f()) and its monadic version (represented here as w()) separately. I suppose this isn’t strictly necessary — one could define monads in terms of lambdas — but this is the way things are done.

So we have specified a side effect, but we are only praying that the system takes an action based on our specification. Nothing more. Obviously I’m glossing over the details of how a request object itself is actually defined, but this is the general idea behind how monads are used to write programs in purely functional languages that have side effects.

An interesting consequence of this idea of a program specifying actions instead of actually taking them is that it is now trivial to write a host environment that deliberately ignores all prayers. In this way writing a host environment purely for the purpose of examining the behavior is also trivial. Its the ultimate sandboxing tool. It is equally easy to write a host environment that selectively observes requests, taking only some and ignoring all others, or ignoring some and taking all others.

I believe the abundance of confusing monad explanations to be a side effect of the Haskell community being disproportionately populated with mathematicians, type theorists and set theorists (“for real” or self-educated) and their explanations assume a knowledge base and vocabulary that is foreign and intimidating to most newcomers. The Clojure community doesn’t seem to have this problem, at least not as deeply.

[*It is easy to reverse this metaphor, of course, and imagine that we are uncorporeal gods, capable of thought and expression of our command to the slavish host to do whatever we specify. But I like the “program as mortals, host as gods” way better, because it extends to include the very real case of writing a test host that takes no action and merely observes (or even checks for the correctness of) the program’s behavior.]

2013.08.14 20:03

Binary Search: Random Windowing Over Large Sets

Filed under: Computing — Tags: , , , , — zxq9 @ 20:03

Yesterday I came across a blog post from 2010 that said less than 10% of programmers can write a binary search. At first I thought “ah, what nonsense” and then I realized I probably haven’t written one myself, at least not since BASIC and Pascal were what the cool kids were up to in the 80’s.

So, of course, I had a crack at it. There was an odd stipulation that made the challenge interesting — you couldn’t test the algorithm until you were confident it was correct. In other words, it had to work the first time.

I was wary of fencepost errors (perhaps being self-aware that spending more time in Python and Lisp(s) than C may have made me lazy with array indexes lately) so, on a whim, I decided to use a random window index to guarantee that I was in bounds each iteration. I also wrote it in a recursive style, because it just makes more sense to me that way.

Two things stuck out to me.

Though I was sure what I had written was an accurate representation of what I thought binary search was all about, I couldn’t actually recall ever seeing an implementation, having never taken a programming or algorithm class before (and still owning zero books on the subject, despite a personal resolution to remedy this last year…). So while I was confident that my algorithm would return the index of the target value, I wasn’t at all sure that I was implementing a “binary search” to the snob-standard.

The other thing that made me think twice was simply whether or not I would ever breach the recursion depth limit in Python on really huge sets. Obviously this is possible, but was it likely enough that it would occur in the span of a few thousand runs over large sets? Sometimes what seems statistically unlikely can pop up as a hamstring slicer in practical application. In particular, were the odds good that a random guess would lead the algorithm to follow a series of really bad guesses, and therefore occasionally blow up. On the other hand, were the odds better that random guesses would be occasionally so good that on average a random index is better than a halved one (of course, the target itself is always random, so how does this balance)?

I didn’t do any paperwork on this to figure out the probabilities, I just ran the code several thousand times and averaged the results — which were remarkably uniform.

binsearch

I split the process of assignment into two different procedures, one that narrows the window to be searched randomly, and another that does it by dividing by two. Then I made it iterate over ever larger random sets (converted to sorted lists) until I ran out of memory — turns out a list sort needs more than 6Gb at around 80,000,000 members or so.

I didn’t spend any time rewriting to clean things up to pursue larger lists (appending guaranteed larger members instead of sorting would probably permit astronomically huge lists to be searched within 6Gb of memory) but the results were pretty interesting when comparing the methods of binary search by window halving and binary search by random window narrowing. It turns out that halving is quite consistently better, but not by much, and the gap may possibly narrow at larger values (but I’m not going to write a super huge list generator to test this idea on just now).

It seems like something about these results are exploitable. But even if they were, the difference between iterating 24 instead of 34 times over a list of over 60,000,000 members to find a target item isn’t much difference in the grand scheme of things. That said, its mind boggling how not even close to Python’s recursion depth limit one will get, even when searching such a large list.

Here is the code (Python 2).

from __future__ import print_function
import random

def byhalf(r):
    return (r[0] + r[1]) / 2

def byrand(r):
    return random.randint(r[0], r[1])

def binsearch(t, l, r=None, z=0, op=byhalf):
    if r is None:
        r = (0, len(l) - 1)
    i = op(r)
    z += 1

    if t > l[i]:
        return binsearch(t, l, (i + 1, r[1]), z, op)
    elif t < l[i]:
        return binsearch(t, l, (r[0], i - 1), z, op)
    else:
        return z

def doit(z, x):
    l = list(set((int(z * random.random()) for i in xrange(x))))
    l.sort()

    res = {'half': [], 'rand': []}
    for i in range(1000):
        if x > 1:
            target = l[random.randrange(len(l) - 1)]
        elif x == 1:
            target = l[0]
        res['half'].append(binsearch(target, l, op=byhalf))
        res['rand'].append(binsearch(target, l, op=byrand))
    print('length: {0:>12} half:{1:>4} rand:{2:>4}'\
                    .format(len(l),
                            sum(res['half']) / len(res['half']),
                            sum(res['rand']) / len(res['rand'])))

for q in [2 ** x for x in range(27)]:
    doit(1000000000000, q)

Something just smells exploitable about these results, but I can’t put my finger on why just yet. And I don’t have time to think about it further. Anyway, it seems that the damage done by using a random index to make certain you stay within bounds doesn’t actually hurt performance as much as I thought it would. A perhaps useless discovery, but personally interesting nonetheless.

2013.08.13 15:58

Don’t Get Class Happy

Filed under: Computing — Tags: , , , , , , — zxq9 @ 15:58

If you find yourself writing a class and you can’t explain the difference between the class and an instance of that class, just stop. You should be writing a function.

This antipattern — needless classes everywhere, for everything — is driving me crazy. Lately I see it in Python and Ruby a good bit, where it really shouldn’t occur. I feel like its a mental contagion that has jumped species from Java to other languages.

In particular I see classes used as inter-module namespaces, which is odd since its not like there is a tendency to run out of meaningful names within a single source file (of reasonable length). Importing the module from outside can either import * from foo,or import foo, or from foo import filename, or even import foo as bar (ah ha!) and make flexible use of namespacing where it is needed most — in the immediate vicinity of use.

So don’t write useless, meaningless, fluffy, non-state&behavior classes. Any time you can avoid writing classes, just don’t do it. Write a function. Its good discipline to see how far you can get writing zero classes in an implementation, and then write it with some classes and see which is actually less complex/more readable. In any case writing two solutions will give you a chance to examine the problem from multiple angles, and that’s nearly always an effort that results in better code since it forces you to value grokness over hacking together a few lines that sort of cover the need for the moment (and forgetting to ever come back and do it right).

Or maybe I should be admonishing people to seek flatness in namespaces, since this implies not writing classes that are stateless containers of a bunch of function definitions which incidentally get referred to as “methods” even though they don’t belong to a meaningful object in the first place.

Or maybe I should be railing against the oxymoronic concept of “stateless objects/classes”.

Or maybe I should be screaming about keeping our programming vocabulary straight so that we can finally say what we mean and mean what we say. (Sound advice in all walks of life.)

This last idea is perhaps the most indirect yet powerful of all. But it implies “standards” and so far in my experience “standards” and “computing” don’t tend to turn out very well once we get beyond, say, TCP/IP or ANSI and Unicode.

2013.06.13 11:59

Development Speed VS Quality

Filed under: Computing,Science & Tech — Tags: , , , — zxq9 @ 11:59

I’ve been working under some pretty insane time constraints lately. Two things jump out at me upon review of my work:

  1. I can, when cornered, churn out thousands upon thousands of lines of functioning code in a flash.
  2. The code works, but it is not particularly insightful or brilliant — it merely works.

The first bit is sort of cool: Typo Monster and Syntax Bear are no longer a part of my life (at least not in my Big Five). Nice.

On the other hand, the second bit isn’t so nice. It means that insightful development requires far more time than people tend to imagine. This has been a point of discussion for decades in engineering and especially software development, but its hard to fully appreciate until you get a chance to compare your own code, written once under duress, with code written for a similar problem domain at a pace that allows more time for grokness.

Now that I think of it, its not the pace exactly which is the problem, it is the way that certain forms of deadline and environmental pressures can re-tune the time budget in a way that extends typing/coding/input time at the expense of grok time.

This is particularly pronounced when it comes to data modeling — no matter if this means in the sense of managed state, a relational database schema, a no-schema blobbulation, a serialization scheme of some sort, or an object/class model.

This is a particularly touchy thing, since rearranging a data model of any sort entails major surgery when more than one thing relies on the way data is stored, but refactoring functions is relatively lightweight (so long as you understand what the intended mapping was to begin with).

I suspect that the vast majority of production code is written under deadline and that most of it relies on expedient and trashy data models. I further suspect that most corporate settings urge programmers to “be programming” and officially establish that “programming” means “typing” instead of “understanding”.

It is shocking to revisit something written in a hurry and simplify it in a flash of insight, and then contemplate what just happened. The surprise is all the more resonant once you fully realize how much less time is required to input insightful code than rushed, verbose-but-functional trash.

Looking to my left and right, it is not particularly evident that much production code benefits from a culture of sacred Grok Time.

2012.11.12 22:12

Language Preachers: A Language’s Worst Enemy

Filed under: Computing — Tags: , , , , — zxq9 @ 22:12

Nothing crushes a language’s chance at becoming popular like the activities of its political and religious agitators. It doesn’t matter if the situation of the language is that its on the rise, on the decline or on the rebound. Language preachers suck, always. The only chance a language has for a successful PR campaign is when that campaign is based completely on lies, is extremely well funded, and appears to provide a remedy-by-aversion to the sort of problems incompetent instructors and consultants can’t think through and yet still encourages the sale of completely abstract content about abstractions without actually having to write useful code (Java). It doesn’t matter if this campaign is a complete farce, it’ll work if it permits enough of the stupid to perceive themselves as now smart enough because it can generate an entire sub-industry based on trying to prove that something can be done in this newly minted language for the incompetent — a heavy side of brand new buzz words helps, too (if only the “cloud” had been a language…).

But that’s just Java. Other languages don’t have the benefit of an academic revolution to assist in shoving a particular interpretation of a single programming paradigm down our throats (complete with a language that enforces just that one paradigm). Most languages just have to make it on their own either on merit (Haskell, a few lisps, Python), be the language of the year’s hottest killer app (PHP and the original web, Ruby and Rails), be the language of a killer app (Javascript and Mozilla/Firefox), or be both strong on merits and be the mother of a whole family of killer apps (C being the obvious example of the closest thing to modern immortality).

Recently I’ve been involved in a few discussions about the relative merits of a few different languages, database systems, distro flavors, kernel details, etc. Of all these sopics the top two likely start a religious war are choice of language and choice of database. My made-up estimate is that language discussions are twice as likely to incite digital jihad than database discussions.

Maybe it is because we spend most of our time dealing directly in a language and actually thinking in that language, so it is internalized to a level nothing else is. I don’t “think” in LibreOffice (though I’ve seen a few people who can think in Calc functions and others in Excel functions — and I hope the two types never meet to talk about it), I don’t carry on an internal monologue of sorts in Firefox or Nautilus or Gnome — but I do talk to myself in a sort of visual way in Python, Guile, Haskell, C, Bash, Perl, plpgsql, opcodes and a few other languages when the need arises. Programmers are prone to do this — and it turns out that they are most likely to do it only in a single language, ever (pick one of Java, Perl, Javascript, PHP, or Ruby for the best religious wars).

To someone who doesn’t speak enough languages to have evolved a sort of quiet disdain for them all a weird form of moral relativism seems to pervade. All languages other than the one they already know well are at once “the best for some job, just depends on what you’re doing” and completely inferior at everything compared to $the_only_lang_i_know_well. And of course the complete n00b who doesn’t even know one language well yet will swear that LangX — which he will never take the time to study but will take the time to flood IRC with airy questions about — is the cure for cancer, war and was handed down by Mayan priests without actually knowing anything about it. Obviously there is a conflict here.

Most languages suck, and some suck far less than others. A very few suck so much less that they are almost good. That list is very short. The list of languages I think to myself in regularly is a mix of languages that I don’t completely despise and ones that I do despise but must use because useful projects are already written in them. It used to grind on me that I had to use inferior tools so often, but I’ve gotten past by accepting that most programmers are either really bad at what they do or are compelled by economic circumstances to abandon both their code and the quest for decent tools prematurely. Meh.

The last few years I’ve noticed a lot of Perl people getting zealous about their language. I’m not writing to bash Perl or its community — different communities go through their own inner social cycles as well. But the last six years or so has seen Perl people transition from a group that still self-assuredly represented the original web/CGI hackers and were held in guruly esteem to a group that is panicked about the low percentage of new projects written in Perl compared to the late 90’s and early 00’s. They’ve gone from a group that is solving problems to a group that is actively trying to tell everyone that Perl is still the thing to use, that its still hip and cool and better than everything else out there.

This is hurting Perl more than it is helping. Perl is indeed a good language, but the days when it was the de facto standard for a large subset of new project types is over. Panicking about it won’t help — but writing useful programs sure might. Today there are many alternatives that are at least as good as Perl ever was and many that are just better. Algol was and still is a great language, but I’m not about to start a new project in it. C is just better, and so far hasn’t been surpassed in that particular space (though its time may come, too). I make the decision to not use Algol out of an informed regard for the difficulty of maintaining Algol code in 2012 versus maintaining other code in 2012. If someone were constantly bombarding me with pro-Algol propaganda and biased trash benchmark comparisons then I would avoid Algol on principle instead of even learning enough about it to know whether or not it could have possibly been a good choice to begin with.

If Perl is the awzumest evar, then there is no need to proselytize. If it is on the decline, then there is a huge need to defend the temple — and this is the perception I have of this sort of behavior. It makes Perl people sound like Java people when it was a crap language trying to gain some mindshare — though in defense of that tactic Java did take over the shallow end of the pool, and that turned out to be the largest part of the pool by a vast margin.

The the Lisp community screwed itself by engaging in Java-style cheerleading for a time. This drove a wedge between the people who really could have benefitted from learning it and the community of language snobs who were already privy to The Great Mystery. The Postgres community was that way as well until very recently. I’m a card carrying lisper myself, but I admit the community just got retarded when it came to interacting with others. The only thing that has redeemed Lisp in recent years is that some of the best commercial Lisp developers also happen to be excellent writers. If it weren’t for that there would have been no revival because the inner community mixed with the outer community like water and oil over this language preaching business.

Perl is a genuinely good language and it has a genuinely good runtime (and the two are totally different issues). If it is fading, let it fade with majesty as one of the traditional uber-hacker languages of yore, bringing out your Perl magic as a way to intrigue and enlighten n00bs instead of making the word “Perl” synonymous with a noisy cult that is more interested in evangelizing than writing solutions to problems.

Don’t make the same mistake the Lispers did when they felt their language was threatened — its the difference between newcomers deliberately avoiding Perl instead of not having tried it yet.

2012.10.11 23:09

Fedora: A Study in Featuritis

Its a creeping featurism! No, its a feeping creaturism! No, its an infestation of Feature Faeries! No, its Fedora!

I’ve been passively watching this thread (link to thread list) on the Fedora development list and I just can’t take anymore. I can’t bring myself to add to the symphony, either, because it won’t do any good — people with big money have already funded people with big egos to push forward with the castration of Fedora, come what may. So I’m writing a blog post way out here in the wilds of the unread part of the internet instead, mostly to satisfy my own urge to scream. Even if alone in the woods. Into a pillow. Inside a soundproof vault.

I already wrote an article about the current efforts to neuter Unix, so I won’t completely rehash all of that here. But its worth noting that the post about de-Nixing *nix generated a lot more support than hatred. When I write about political topics I usually get more hate mail than support, so this was unique. “But Unix isn’t politics” you might naively say — but let’s face it, the effort to completely re-shape Unix is nothing but politics; there is very little genuinely new or novel tech going on there (assloads of agitation, no change in temperature). In fact, that has ever been the Unix Paradox — that most major developments are political, not technical in nature.

As an example, in a response to the thread linked above, Konstantin Ryabitsev said:

So, in other words, all our existing log analysis tools have to be modified if they are to be of any use in Fedora 18?

In a word, yes. But what is really happening is that we will have to replace all existing *nix admins or at a minimum replace all of their training and habits. Most of the major movement within Fedora from about a year ago is an attempt to un-nix everything about Linux as we know it, and especially as we knew it as a descendant in the Unix tradition. If things keep going the way they are OS X might wind up being more “traditional” than Fedora in short order (never thought I’d write that sentence — so that’s why they say “never say ‘never'”).

Log files won’t even be really plain text anymore? And not “just” HTML, either, but almost definitely some new illegible form of XML by the time this is over — after all, the tendency toward laughably obfuscated XML is almost impossible to resist once angle brackets have made their way into any format for any reason. Apparently having log files sorted in Postgres wasn’t good enough.

How well will this sit with embedded systems, existing utilities, or better, embedded admins? It won’t, and they aren’t all going to get remade. Can you imagine hearing phrases like this and not being disgusted/amused/amazed: “Wait, let me fire up a browser to check what happened in the router board that only has a serial terminal connection can’t find its network devices”; or even better, “Let me fire up a browser to check what happened in this engine’s piston timing module”?

Unless Fedora derived systems completely take over all server and mobile spaces (and hence gain the “foist on the public by fiat” advantage Windows has enjoyed in spite of itself) this evolutionary branch is going to become marginalized and dumped by the community because the whole advantage of being a *nix admin was that you didn’t have to retrain everything every release like with Windows — now that’s out the window (oops, bad pun).

There was a time when you could pretty well know what knowledge was going to be eternal (and probably be universal across systems, or nearly so) and what knowledge was going to change a bit per release. That was always one of the biggest cultural differences between Unix and everything else. But those days are gone, at least within Fedoraland.

The original goals for systemd (at least the ones that allegedly sold FESCO on it) were to permit parallel service boot (biggest point of noise by the lead developer initially, with a special subset of this noise focused around the idea of Fedora “going mobile” (advanced sleep-states VS insta-boot, etc.)) and sane descendant process tracking (second most noise and a solid idea), with a little “easy to multi-seat” on the side to pacify everyone else (though I’ve seen about zero evidence of this actually getting anywhere yet). Now systemd goals and features have grown to cover everything to include logging. The response from the systemd team would likely be”but how can it not include logging?!?” Of course, that sort of reasoning is how you get monolithic chunk projects that spread like cancer. Its ironic to me that when systemd was introduced HAL was held up as such a perfect example of what not to do when writing a sub-system specifically because it became such an octopus — but at least HAL stayed within its govern-device-thingies bounds. I have no idea where the zone of responsibility for systemd starts and the kernel or userland begins anymore. That’s quite an achievement.

And there has been no end to resistance to systemd, and not just because of init script changeover and breakages. There have been endless disputes about the philosophy underlying its basic design. But don’t let that stop anybody and make them think. Not so dissimilar to the Gnome3/Unity flop.

I no longer see a future where this distro and its commercially important derivative is the juggernaut in Linux IT — particularly since it really won’t be Linux as we understand it, it will be some other operating system running atop the same kernel.

Come to think of it, changing the kernel would go over better than making all these service and subsystem changes — because administrators and users would at least still know what was going on for the most part and with a change in kernel the type of things that likely would be different (services) would be expected and even well-received if they represented clear improvements over whatever had preceded them.

Consider how similar administering Debian/Hurd is to administering Debian/Linux, or Arch/Hurd is to administering Arch/Linux. And how similar AIX and HP/UX are to administering, say, RHEL 6. We’re making such invasive changes through systemd that a change of kernel from a monolothic to a microkernel is actually more sensible — after all, most of the “wrangle services atop a kernel a new way” ideas are already managed a more robust way as part of the kernel design, not as an intermediate wonder-how-it’ll-work-this-week subsystem.

Maybe that is simpler. But it doesn’t matter, because this is about deliberately divisive techno politicking on one side (in the vain hope that “if our wacko system dominates the market, we’ll own the training market by default even if Scientific Linux and CentOS still dominate in raw numbers!”), and ego masturbation on the other (“I’ll be such a rebel if I shake up the Unix community by repeatedly deriding so-called ‘Unix traditions‘ as outdated superstitions and generally giving the Unix community the bird!”) on the other.

Here’s a guide to predicting the most likely outcomes:

  • To read the future history* of how these efforts work out as a business tactic, check the history of Unix from the mid-1980’s to early 2000’s and see how well “diversification” in the interest of carving out corporate empires works. I find it strikingly suitable that political abuse of language has found its way into this effort — conscious efforts at diversification (defined as branching away from every other existing system, even your own previous releases) is always performed under the label of “standardization” or “conformance to existing influences and trends”. Har har. Joke’s on you, though, Fedora. (*Yeah, its already written, so you can just read this one. Easy.)
  • To predict the future history of a snubbed Unix community, consider that the Unix community is so used to getting flipped the bird by commercial interests that lose their way that it banded together to write Linux and the entire GNU constellation from scratch. Consider also that the original UNIX was started by developers who were snubbed and felt ill at ease with another, related system whose principal flaw was (ironically) none other than the same featuritis the Linux community is now enduring.

I don’t see any future where Fedora succeeds in any of its logarithmically expanding goals as driven by Red Hat. And with that, I don’t see a bright future for Red Hat beyond v7 if they don’t get this and other priorities sorted**. As a developer who wishes for the love of everything holy that I could just focus on developing consumer business applications, I’m honestly sad to say that I’m having to look for a new “main platform” to develop for, because this goose looks about cooked.

** (sound still doesn’t work reliably — Ekiga is broken out of the box, Skype is owned by Microsoft now — Fedora/Red Hat don’t have a prayer at getting on mobile (miracles aside) — nobody is working on anything solid to stand a business on once the “cloud” dream bubble pops — virtualization is already way overinvested in and done better elsewhere already anyway — easy-to-fix media issues aren’t being looked at — a new init system makes everything above worse, not better, and is distracting and requires admins to completely retrain besides…)

2012.08.27 11:55

Lean on Your Database

Filed under: Computing — Tags: , , , — zxq9 @ 11:55

Doing computations in the database is almost always the right answer. Writing your own database procedures instead of relying on an ORM framework is even better. These go hand in hand since ORMs don’t allow you enough control over your data schema to define calculations in the first place (most can’t even properly handle multi-column primary keys and instead invent meaningless integer ID for everything; this should tell you something). Many, maybe most (at least that I’ve met), web developers don’t even know what sort of calculations are available to be performed in the database because they’ve been taught, in an absolute vacuum of personal experience, that “SQL hard. Relational thinking hard. OOP good. Trust framework”. There is so much missing here.

Frameworks try to be “database agnostic”. This is fundamentally flawed thinking. This implies that the data layer is merely there for “persistence” and that the “persistence layer” can be whatever — all RDBMS systems “aren’t OO and therefore suck” and so any old database will do. If this is true then it follows that frameworks and applications should be designed to work against any database system and not delve too deeply into any specific feature sets — after all, the application functionality is the focus, not the data, right? This is exactly backwards.

Even forgetting that this condemns you to least-common-denominator data design, this is still exactly backwards. Let me put the right way on a line all by itself, because it just that important:

Data designs should strive to be application agnostic.

Data drives everything. Your functions are what you can change around easily, but your data schema is critical and represents everything about your system logic. If you show me a well-labeled data schema I can probably guess what you are trying to do, but if you show me just your functions and objects I’ll require either a code tour or a lot of familiarization time before getting anything serious done (that project documentation will be lacking is a truism not worth addressing here).

Consider that changing your app code is cake whereas changing your data schema is major project surgery. OOP has us so in a stupor that we think if we just get our objects right everything will be fine. It won’t. Ever. As long as you think that you’ll also believe other crap like that each object should map directly to a table. There are certain basic truths about certain types of data. It is striking that I can give a data requirement to two DBAs schooled on two different RDBMSes and ask for a normalized data model (let’s just say NF3 for argument) and get back two very similar looking schemas, but I can give a feature requirement to two Java programmers and get back radically different system designs.

This should tell us something. In fact, it screams the truth that data is a foundation from which you must work up toward the application code, not the other way around. The database layer is the most important place to make sound choices. The choice of database system itself should be based on project requirements, because that choice matters. Most critically, I’ll say it again here because it is so important and implies so much on contemplation: the database designs should strive to be application agnostic.

« Newer PostsOlder Posts »

Powered by WordPress