The Intellectual Wilderness There is nothing more useless than doing efficiently that which should not be done at all.

2020.11.25 14:12

Erlang: FizzBuzz in Python vs Erlang — a discussion about conditionals

Filed under: Computing — Tags: , , , , , — zxq9 @ 14:12

I’ve had a few discussions with beginners over the last few months that often come to center around not really understanding the conditional branching constructs in Erlang, so I decided to do a video on it using “FizzBuzz” as the central example.

The traditional if/else if/else boolean paradigm is quite different from Erlang’s concept of matching and guards in case and function head matches and an if construct that is really more appropriate for range checks than general boolean branching. Hopefully I was able to explain at least a little bit about why Erlang’s matching and guards are a much more flexible method for writing conditionals once you grow accustomed to thinking in these terms.

2020.11.22 19:20

Erlang: Building a Telnet Chat Server from Scratch Using ZX

Filed under: Computing — Tags: , , , , , , , — zxq9 @ 19:20

A few weeks ago I made a two-part video discussion about building a telnet chat server from scratch using ZX and forgot to post any prose reference to it. (Most people are following my blog RSS, not my (extremely tiny) video channels.)

The resulting project is called “Trash Talk” and it has a repo on GitLab here. The license is MIT and is quite simple to hack around on, so have fun.

Part 1

The first video is a bit slower paced than the second. I cover:

  • What ZX templates when you do zx create project and select the “Traditional Erlang Application” project option
  • How everything fits together and works
  • Discussion about why things are structured the way they are in OTP applications
  • Demonstrate a little bit of command implementation in the telnet server (mostly to show where in the code you would do that).

This video is a lot less exciting than the second one because there aren’t a lot of jazzy features demonstrated, but it is probably the one you want to watch once now if you’re new to Erlang and a second time a few weeks later once you’ve written a few projects and stubbed your toes a few times (a second viewing brings different insights, not because the video changes but because you change through your experiences).

Part 2

The second video is the more fun one because the initial explanation that covers the critical question of “Why does anything do anything?” has already been covered in the first one, and while you might find the first video interesting, it isn’t as exciting as this second one where you get to see features that do stuff that might be more relevant to problems you have in your real-world projects get implemented. In this one I cover:

  • Checking the state of the running system using observer
  • The difference between zx and zxh when running your code in development
  • The “Service ⇒ Worker Pattern” (aka “SWP”) and how ZX can help you drop one in via zx template swp
  • One way to implement text commands in a structured way, with categories of commands indicated by prefixes (this seems elementary and very IRC-ish at first, but in command protocols a direct parallel of this often happens at the byte level — so think on that)
  • How to separate the “clients” concept as a service from the “channels” concept as a service, and discuss how that idea extends to much more complex systems
  • A bit more useful version of a “service manager” process in the form of the chan_man
  • And much much more! (advertising people from the 1980’s tell me I should always say that instead of “etc.”)

These aren’t super jazzy, but they get the job done. Hopefully they give some new Erlanger out there a leg-up on the problem of going from a “Hello, World!” escript to a proper OTP application.

2020.11.13 12:37

Comments on Dr. Shiva’s Election System Analysis

Filed under: Computing,Politics / Geopolitics,Society — Tags: , , , , — zxq9 @ 12:37

UPDATE 2020-11-20: Scroll to the bottom for a follow-up.

Commentary

Dr. Shiva performed a data analysis on the automated voting system results in the 2020 U.S. election and made a video presentation of it just a few days ago. I was asked to give my thoughts on it yesterday, so I watched it this morning and wrote some comments as I went along. The video link is below and my timestamped comments follow.

0:05:03
Votes are stored as “decimal fractions”? This would mean they are stored not as integer values, but as floating point values (unless there is a fixed-point library involved, but that seems extremely unlikely). This is insane. A vote can’t increment the tally by a fraction unless we’re trying to go back to the 3/5 rule or some other nonsense. Right off the bat this is pretty ridiculous.

0:07:12
“Provided SYSTEM cannot change the OUTPUT”.
A bit of elaboration… Of course the system changes the output (it generates the output), but given that the input has two parts, the ballot and the current count, vote(Ballot, Count) -> NewCount. the output value NewCount must be reproducible every time in a consistent way given the same initial inputs. Because we have an iterative counting system, this means the counts should be replayable transactions. The system must be able to fast-forward or rewind to any point in the sequence of transactions. As vote counting is distributed, this also means that each instance of a vote counter must record the sequence in which it counted its votes so that the aggregated total can be broken down and each part replayed, and that the aggregator also log and be capable of replaying each instance of aggregation of a new total update. His statement is correct, but it is useful to break down why it is both correct and also somewhat of an oversimplification.

0:08:33
Ballots are converted to ballot images (a digital representation of the ballot generated by the scanning process). The ballot images are being saved in some places and destroyed in others. This could be OK if the physical ballots are available for replay/recount and there was never any need to conduct a forensic analysis of the operation of the system, but this is not the case and there appears to be no standard for what to do about them. Deleting the ballot images prevents any possibility of comparing the counted ballot images and the original ballots, and this prevents a full inspection the system.

0:09:58
WAT
“Weighted votes?” There must be some explanation for this. Weighted voting is insane. It is supposed to be 1 vote 1 increment. What would be a legitimate explanation for having a weight system? This may be the reason why floats (of all insanity) are being used to store the vote tally.

0:11:03
The system DB is written in Access? Like MS Access? Is this accurate? Holy shit… No way that can be accurate. And yes, the numbers are indeed stored as floats (a double, specifically)

0:13:03
WAT

0:14:16
WAT

0:23:30
WAAAAT? This would mean that higher % Republican straight party vote correlates to lower individual candidate vote support. That also means he gets much higher individual vote support in Democrat straight party vote districts. This is extraordinarily unlikely given that he has 98% support across Republicans.

0:31:30
The phrase “taken from Trump and given to Biden”. I’m not seeing the basis for this yet. It is possible, however extremely unlikely, that voters actually voted this way. So I don’t agree with the language used in the discussion as a first-time watcher who does not know what else is coming up later in the talk, but the graph is clearly too structured to be true without some sound explanation of mechanism (and to be fair, the explanation that voters are more likely to dislike Trump the more likely they are to like Republicans is almost impossible to believe, but I’m keeping an open mind for the moment…).

0:32:20
Ah. They mention the same thing: It is “too structured” and “too perfect”. Because it is.

0:36:45
The idea that Trump did overwhelmingly better in Democrat districts than Republican districts is pretty hard to accept. The differences shown here are big enough for there to have some general revolt in the Republican base, which would have been very noticeable. This election Trump won the support of a large percentage of the original “Never Trumper” faction vote, a lot of the support from which manifested after the riots started. These vote tallies are grading on a curve really hard.

0:37:15
“You could even make the argument that even if you want to believe that Republicans hated Trump so so much, the larger that the population size was, they still wouldn’t be able to hate him in such a straight line.”

0:38:10
Wow. That looks a lot more like natural data. Though there is one really straight line hidden (downward) in there.

0:39:45
OK. Here is where he postulates the existence of an algorithm. He’s hinted about it up to this point, but this is where he comes out and says that it appears there is an algorithm that is applied to districts with a high percentage of Republican votes. That does appear to be the case from what is presented so far, though I would like to see some comparisons with districts of various alignment that were counted by hand and did not use these machines. The difference should be painfully obvious (in the same way the chart at 38:10 was striking).

0:42:00
The “Weighted Race” is a feature of the system that is documented? WTF? That’s insane. And yes, this looks very much like a weighted redistribution. The claim here is that all of the major vote counting system vendors have implemented this feature. Was that in the original contract? What was the motivation for its inclusion? Who came up with this nonsense? There must be some explanation for it to exist at all.

0:44:05
This is the core question: “Is it possible that this voting pattern is normal?” If it is then we should see it recur the same way with hand-counted votes. The slope, though, really does not make sense from the explanation that “Trump pissed off some % of Republicans” — the line would be flat, not a slope, and definitely not such a steep slope.

0:55:00
This is a pretty crazy story. Bad stuff.

0:57:58
YES. The idea of an auditable system is very important. That gets back to my original comments at the top about being able to replay each vote as an individual transaction, and each aggregate rollup event as an individual transaction, in addition to being able to independently verify that the ballot images match the paper ballots and that an alternative method of counting matches the results given by the automatic system over any random sample of ballots (hand counting results should match automated counts among random samples).

1:00:34
“Put a bunch of CPAs on this and they’ll tell you what the problems are.” Yes.

Followup

I was discussing with a mathematician about the best way to verify the above claims independently. His suggestion was that making an attempt to debunk the claims would provide the fastest path to verification: if we couldn’t debunk them then the claims are probably strong.

To debunk the claims (or alternately, verify them) would require performing an analysis on data from all districts in all counties in the state so we could compare histrionic data from each for anomalies. The reason a histrionic analysis is the most interesting has to do with the way a “weighted” vote algorithm would look over time compared to a natural count. To me the slopes that Dr. Shiva shows are less compelling (though interesting) than the histrionic plots that inspired him to perform an anlysis at all.

These two screenshots are interesting:

I don’t like that the second one is named only “Other Counties”. I have seen similar curves in other data, and it is indeed how weighted systems tend to plot unless the ballots are sorted before being counted. That is to say, a time sequence plot is interesting, a sorted count plot is not. There is not enough information here to know which one we are looking at though this is extremely likely to be a time series. We would need extremely complete data to evaluate this, though.

As for the slope that the next 40 minutes or so of Dr. Shiva’s video focuses on, that data is indeed quite interesting, but the average of the offset turns out to be more compelling than the fact that there is a slope at all.

In fact, another math channel on YouTube, “Stand-up Maths”, has performed a comparative analysis on the Biden data and found the same slope though its magnitude is different (and he never commented on that, nor did he comment on the time plot, which I thought was particularly weird to leave out since it is the original information that prompted the analysis in the first place).

It is an interesting video (though I don’t particularly care for the cheap ad hominem shots occasionally taken) and that channel has a ton of great math videos that have nothing whatsoever to do with politics (yay!) so go watch it at the link above if you are interested.

Unfortunately collecting data with the granularity necessary to perform the kind of analysis we want turns out to be possible, but quite time consuming. At the state level only aggregates are available, and those can’t tell us very much. Each county has the kind of data necessary to recreate the scatter plots (which turn out to be a bit less interesting than Dr. Shiva’s video make them out to be), but to the graphs of how the count tallies evolved over time is held by each district (and probably each county, but not available on the websites for each). The data is available publicly, but it is just very hard to get a hold of due to the sheer number of districts and counties in the state.

Conclusion

The details of the way the voting machine system is designed and implemented are clearly suspect. Verifying these details should be a priority, as should switching to an auditable and publicly visible system (hopefully something open source that lives on a public repo). The basic principles of its design and requirements seem antithetical to the way voting is supposed to work in the U.S.

As for the data analysis, there may be something weird in the data itself, but verifying it takes a lot of time just due to the level of detail required in the data (aggregates are useless to investigate the issues we really want to see), and whatever weirdness may exist doesn’t seem to have much to do with the bulk of both videos: the slopes shown.

2020.09.25 10:48

Excerpts from Chat: Discovering Structure and Grokking the True Nature of a Problem

This is an excerpt from a private chat with someone about code and I think it expresses something universal about programming, mathematics and engineering or any discipline where iterative development is necessary to manage complexity and discover simplicity.

…about a bit of code where I haven’t quite grasped what needs to happen, or the core essence of what problem the computer is actually solving for me (because this is sometimes distinct from the real world or abstract world where the problem really exists) — and so my code is convoluted and annoying and I can’t figure out how to untangle it without lots of extra complexity.

And then one day I make a change to the way the data is represented (it usually manifests with a data representation change, not always, but usually) and I refactor the code to accommodate that and… VOILA! It just sorts itself out and I find all these ways to simplify and it all makes so much sense.

That kind of code, especially if it is functional, is sort of annoying for people to read sometimes. They look at the code and see this elegant solution that tightly represents the program in a novel way and think “Ah, this is great. I should learn FP!” and then when they themselves tackle a difficult problem they don’t understand the nature of and start writing their own ball of mud they go back and look at that elegant, ideal example that inspired their current task and suddenly feel like a child looking up at the roof of the Sistine Chapel wondering how the hell Michelangelo managed to paint all that with a 12-meter-long paint brush.

They’ve never seen or imagined all the scaffolding that was once in place.

— Me, just now

I am quoting myself above, but it doesn’t really matter who wrote it. This expression came out well and I thought it was worth saving somewhere the world can see it (still following Joe Armstrong’s admonishment that I should publish and share too much and let search engines and individuals sort things out).

2020.09.24 11:02

Erlang: Dispatching Closures (aka: why we don’t do OOP)

Filed under: Computing — Tags: , , , , , — zxq9 @ 11:02

Oh my! It’s like “Creating nouns in the kingdom of verbs“!
(The link above is to a great post from Steve Yegge — read it and come back if you aren’t already familiar with it.)

A video talk-through of the code below and a bit of discussion

I wrote a funny little module today to demonstrate to a friend why FP folks sometimes refer to OOP objects as “dispatching closures” and occasionally make quips like “objects are a poor man’s closures”. I thought it would be fun to share it and also wanted a reference page to which I can refer people who have questions because this comes up every so often when a new OOP programmer starts pondering the nature of the universe after reading some introductory FP material.

Quite a lot of extra functionality could be added to this, such as distinguishing between functions that update the state of Self and functions that don’t to alleviate the annoyance of having to ignore the NewSelf return element of any “methods” beyond the pure built-in ‘get’, but we can leave that for another day as I really don’t expect anyone in their right mind would use the module below in real world code — it just introduces complexity, mental overhead, and potential weirdness if you pass an object between two different processes (send it to a process on another node and boom!), and in Erlang funs really shouldn’t be all that long-lived anyway.

In the below code a syntactic rewrite is needed to understand how to call the methods:

Pseudo PythonOOPsy Pseudo Erlang
class MyClass:
# stuff
MyClass = class(Data, Methods)
my_object.data_elementMyObject({get, DataElement})
my_object.data_element = valueMyObject({set, DataElement, Value})
my_object.do_something()MyObject({do, Something})
my_object.update(stuff).do_something(){ok, NextObject} = MyObject({do, update, Stuff}),
NextObject({do, something})

Here is the definition bit of the code, there is also a slightly more extensive and commented snippet on GitLab that has an additional demo/0 function that demonstrates one way dispatching closures can be called, manipulated, and extended through inheritance:

-module(oop).
-author("Craig Everett <zxq9@zxq9.com>").
-export([class/2, class/3, demo/0]).


-record(s,
        {data    = #{},
         methods = #{}}).


class(Defaults, Methods) ->
    fun
        (data) ->
            maps:keys(Defaults);
        (methods) ->
            maps:keys(Methods);
        (Data) when is_map(Data) ->
            State = #s{data = maps:merge(Defaults, Data), methods = Methods},
            object(State);
        ({subclass, NewDefaults, NewMethods}) ->
            class(maps:merge(Defaults, NewDefaults), maps:merge(Methods, NewMethods))
    end.


class(Inherits, Defaults, Methods) ->
    Inherits({subclass, Defaults, Methods}).


object(State = #s{data = Data, methods = Methods}) ->
    fun
        ({get, Label}) ->
            maps:get(Label, Data);
        ({set, Label, Value}) ->
            NewData = maps:put(Label, Value, Data),
            object(State#s{data = NewData});
        ({add, Label, Method}) ->
            NewMethods = maps:put(Label, Method, Methods),
            object(State#s{methods = NewMethods});
        ({do, Label}) ->
            F = maps:get(Label, Methods),
            F(object(State));
        ({do, Label, Args}) ->
            F = maps:get(Label, Methods),
            F(object(State), Args);
        (data) ->
            maps:keys(Data);
        (methods) ->
            maps:keys(Methods)
    end.

While this is an interesting construct, it is absolutely insane that anyone thought it was so utterly and all-encompassingly important that an entire new syntax should be developed just to hide the mechanics of it, and further, than entire languages should be created that enforce that this is the One True Way and impose it on the programmer. It’s cool, but not that cool.

2020.09.23 17:57

Erlang: [video] The GUI experience on Windows with ZX and Vapor

Filed under: Computing — Tags: , , , , , , , — zxq9 @ 17:57

I’ve written and spoken a bit about how ZX makes it easy to create, distribute and launch templated GUI projects in Erlang. I also showed how the (still ugly) GUI program launcher Vapor can make it easy for non-technical desktop users to use those programs.

So far I’ve been doing my examples on Linux where everything works in a familiar way and the terrain is fairly well known to us as developers, so in this video I want to show a bit about how things feel to Windows users who run client-side Erlang via Vapor using the standard Erlang runtime installation provided by Erlang Solutions and point out a few things that I find significant or can be improved in the experience.

If you have any insights into the issues highlighted in the video or have ideas about cross platform client development in general please let me know!

2020.09.22 14:23

Erlang: [Video redo!] Creating and running GUI apps with ZX

Filed under: Computing — Tags: , , , , , — zxq9 @ 14:23

I had a little bit of time to re-make a video on how to use ZX to create GUI applications. Hopefully the video demonstrates the basic use case well enough to make the purpose of the tool itself obvious.

Erlang: [ビデオ] ZXでGUIプログラムの作成と実行のしかた

Filed under: Computing,日本語 — Tags: , , , — zxq9 @ 14:22

2020.09.20 21:01

Erlang: [Video] Creating and running GUI apps with ZX

Filed under: Computing — Tags: , , , , , , , , — zxq9 @ 21:01

I had a little bit of time to make a video on how to use ZX to create GUI applications, but not enough time to do any post processing. Hopefully the video demonstrates the basic use case well enough to make the purpose of the tool itself obvious. (The audio isn’t great — hopefully I’ll have to either go back and dress that up a bit or make a better version of this video.)

2020.09.11 16:05

Hardware Development VS Software Development Budget Allocation

Filed under: Computing — Tags: , , , , , , — zxq9 @ 16:05

It is funny to me that hardware engineers are able to test the bejeezus out of essentially solved problems like how many keypresses a keyboard can take before mean operations to failure and document every minute aspect of their development, but software developers are pretty much prohibited from being given the time to document code.

In fact, it strikes me that hardware engineers’ actual job is to produce a specification from which hardware can be built, nearly always by people other than the engineers developing the product. That is to say, the product of the engineers’ labor is documentation.

The software developers, on the other hand, have the ability to document their product in coordination with its development, but are almost always forbidden from taking the time to do so, despite that documentation effort costing far less than the process hardware manufacturers have to go through to tool up a production line for a physical product.

This is a pretty insane state of affairs. If I could only show the public some of the chewed-to-crap, undocumented, wazoo code I’ve seen in production on closed source projects…

Older Posts »

Powered by WordPress