I notice that there are a few little helper functions I seem to always wind up writing given different contexts. In Erlang one of these is an inverse function for file:consult/1, which I have to write any time I use a text file to store config data*.
Very simply:
write_terms(Filename, List) ->
Format = fun(Term) -> io_lib:format("~tp.~n", [Term]) end,
Text = unicode:characters_to_binary(lists:map(Format, List)),
file:write_file(Filename, Text).
[NOTE 1: This should return the atom 'ok' — and if you want to check and assertion or crash on failure, you want to do ok = write_terms(Filename, List) in your code.]
[NOTE 2: The Erlang runtime will need to be started with the +pc unicode flag if you want to print UTF-8 characters in a string as characters instead of printing a list of their codepoints into a file.]
This separates each term in a list by a period in the text file, which causes file:consult/1 to return the same list back (in order — though this detail usually does not matter because most conf files are used as proplists and are keysearched anyway).
An annoyance with most APIs is a lack of inverse functions where they could easily be written. Even if the original authors of the library don’t conceive of a use for an inverse of some particular function, whenever there is an opportunity for this leaving it out just makes an API feel incomplete (and don’t get me started on “web APIs”… ugh). This is just one case of that. Why does Erlang have a file:consult/1 but not a file:write_terms/2 (or “file:deconsult/2” or whatever)? I don’t know. But this bugs me in most libs in most languages — this is the way I usually deal with this particular situation in Erlang.
[* term_to_binary/1 ⇔ binary_to_term/1 is not an acceptable solution for config data!]
You can use lists:map/2 instead of lists:foldl/3 + lists:reverse/1, for example
write_terms(Filename, List) ->
file:write_file(Filename, lists:map(fun(Term) -> io_lib:format("~tp.~n", [Term]) end, List).
@Jihyun Yu
Indeed! The most recent time I was doing this I had been doing some other things, and just cut the other code out because it worked. Then without thinking I had just reversed the list to make the list retain its order — totally overlooking the fact that I was reimplementing map in the process.
Just had a good laugh at myself. (^.^)
> [* term_to_binary/1 ↔ binary_to_term/1 is not an acceptable solution for config data!]
Why?
Because the binary data on disk is not something that an administrator, user or anyone else can edit if your system goes to shit. Its not scriptable with non-Erlang tools… [insert 40-year-old arguments against non-text settings files] and so on.
Settings should be something that a user who knows nothing about Erlang should be able to modify. You strip them of that when you start putting settings in binary files that only an Erlang program can read. Considering the ease of simply not putting settings in binary files, and that it gains you nothing, there is no reason to contemplate this.
This is exactly what I was looking for! Thanks!
I was considering using term_to_binary() and binary_to_term(), before I found your post.
I do wonder about that name. I just can’t connect “consult” with reading terms from a file. Oh well as long as it works. Thanks again!
The wording is indeed silly.
file:consult(File)doesn’t feel as natural asfile:read_terms(File)andfile:write_terms(File, Terms). I get the effect they were going for, but what is the reverse operation offile:consult(File)? I have no idea — it paints you into a corner semantically.I’m finally working on breaking out a few chunks of the utilities I’ve written into some other projects over the last year as their own library applications — and this pair of functions will be in there alongside my wxErlang meta-widget wrappers, string translation library and other odds and ends.
term_to_binary/1<->binary_to_term/1is a really great way of sidestepping the establishment of some arbitrary (and almost always lossy) serialization across the network. It lets you easily establish multi-cluster constellations of services composed of arbitrary node-sized clusters. That’s great any place that disterl doesn’t fit (especially when you don’t control all the nodes, due to the implicit trust among nodes in a cluster — you can’t write client-server software as peer nodes, for example) but it just doesn’t quite cut it for serializing data to disk in many cases (settings, options, config, history cache, logging, etc.).Note that write_file’s default encoding is latin1, so it will mangle any non-latin1 utf8, as I just found out.
Yes, indeed. There is a utf8 version I actually use in several projects that I should locate and paste here in addition to the above example. Thank you for pointing this out!
Thank you. I was looking for this one indeed.
Thank you for publishing this snippet of code. The function works well! I’ve copied it to my script.
You raised a valid concern about the absence of the opposite function in the standard library, so I did a little investigation.
The function file:consult/1 is written in Erlang (not C) here:
https://github.com/erlang/otp/blob/master/lib/kernel/src/file.erl#L973-L987
You can fork the repository, add the opposite function below, and send a pull request to the maintainers. I think that, as reasonable people, they will accept it, and the next version of Erlang/OTP will have it in stdlib :)
I’ve been intending to do that for a while — and as you can see I still haven’t submitted it.
I’ll write myself a TODO note and spend a day this weekend catching up on things like this I’ve set to the side for too long (like adding some of my utility packages to hex.pm and so on…).
I’m glad you found it useful.
I dug into the insides of file:consult/1 a bit so I could write a strings-to-terms function.
You might find this useful also (if hackish): https://zxq9.com/archives/1354