We all love file:consult/1 and are familiar now with its inverse function. And of course everyone knows how comfortable it is to use the BIFs term_to_binary/1 and binary_to_term/1,2 to communicate over the network between nodes and even among other networked thingies written in other programming languages using BERT-RPC.
But we still have a gap.
There is not a very well known way to convert a text string that represents Erlang terms directly into a list of actual Erlang terms without writing to a file first and then calling file:consult/1. Most of the time you will never have this problem. But when you do encounter this problem it can be mighty annoying to figure out the steps to convert the string or binary to internal Erlang terms (to the point that I sometimes see people actually write to a temporary file just so they can then call file:consult/1 and then delete the file).
So, let’s take a look:
scan_binary(Bin) ->
TermString = binary_to_list(Bin),
scan_string(TermString).
scan_string(TermString) ->
{_, Strings} = lists:foldl(fun break_terms/2, {"", []}, TermString),
Tokens = [T || {ok, T, _} <- lists:map(fun erl_scan:string/1, Strings)],
AbsForms = [A || {ok, A} <- lists:map(fun erl_parse:parse_exprs/1, Tokens)],
[V || {value, V, _} <- lists:map(fun eval_terms/1, AbsForms)].
break_terms($., {String, Lines}) ->
Line = lists:reverse([$. | String]),
{"", [Line | Lines]};
break_terms(Char, {String, Lines}) ->
{[Char | String], Lines}.
eval_terms(Abstract) ->
erl_eval:exprs(Abstract, erl_eval:new_bindings()).
You’ll notice that I did not simply use string:lexemes(TermString, [$.]) (the successor to the now obsolete string:tokens/2) to break the original into discrete strings. That is because each string requires a period at the end or else erl_scan:string/1 will reject it. It is dramatically more efficient to run through the string a single time breaking at the periods and adding them back than traversing it once to break it into segments, then traversing every resulting string again just to add a period at the end (which also means an extra traversal of the list of that list to make the adjustments!).
Everything in that happens in scan_string/1 can, of course, crash if there is anything wrong in the input. If used as-is it should probably be run inside of a try..catch clause (and you should almost never, ever be using try..catch in Erlang to begin with, but this is one of the very few cases it is probably a good idea to). That could be accomplished by wrapping it in a non-insane function such as:
-spec maybe_scan(String) -> Outcome
when String :: string(),
Outcome :: {ok, [term()]}
| {error, Reason :: term()}.
maybe_scan(String) ->
try
Terms = scan_string(String),
{ok, Terms}
catch
error:Reason -> {error, Reason}
end.
You’ll notice that I have a specific scan_binary/1 and a scan_string/1 also. I haven’t played around with this enough yet to feel comfortable throwing a full-blown io_list() at this, so my assumption is that you’re either reading data in from a file and will have a binary to start with, or would have a string that arrives or is constructed somewhere internally and know that you should flatten it yourself before calling scan_string/1 or maybe_scan/1.
How did I arrive at this?
The larger problem I have had to solve just now is unpacking and reading in configuration data from a large number of tar archives that I receive over the wire. While I could unpack them to disk, then read the file I want with file:consult/1, it is dramatically faster to unpack only the file I wanted from the archive in memory (as the archive itself has never been written to disk anyway), and that leaves me with a binary string of the file contents, but nothing on which I can call file:consult/1. Dhoh!
My solution to that problem was the above. This function has done its work now and I don’t need it anymore, but it strikes me as not such a crazy situation for other programmers to run into at some point so I’m leaving this here for my future self. I’ll probably include this function in a future version of a convenience library, and at that point I will either refactor it to break down all the possible error returns in a proper way (crash reports from within list operations inside list comprehensions can be mysterious), or decide that the details of an error from, say, erl_scan would be more confusing than its worth and instead provide a more generic return from some interface like maybe_scan/1.