erlang rebar escriptize & nifs

Question

I can use nif's if I write the escript myself, however when I use rebar escriptize the nif functions cannot be found. I think it is because *.so objects are not getting packed like beam files. Here is an simple example;

rebar.config:

{deps, [
   {'jiffy', "", {git, "https://github.com/davisp/jiffy.git", {branch, master}}}
]}.
{escript_incl_apps, [jiffy]}.
%% I tried this to see what happens if the so got in there but didn't help
{escript_incl_extra, [{"deps/jiffy/priv/jiffy.so", "/path/to/my/proj"}]}.

test.erl:

-module(test).

-export([main/1]).

main(_Args) ->
    jiffy:decode(<<"1">>),
    ok.

rebar get-deps compile escriptize
./test

and the result is

escript: exception error: undefined function jiffy:decode/1
  in function  test:main/1 (src/test.erl, line 7)
  in call from escript:run/2 (escript.erl, line 741)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_it/1
  in call from init:start_em/1

Is there a way to overcome this ?

Lydon

From the error it looks like it is jiffy.beam which cannot be found, not *.so. Maybe escriptize takes into account only beams from ebin ignoring dependencies' ebins? – Biographical

@Ed'ka, nope if you add a dependency which is not a nif, it works fine. – Lydon

But if you try to call jiffy:decode/1 with jiffy.so removed from priv you should get Failed to load NIF library error, not undefined function – Biographical

Answer 1

The problem is that the erlang:load_nif/1 function does not implicitly use any search path nor do anything smart in trying to find the .so file. It just tries to load the file literally as given by the file name argument. If it is not an absolute file name then it will try to load the file relative to the current working directory. It loads exactly what you tell it to load.

So if you call erlang:load_nif("jiffy.so") then it will try to load "jiffy.so" from your current working directory. A simple work around that I have used is to do something like this which uses the NIF_DIR environment variable:

load_nifs() ->
    case os:getenv("NIF_DIR") of
        false -> Path = ".";
        Path -> Path
    end,
    ok = erlang:load_nif(Path ++ "/gpio_nifs", 0).

This can easily be extended to loop down a search path to find the file. Note that NIF_DIR is not a special name, just one I have "invented".

Doit

I edited jiffy code to do this but didn't help. This is similar to what @lastcanal suggested. Instead of copying to .so to correct place, you look for it recursively. I think this problem is not relavent to loading .so. As Ed'ka pointed out it says undefined function jiffy:decode/1. – Lydon

Yes, but it was never stated where jiffy:decode/1 was supposed to be defined. If it was in the .so file then a reason could have been the file was never loaded. Or not defined at all. – Doit

it is defined here. And it makes a call to actual nif function, if it was defined properly defined and in the code path it should give an erlang:nif_error right? – Lydon

Then cannot load the jiffy module and the undef error refers to it not being able to find the jiffy module and not that the decode/1 function is undefined. – Doit

I guess the only way to is to modify the code of the dep to look at 'right' place(s) which is less then elegant. I wish there was a better way to this. – Lydon

Answer 2

It does not seem possible to load a nif from an escript because erlang:load_nif does not look into archives. This is because most operating systems require a physical copy of the *.so that can be mapped to memory.

The best way to overcome this is by copying the *.so files into the output directory of the escript.

  {ok, _Bytes} = file:copy("deps/jiffy/priv/jiffy.so", "bin/jiffy.so"),

Take a look at the escript builder for edis. You will see this is how they load the eleveldb's nif for execution from an escript.

Psoriasis

edis code is very similar to rebar code. I tried copying jiffy.so to the same directory with the escript but didn't help. – Lydon