Modifying a C++ array in main() from Lua without extra allocation

Question

I am sketching a small C++ program that will pass arrays to Lua and have them modified there, where I intend to have a lua script read in the program so I can modify it without needing to recompile the program

My first obstacle is to ensure Lua is able to modify arrays already allocated instead of having them allocated again in the Lua space. The data will be float and the size will be really large, but I am starting small for the moment.

To simplify this interface I tried LuaBridge 2.6, but it doesn't provide the expected result. Below is a fully "working" program.

#include <iostream>
#include <cstdint>
#include <cstring>
#include <vector>
#include <lua5.3/lua.hpp>
#include <LuaBridge/LuaBridge.h>

int main(void)
    {
    const uint32_t      LENGTH = 512 * 256;
    std::vector <float> input(LENGTH),
                        output(LENGTH);

    memset(output.data(), 0, LENGTH * sizeof(float));   // Zero the output
    for(uint32_t i = 0; i < LENGTH; i++)                // Populate input
        input[i] = (float)i + 0.5f;

    lua_State *luastate = luaL_newstate();
    luabridge::push(luastate, input.data());    // Supposedly passing a pointer to the first element of input, according to LuaBridge manual chap 3-3.1
    luabridge::push(luastate, output.data());   // Same for output

    luaL_dostring(luastate, "output[10] = input[256]");     // Expecting to assign this value in the C++ arrays, not in the Lua space
    lua_getglobal(luastate, "output[10]");                  // Find this assigned value in the Lua stack
    lua_Number val = lua_tonumber(luastate, 1);             // Retrieving this value from Lua to C++

    std::cout << input[256] << ", " << output[10] << ", " << val << std::endl;  // The values of val and in output[10] don't match

    lua_close(luastate);

    return 0;
    }

Notice that nothing matches. What is going to output[10] in Lua is not the value of input[256] in the C++ space, but input[0]. The C++ output array is not updated from within Lua, cout shows that it remains as we initialized (0). To confirm that, we pushed this value of output[10] to the stack, which is not input[256] in C++, and retrieved from C++. Can you guys correct me or point me to where I should be going to achieve this?

======= UPDATE 08/11/2020 =======

To clarify what the program is doing (or supposed to do), after reading Robert's and Joseph's considerations, I post below an updated version of both the C++ part and the lua script called by it. Notice I abandoned LuaBridge since I didn't succeed in the first attempt:

C++:

#include <iostream>
#include <cstdint>
#include <cstring>
#include <vector>
#include <luajit-2.0/lua.hpp>  // LuaJIT 2.0.4 from Ubuntu 16.04

int main(void)
    {
    const uint32_t      LENGTH = 256 * 512;
    std::vector <float> input(LENGTH),
                        output(LENGTH);

    memset(output.data(), 0, LENGTH * sizeof(float));
    for(uint32_t i = 0; i < LENGTH; i++)
        input[i] = (float)i + 0.5f;

    lua_State *luastate = luaL_newstate();
    luaL_openlibs(luastate);

    // Here I have to pass &input[0], &output[0] and LENGTH
    // to Lua, which in turn will pass to whatever functions
    // are being called from a .so lib opened in Lua-side

    luaL_dofile(luastate, "my_script.lua");    
    lua_close(luastate);

    return 0;
    }

The Lua script looks like this:

local ffi = require("ffi")
local mylib = ffi.load("/path_to_lib/mylib.so")

-- Here I import and call any fuctions needed from mylib.so
-- without needing to recompile anything, just change this script
-- At this point the script has to know &input[0], &output[0] and LENGTH

ffi.cdef[[int func1(const float *in, float *out, const uint32_t LEN);]]
ffi.cdef[[int func2(const float *in, float *out, const uint32_t LEN);]]
ffi.cdef[[int funcX(const float *in, float *out, const uint32_t LEN);]]

if(mylib.func1(input, output, LENGTH) == 0) then
    print("Func1 ran successfuly.")
else
    print("Func1 failed.")
end

Arse

This line is wrong: lua_getglobal(luastate, "output[10]"); – Piling

I am using LuaJIT 2.0.4, it is the one available in Synaptic for Ubuntu 16.04. Notice it is has the Lua header I am using for the update above. In the previous try (before the update) I used Lua 5.3 with LuaBridge 2.6, but I gave up this route for the moment. – Arse

Answer 1

I recommend you create a userdata that exposes the arrays via __index and __newindex, something like this (written as a C and C++ polyglot like Lua itself):

#include <stdio.h>
#include <string.h>

#ifdef __cplusplus
extern "C" {
#endif
#include <lua5.3/lua.h>
#include <lua5.3/lauxlib.h>
#ifdef __cplusplus
}
#endif

struct MyNumbers {
    lua_Number *arr;
    lua_Integer len;
};

int MyNumbers_index(lua_State *L) {
    struct MyNumbers *t = (struct MyNumbers *)luaL_checkudata(L, 1, "MyNumbers");
    lua_Integer k = luaL_checkinteger(L, 2);
    if(k >= 0 && k < t->len) {
        lua_pushnumber(L, t->arr[k]);
    } else {
        lua_pushnil(L);
    }
    return 1;
}

int MyNumbers_newindex(lua_State *L) {
    struct MyNumbers *t = (struct MyNumbers *)luaL_checkudata(L, 1, "MyNumbers");
    lua_Integer k = luaL_checkinteger(L, 2);
    if(k >= 0 && k < t->len) {
        t->arr[k] = luaL_checknumber(L, 3);
        return 0;
    } else {
        return luaL_argerror(L, 2,
                             lua_pushfstring(L, "index %d out of range", k));
    }
}

struct MyNumbers *MyNumbers_new(lua_State *L, lua_Number *arr, lua_Integer len) {
    struct MyNumbers *var = (struct MyNumbers *)lua_newuserdata(L, sizeof *var);
    var->arr = arr;
    var->len = len;
    luaL_setmetatable(L, "MyNumbers");
    return var;
}

int main(void) {
    const lua_Integer LENGTH = 512 * 256;
    lua_Number input[LENGTH], output[LENGTH];

    memset(output, 0, sizeof output);
    for(lua_Integer i = 0; i < LENGTH; ++i)
        input[i] = i + 0.5f;

    lua_State *L = luaL_newstate();

    luaL_newmetatable(L, "MyNumbers");
    lua_pushcfunction(L, MyNumbers_index);
    lua_setfield(L, -2, "__index");
    lua_pushcfunction(L, MyNumbers_newindex);
    lua_setfield(L, -2, "__newindex");
    /* exercise for the reader: implement __len and __pairs too, and maybe shift the indices so they're 1-based to Lua */
    lua_pop(L, 1);

    MyNumbers_new(L, input, LENGTH);
    lua_setglobal(L, "input");
    MyNumbers_new(L, output, LENGTH);
    lua_setglobal(L, "output");

    luaL_dostring(L, "output[10] = input[256]");
    lua_getglobal(L, "output");
    lua_geti(L, -1, 10);
    lua_Number val = lua_tonumber(L, -1);

    printf("%f, %f, %f\n", input[256], output[10], val);

    lua_close(L);
}

With this approach, there is no copy of any data in Lua, and your own MyNumbers_ functions control how all access to them is done.

---

If you want to be able to use the arrays through LuaJIT's FFI instead of directly manipulating them in Lua, then you can pass their addresses in a light userdata instead, like this:

#include <string.h>

#ifdef __cplusplus
extern "C" {
#endif
#include <luajit-2.0/lua.h>
#include <luajit-2.0/lualib.h>
#include <luajit-2.0/lauxlib.h>
#ifdef __cplusplus
}
#endif

int main(void) {
    const lua_Integer LENGTH = 256 * 512;
    lua_Number input[LENGTH], output[LENGTH];

    memset(output, 0, sizeof output);
    for(lua_Integer i = 0; i < LENGTH; ++i)
        input[i] = i + 0.5f;

    lua_State *L = luaL_newstate();
    luaL_openlibs(L);

    lua_pushlightuserdata(L, input);
    lua_setglobal(L, "input");
    lua_pushlightuserdata(L, output);
    lua_setglobal(L, "output");
    lua_pushinteger(L, LENGTH);
    lua_setglobal(L, "LENGTH");

    luaL_dofile(L, "my_script.lua");
    lua_close(L);
}

Treasatreason

I am reading both your approach and Robert's above and taking a step back to make sure I understand the concepts both are presenting. In my mind it was just a matter of "pass to Lua the address of the first element of these arrays and let it do what the script says", but it doesn't seem to be exactly that. With LuaJIT I already managed to do what I intended: open a .so and map the functions to process the float input[] and write to float output[], and do something else in C/C++ side. But what you said about userdata seems to be the way. I am working on it. – Arse

@Arse Here's the key point: Accessing an entire array from just a pointer to its first element requires pointer arithmetic, and Lua doesn't have pointer arithmetic, so it follows that the code to access the array must be written not in Lua, but rather in C (or C++ in your case). – Treasatreason

Thanks for pointing this out. In my particular case the workflow is: C++ (allocate and populate the float arrays) -> Lua (opens a C++ lib and pass these array references to it) -> C++ Lib (will do all the processing of these arrays). In such scenario, where Lua is opening the lib and calling the functions to process the arrays so I don't have to recompile neither the host program nor the library and just modify the script instead, maybe I don't have to worry about pushing functions to deal with the pointer arithmetic, since it is dealt with by the library? Thanks for your patience explaining. – Arse

@Arse What library are you referring to? – Treasatreason

I have updated the original post with better information and the current state of the code. BTW I am unable to tag you with @Joseph at the beginning of the comment. – Arse

@Arse I added an update to my answer based on that. Also, you can't tag me here since this is my own answer, which I always receive notifications for comments on, so the software automatically removes the tag as redundant. – Treasatreason

Your answer was killer. I was working on the lua_pushlightuserdata() and thought "ok, I pusdhed it but how can I refer to this thing from inside Lua script?!?!". And then you show the need of setglobal. It works perfectly as C++ and Lua agree on the addresses when printing them. – Arse

Answer 2

I am sketching a small C++ program that will pass arrays to Lua

The data will be float and the size will be really large,

My suggestion:

• Keep the buffer on the C side (as a global variable for example)
• Expose a C-function to LUA GetTableValue(Index)
• Expose a C-function to Lua SetTableValue(Index, Value)

It should be something like this:

static int LUA_GetTableValue (lua_State *LuaState)
{
  float Value;

  /* lua_gettop returns the number of arguments */
  if ((lua_gettop(LuaState) == 1) && (lua_isinteger(LuaState, -1)))
  {
    /* Get event string to execute (first parameter) */
    Offset = lua_tointeger(LuaState, -1);

    /* Get table value */
    Value  = LUA_FloatTable[Offset];

    /* Push result to the stack */
    lua_pushnumber(Value);
  }
  else
  {
    lua_pushnil(LuaState);  
  }

  /* return 1 value */
  return 1;
}

And you also need to register the function:

lua_register(LuaState, "GetTableValue", LUA_GetTableValue);

I let you write the SetTableValue but it should be very close. Doing so, the buffer is on C side and can be accessed from Lua with dedicated functions.

Swoosh

I have added an update to the question and the answer is below. I also want to thank you for your initial suggestion as it shows the necessity to do the offsetting in the C side. So I left a thumbs up for your considerations. – Arse