Multiple scripts in a single Lua state and working with _ENV

Question

I'm currently learning how to use the Lua C API and while I've had success binding functions between C/C++ and Lua, I have a few questions:

1. Is it a good idea to load multiple scripts into a single lua_State? Is there a way to close specific chunks? If a script is no longer in use how can I clear it from the lua_State while retaining everything else?

2. What is the best way use scripts that may use the same name for functions/global variables? If I load all of them the newer definitions override the older ones.

   After reading online I think I need to separate each loaded chunk into different environments. The way I envision this working is each time a chunk is loaded I assign it a unique environment name, when I need to work with it it I just use that name to fetch the environment from the LUA_REGISTRYINDEX and perform the operation. So far I haven't figured out how to do this. There are examples online but they use Lua 5.1.

Conjunctiva

Answer 1

After poking around some more I found what I think is the solution I was looking for. I'm not sure if this is the correct/best way to do it, but it works in my basic test case. @jpjacobs answer on this question helped a lot.

test1.lua

x = 1
function hi()
    print("hi1");
    print(x);
end
hi()

test2.lua

x =2
function hi()
    print("hi2");
    print(x);
end
hi()

main.cpp

int main(void)
{
    lua_State* L = luaL_newstate();
    luaL_openlibs(L);

    char* file1 = "Rooms/test1.lua";
    char* file2 = "Rooms/test2.lua";

    //We load the file
    luaL_loadfile(L, file1);
    //Create _ENV tables
    lua_newtable(L);
    //Create metatable
    lua_newtable(L);
    //Get the global table
    lua_getglobal(L, "_G");
    lua_setfield(L, -2, "__index");
    //Set global as the metatable
    lua_setmetatable(L, -2);
    //Push to registry with a unique name.
    //I feel like these 2 steps could be merged or replaced but I'm not sure how
    lua_setfield(L, LUA_REGISTRYINDEX, "test1");
    //Retrieve it. 
    lua_getfield(L, LUA_REGISTRYINDEX, "test1");
    //Set the upvalue (_ENV)
    lua_setupvalue(L, 1, 1);
    //Run chunks
    lua_pcall(L, 0, LUA_MULTRET, 0);

    //Repeat
    luaL_loadfile(L, file2);
    lua_newtable(L);
    lua_newtable(L);
    lua_getglobal(L, "_G");
    lua_setfield(L, -2, "__index");
    lua_setmetatable(L, -2);
    lua_setfield(L, LUA_REGISTRYINDEX, "test2");
    lua_getfield(L, LUA_REGISTRYINDEX, "test2");
    lua_setupvalue(L, 1, 1);
    lua_pcall(L, 0, LUA_MULTRET, 0);

    //Retrieve the table containing the functions of the chunk
    lua_getfield(L, LUA_REGISTRYINDEX, "test1");
    //Get the function we want to call
    lua_getfield(L, -1, "hi");
    //Call it
    lua_call(L, 0, 0);
    //Repeat
    lua_getfield(L, LUA_REGISTRYINDEX, "test2");
    lua_getfield(L, -1, "hi");
    lua_call(L, 0, 0);
    lua_getfield(L, LUA_REGISTRYINDEX, "test2");
    lua_getfield(L, -1, "hi");
    lua_call(L, 0, 0);
    lua_getfield(L, LUA_REGISTRYINDEX, "test1");
    lua_getfield(L, -1, "hi");
    lua_call(L, 0, 0);

    lua_close(L);
}

Output:

hi1
1
hi2
2
hi1
1
hi2
2
hi2
2
hi1
1

I'm using Lua 5.3.2 with Visual Studio 2013 if that means anything.

This basic test case works as needed. I'll continue testing to see if any issues/improvements come up. If any sees any way I could improve this code or and glaring mistakes, please leave a comment.

Conjunctiva

I really like this answer, and am working on using something similar. How could I share memory between functions within the same file? Ideally, I'd like to create variables in the file outside of the functions, then give modifiable access to the functions within the file; so far, as-is, they have access, however, their modifications don't last past the function itself – Legionary

Is it possible to declare a variable in the file? In my example the variable x is global for all functions and changes should stay. Can you give me an example? – Conjunctiva

Sure. "local y = 0; function First(); y = 12; z = 21; end; function Second(); print( y ) -- 12; print( z ) -- nil; end" sorry, no newlines in comments, right? So, I have two functions, I have a "file level" variable Y which is accessible by the functions (closures, right?), I then create a variable in First function, however, calling Second, I don't see the variable. – Legionary

Is Second() in the same file? – Conjunctiva

Yes, those are all in the same file. The idea is that file1 is treated like a "class", where anything defined outside of a function should be a member variable (I think in the _ENV you've created, right?), and functions are then member functions. So in C++, I will pcall the entire file to have the members created and init'd, then randomly called the First and Second functions based on C++ -side events (like Update, Start, Stop, etc). – Legionary

Ahhhh, well see, if you call Second() without calling First(), z wont be created. When you load a file in lua, everything is shoved onto the stack (from what I know), First() and Second() are not called, they're just stored on the stack for later use.When you call First() z is created as a global. But if your file was like ""local y = 0; function First(); y = 12; z = 21; end; function Second(); print( y ) -- 12; print( z ) -- nil; end First()", it would work since that last First() would be pushed onto the stack as a function call. I recommend creating an "Init()" function like this. – Conjunctiva

Let us continue this discussion in chat. – Legionary

Answer 2

Is it a good idea to load multiple scripts into a single lua_State?

Yes, definitely. Unless those scripts is unrelated and should run in multiple parallel threads.

Is there a way to close specific chunks?

Chunk is just a value of type "function". When you don't have that value stored anywhere - chunk will be garbage-collected.
Anything chunk produced - globals, or locals that has references somewhere outside - those will live on.

how to clear it from the lua_State while retaining everything else?

That depends on how do you see that chunk. Is it just a set of functions, or represents some entity with own state. If you don't create global functions and variables, then everything defined in separate script file will be local to chunk, and will be removed when there's no references to chunk left.

What is the best way use scripts that may use the same name for functions/global variables?

Consider rewriting your code. Do not create any globals, unless it's explicitly required to build communications with other parts of your program. Make variables local (owned by chunk), or store it in table/closure that will be returned by chunk as new object - chunk might be a factory producing new objects, and not jush the script.
Also Lua runs just faster with local variables.

The way I envision this working is each time a chunk is loaded I assign it a unique environment name

You should do that if scripts comes from outside - written by users, or received from other external sources. Sandboxing is cool, but there's no need in sandboxing if chunks is your internal stuff. Consider rewriting code without globals. Return some object (api table, or closure) if your chunk produces other objects - you can call that chunk many times without reloading it. Or save one global - module interface, if chunk represents Lua-like module. If you don't organize your code well, then you will be forced to use separate environments, and you'll have to prepare new environment for every script, copy basic stuff like print/pairs/string/etc. You'll have many breaks in run time until you figure out what's more is missing from new environment, and so on.

Ariana

Thanks for your answer! I guess I should have mentioned in my original post that my end goal is to sandbox, or at least run scripts not written by me. I will have general guidelines that must be followed but I also want users and contributes to have freedom. These guidelines may require users to have a globally defined table A in all their scripts. This means I need to use environments since their might be any number of these scripts in the state at any time, but I can't figure out how to do it from C/C++ – Conjunctiva

Answer 3

you should treat each of your scripts as different module. just like you have more then 1 "require" in your code.

your 'loaded chunk' should return table that will be stored globally.

this isn't good idea to load a lot of global variables, this can cause bad thing after you will add more modules.

Geodetic

Isn't that what I did in my answer below? Isn't _ENV a table? – Conjunctiva