Iterating through a Lua table from C++?

Question

I'm trying to load tables from Lua to C++ but I'm having trouble getting it right. I'm getting through the first iteration just fine but then at the second call to lua_next it crashes. Any ideas?

Lua file:

level = { 1, 2, 3, }

C++ file - First I did this:

lua_getglobal( L, "level" );
for( lua_pushnil( L ); lua_next( L, 1 ); lua_pop( L, -2 ) )
{
    if( lua_isnumber( L, -1 ) ) {
        int i = (int)lua_tonumber( L, -1 );
        //use number
    }
}
lua_pop( L, 1 );

Then I tried from the reference manual:

lua_getglobal( L, "level" );
int t = 1;
lua_pushnil( L );
while( lua_next( L, t ) ) {
    printf( "%s - %s", 
        lua_typename( L, lua_type( L, -2 ) ),
        lua_typename( L, lua_type( L, -1 ) ) );
    lua_pop( L, 1 );
}
lua_pop( L, 1 );

And finally this:

lua_getglobal( L, "level" );
lua_pushnil( L );

lua_next( L, 1 );
if( lua_isnumber( L, -1 ) ) {
    int i = (int)lua_tonumber( L, -1 );
    //use number fine
}
lua_pop( L, 1 );

lua_next( L, 1 ); //crashes

etc...

Naturally L is a lua_State* and I'm initializing it and parsing the file okay.

Edit: In response to Jesse Beder answer I tried this code, with a logger, but I still can't get it to work.

Log::Get().Write( "engine", "stack size: %i", lua_gettop( L ) );

lua_getglobal(L, "level");
if( lua_istable( L, -1 ) )
    Log::Get().Write( "engine", "-1 is a table" );

lua_pushnil(L);
if( lua_isnil( L, -1 ) )
    Log::Get().Write( "engine", "-1 is now nil" );
if( lua_istable( L, -2 ) )
    Log::Get().Write( "engine", "-2 is now table" );

int pred = lua_next( L, -2 );
Log::Get().Write( "engine", "pred: %i", pred );
while( pred ) {
    Log::Get().Write( "engine", "loop stuff" );
    if( lua_isnumber( L, -1 ) ) {
        int i = (int)lua_tonumber( L, -1 );
        //use number
        Log::Get().Write( "engine", "num: %i", i );
    }
    Log::Get().Write( "engine", "stack size: %i", lua_gettop( L ) );
    if( lua_istable( L, -3 ) )
        Log::Get().Write( "engine", "-3 is now table" );

    lua_pop( L, 1 );
    Log::Get().Write( "engine", "stack size: %i", lua_gettop( L ) );
    if( lua_istable( L, -2 ) )
        Log::Get().Write( "engine", "-2 is now table" );

    pred = lua_next( L, -2 );
    Log::Get().Write( "engine", "pred: %i", pred );
}
lua_pop( L, 1 );

Which gave this output:

stack size: 0
-1 is a table
-1 is now nil
-2 is now table
pred: 1
loop stuff
num: 1
stack size: 3
-3 is now table
stack size: 2
-2 is now table

Everything you said, Jesse, seems to hold true. But it still fails to go to the next iteration.

Edit2: I tried to copy the exact code into a new project, skipping all the surrounding classes and stuff I didn't bother to include here and there it works. But here it doesn't, and it will just survive one call the lua_next.

Edit3: I've narrowed it down a bit further now. I'm using hge as my 2D engine. I put all the previous code in the function test:

test(); //works
if( hge->System_Initiate() )
{       
    test(); //fails
    hge->System_Start();
}

As far as I understand hge doesn't do anything with lua. Here's the source code for a small test I made. The source for hge 1.81 is here.

Edit4: The question size is getting out of control but it can't be helped. This is the smallest code I've been able to reduce it to.

extern "C"
{
    #include <lua/lua.h>
    #include <lua/lualib.h>
    #include <lua/lauxlib.h>
}
#include <hge\hge.h>

bool frame_func()
{   
    return true;
}

bool render_func()
{
    return false;
}

void test()
{
    lua_State *L = lua_open();
    luaL_openlibs( L );

    if( luaL_dofile( L, "levels.lua" ) ) {
        lua_pop( L, -1 );
        return;
    }
    lua_getglobal(L, "level");
    lua_pushnil(L);

    while( lua_next( L, -2 ) ) {
        if( lua_isnumber( L, -1 ) ) {
            int i = (int)lua_tonumber( L, -1 );
            //use number
        }
        lua_pop( L, 1 );
    }
    lua_pop( L, 1 );

    lua_close( L );
}

int main()
{
    HGE *hge = hgeCreate( HGE_VERSION );

    hge->System_SetState( HGE_FRAMEFUNC, frame_func );
    hge->System_SetState( HGE_RENDERFUNC, render_func );
    hge->System_SetState( HGE_WINDOWED, true );
    hge->System_SetState( HGE_SCREENWIDTH, 800 );
    hge->System_SetState( HGE_SCREENHEIGHT, 600 );
    hge->System_SetState( HGE_SCREENBPP, 32 );

    //test(); //works

    if( hge->System_Initiate() )
    {       
        test(); //fails
        hge->System_Start();
    }

    hge->Release();

    return 0;
}

Wellworn

So the second call to lua_next crashes? This is weird... do you have any debug info on the crash (like where it crashes exactly)? Also, just to make sure things are working properly, you should log the key at each step (it should be a number also) and edit this answer – Neighboring

I added the return value of lua_next. I don't have any debug info and I don't really know how to add it either... – Wellworn

That second edit is a hint - check out the edit on my answer – Neighboring

At what point are you initializing the lua state? If you push that to after System_Initiate, what happens? And when are you calling hgeCreate? I'm wondering if they both use some memory pool that's conflicting. It does appear that you can use both lua and hge together (google "lua hge") but maybe there's some trick you need to do when initializing? – Neighboring

I've tried initializing the lua state both before and after with no result. hgeCreate is called way before. I don't understand why though, I've used lua before just not with lua_next. Maybe it's something else I've done which is messing everything up? But I just can't see it. – Wellworn

Glen Maynard created some useful debug info for visual studio, which makes parsing that lua_State object so much easier. Instructions here: lua-users.org/lists/lua-l/2006-10/msg00491.html – Nisbet

Answer 1

When you call lua_next, the second argument should be the index of the table. Since you're just pushing the table onto the stack with

lua_getglobal(L, "level");

after that call your stack will look like

-1: table "level"

(not +1, since the stack is read going down). Then you call

lua_pushnil(L);

so your stack will be

-1: key (nil)
-2: table "level"

Your table is at -2, so when you call lua_next, you should use the index -2. Finally, after each iteration, your stack should look like:

-1: value
-2: key
-3: table "level"

So you want to read the value (at -1) and then pop it (so just pop once), and then call lua_next to get the next key. So something like this should work:

lua_getglobal(L, "level");
lua_pushnil(L);

while(lua_next(L, -2)) {  // <== here is your mistake
    if(lua_isnumber(L, -1)) {
        int i = (int)lua_tonumber(L, -1);
        //use number
    }
    lua_pop(L, 1);
}
lua_pop(L, 1);

Edit based on your second edit

Since it works when you remove external stuff, but doesn't when you add it back in, my best guess is that you're corrupting the stack somehow (either the C++ stack or the lua stack). Look really carefully at your pointers, especially when you manipulate the lua state.

Neighboring

I narrowed it down further, there's yet another edit for you. – Wellworn

This is very old so maybe this will never be answered, but why do we have to use lua_pushnil before iterating over the table? – Cabanatuan

Found my answer in a comment from this SO answer from 2012. The comment says You start with a "key" of nil, which tells lua_next to get the "first" key which helped me understand why its needed. – Cabanatuan

Answer 2

Reading LUA manual lua_next you find that problems may arise on using lua_tostring on a key directly, that is you have to check the value of the key and then decide to use lua_tostring or lua_tonumber. So you could try this code:

   std::string key
   while(lua_next(L, -2) != 0){ // in your case index may not be -2, check

    // uses 'key' (at index -2) and 'value' (at index -1)

    if (lua_type(L, -2)==LUA_TSTRING){ // check if key is a string
         // you may use key.assign(lua_tostring(L,-2));
    }
    else if (lua_type(L, -2)==LUA_TNUMBER){ //or if it is a number
         // this is likely to be your case since you table level = { 1, 2, 3, }
         // don't declare field ID's
         // try:
         // sprintf(buf,"%g",lua_tonumber(L,-2));
         // key.assign(buf);
    }
    else{
        // do some other stuff
    }
    key.clear();
    lua_pop(L,1)
   }

Hope it helps.

Luciennelucier

Answer 3

Why are you doing the extra lua_pop(L, 1) at the end of the version from the reference manual? I can see how that could be a problem, since you're popping beyond the stack depth.

Uball

I'm removing the table from the stack, in the manual there's only the index to a table but you have to push and pop it somewhere too. – Wellworn