In Lua, how should I handle a zero-based array index which comes from C?

Question

Within C code, I have an array and a zero-based index used to lookup within it, for example:

char * names[] = {"Apple", "Banana", "Carrot"};
char * name = names[index];

From an embedded Lua script, I have access to index via a getIndex() function and would like to replicate the array lookup. Is there an agreed on "best" method for doing this, given Lua's one-based arrays?

For example, I could create a Lua array with the same contents as my C array, but this would require adding 1 when indexing:

names = {"Apple", "Banana", "Carrot"}
name = names[getIndex() + 1]

Or, I could avoid the need to add 1 by using a more complex table, but this would break things like #names:

names = {[0] = "Apple", "Banana", "Carrot"}
name = names[getIndex()]

What approach is recommended?

Edit: Thank you for the answers so far. Unfortunately the solution of adding 1 to the index within the getIndex function is not always applicable. This is because in some cases indices are "well-known" - that is, it may be documented that an index of 0 means "Apple" and so on. In that situation, should one or the other of the above solutions be preferred, or is there a better alternative?

Edit 2: Thanks again for the answers and comments, they have really helped me think about this issue. I have realized that there may be two different scenarios in which the problem occurs, and the ideal solution may be different for each.

In the first case consider, for example, an array which may differ from time to time and an index which is simply relative to the current array. Indices have no meaning outside the code. Doug Currie and RBerteig are absolutely correct: the array should be 1-based and getIndex should contain a +1. As was mentioned, this allows the code on both the C and Lua sides to be idiomatic.

The second case involves indices which have meaning, and probably an array which is always the same. An extreme example would be where names contains "Zero", "One", "Two". In this case, the expected value for each index is well-known, and I feel that making the index on the Lua side one-based is unintuitive. I believe one of the other approaches should be preferred.

Alphonse

What's the problem doing it like you showed in your first example? – Zellner

@TonyTheLion: The problem is the need to remember to add 1 every time the index is used. The + 1 everywhere is also unsightly, whereas the second example has "ugly" code in only one place. – Alphonse

Re: indices are "well-known" -- this use of "magic numbers" is a poor practice, and should be avoided. The better alternative is to use the getIndex function, or the string. Note that in Lua strings are cheap to use because they are interned and string comparison is a pointer comparison. – Ferro

Regarding your edit on "indices which have meaning"... of course, if you need to translate a number to an object because the number is significant, by all means use the number as the index to the Lua table. This is especially true if the number can be the result of a computation. This is similar to memoization. In these cases, the table may also be sparse, e.g., a table from street numbers to occupant names. – Ferro

Answer 1

Use 1-based Lua tables, and bury the + 1 inside the getIndex function.

Ferro

This. Hide the idiosyncratic differences between C and Lua in the glue between C and Lua. This allows your Lua user to see the world in a way that is idiomatic in Lua, while preserving C idiom. The friction is concealed in getIndex, and in the function you write to publish a C array into Lua as a table holding a proper Lua sequence. – Gorlin

Answer 2

I prefer

names = {[0] = "Apple", "Banana", "Carrot"}
name = names[getIndex()]

Some of table-manipulation features - #, insert, remove, sort - are broken.
Others - concat(t, sep, 0), unpack(t, 0) - require explicit starting index to run correctly:

print(table.concat(names, ',', 0))  --> Apple,Banana,Carrot
print(unpack(names, 0))             --> Apple   Banana  Carrot

I hate constantly remembering of that +1 to cater Lua's default 1-based indices style.
You code should reflect your domain specific indices to be more readable.
If 0-based indices are fit well for your task, you should use 0-based indices in Lua.

I like how array indices are implemented in Pascal: you are absolutely free to choose any range you want, e.g., array[-10..-5]of byte is absolutely OK for an array of 6 elements.

Houseraising

Answer 3

First of all, this situation is not unique to applications that mix Lua and C; you can face the same question even when using Lua only apps. To provide an example, I'm using an editor component that indexes lines starting from 0 (yes, it's C-based, but I only use its Lua interface), but the lines in the script that I edit in the editor are 1-based. So, if the user sets a breakpoint on line 3 (starting from 0 in the editor), I need to send a command to the debugger to set it on line 4 in the script (and convert back when the breakpoint is hit).

Now the suggestions.

(1) I personally dislike using [0] hack for arrays as it breaks too many things. You and Egor already listed many of them; most importantly for me it breaks # and ipairs.

(2) When using 1-based arrays I try to avoid indexing them and to use iterators as much as possible: for i, v in ipairs(...) do instead of for i = 1, #array do).

(3) I also try to isolate my code that deals with these conversions; for example, if you are converting between lines in the editor to manage markers and lines in the script, then have marker2script and script2marker functions that do the conversion (even if it's simple +1 and -1 operations). You'd have something like this anyway even without +1/-1 adjustments, it would just be implicit.

(4) If you can't hide the conversion (and I agree, +1 may look ugly), then make it even more noticeable: use c2l and l2c calls that do the conversion. In my opinion it's not as ugly as +1/-1, but has the advantage of communicating the intent and also gives you an easy way to search for all the places where the conversion happens. It's very useful when you are looking for off-one bugs or when API changes cause updates to this logic.

Overall, I wouldn't worry about these aspects too much. I'm working on a fairly complex Lua app that wraps several 0-based C components and don't remember any issues caused by different indexing...

Gewirtz

Answer 4

This is where Lua metemethods and metatables come in handy. Using a table proxy and a couple metamethods, you can modify access to the table in a way that would fit your need.

local names = {"Apple", "Banana", "Carrot"} -- Original Table
local _names = names -- Keep private access to the table
local names = {}    -- Proxy table, used to capture all accesses to the original table

local mt = {
  __index = function (t,k)
    return _names[k+1]   -- Access the original table
  end,

  __newindex = function (t,k,v)
    _names[k+1] = v   -- Update original table
  end
}

setmetatable(names, mt)  

So what's going on here, is that the original table has a proxy for itself, then the proxy catches every access attempt at the table. When the table is accessed, it increment the value it was accessed by, simulating a 0-based array. Here are the print result:

print(names[0]) --> Apple
print(names[1]) --> Banana
print(names[2]) --> Carrot
print(names[3]) --> nil

names[3] = "Orange" --Add a new field to the table
print(names[3]) --> Orange

All table operations act just as they would normally. With this method you don't have to worry about messing with any unordinary access to the table.

EDIT: I'd like to point out that the new "names" table is merely a proxy to access the original names table. So if you queried for #names the result would be nil because that table itself has no values. You'd need to query for #_names to access the size of the original table.

EDIT 2: As Charles Stewart pointed out in the comment below, you can add a __len metamethod to the mt table to ensure the #names call gives you the correct results.

Febrific

Does this approach have any advantages over local names = {[0] = "Apple", "Banana", "Carrot"}? Although the two approaches have different behavior for both #names and ipairs(names), for neither approach is the behavior ideal. – Alphonse

From what I can tell, the advantages stem from the fact that the original array is still 1-based, so all the standard Lua function remain intact (insert, ipairs, remove, #) as long as they are being called on the original table, not the proxy table. It is only accessed as a 0-based array through the proxy. If you'd like to learn more about Lua metatables and proxies I'd recommend checking out Programming in Lua @ link. The 1st Edition is free online although slightly outdated. – Febrific

In Lua 5.2, by setting the __len method in mt, above, you can ensure #names gives the correct result. – Amphibology

Answer 5

Why not just turn the C-array into a 1-based array as well?

char * names[] = {NULL, "Apple", "Banana", "Carrot"};
char * name = names[index];

Frankly, this will lead to some unintuitive code on the C-side, but if you insist that there must be 'well-known' indices that work in both sides, this seems to be the best option.

A cleaner solution is of course not to make those 'well-known' indices part of the interface. For example, you could use named identifiers instead of plain numbers. Enums are a nice match for this on the C side, while in Lua you could even use strings as table keys.

Another possibility is to encapsulate the table behind an interface so that the user never accesses the array directly but only via a C-function call, which can then perform arbitrarily complex index transformations. Then you only need to expose that C function in Lua and you have a clean and maintainable solution.

Intersidereal

Thank you for these suggestions. Unfortunately I am unable to change the interface and/or the underlying C code - I am looking for the best way to structure my Lua code around an existing interface. – Alphonse

Answer 6

Why not present your C array to Lua as userdata? The technique is described with code in PiL, section 'Userdata'; you can set the __index, __newindex, and __len metatable methods, and you can inherit from a class to provide other sequence manipulation functions as regular methods (e.g., define an array with array.remove, array.sort, array.pairs functions, which can be defined as object methods by a further tweak to __index). Doing things this way means you have no "synchronisation" issues between Lua and C, and it avoids risks that "array" tables get treated as ordinary tables resulting in off-by-one errors.

Amphibology

Answer 7

You can fix this lua-flaw by using an iterator that is aware of different index bases:

function iarray(a)
  local n = 0
  local s = #a
  if a[0] ~= nil then
    n = -1
  end
  return function()
    n = n + 1
    if n <= s then return n,a[n] end
  end
end

However, you still have to add the zeroth element manually:

Usage example:

myArray = {1,2,3,4,5}
myArray[0] = 0
for _,e in iarray(myArray) do
  -- do something with element e
end

Coleridge