Why does formatting a number round differently in LuaJIT vs. Lua?
Asked Answered
M

0

3

Using string.format() (which supposedly defers to C's sprintf())) to format a number in LuaJIT rounds differently than every other Lua interpreter I've tried:

$ lua -v
Lua 5.4.1  Copyright (C) 1994-2020 Lua.org, PUC-Rio
$ lua -e 'print(string.format("%.4f", 32.90625))'
32.9062
$ luajit -v
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
$ luajit -e 'print(string.format("%.4f", 32.90625))'
32.9063

Why does this happen and how can I‌ make it stop? I'd like to tell LuaJIT to round the same direction every other Lua interpreter does.

Metastasize answered 27/10, 2020 at 20:2 Comment(11)
Looks like they have a custom implementation. Followed function calls to github.com/LuaJIT/LuaJIT/blob/… then quickly gave up :)Cutler
I'd like to tell LuaJIT to round - Implement your own string.format in LuaJIT FFI by invoking standard function sprintf as vanilla Lua does.Unreality
In LuaJIT do local lambda=function(f) return f-0.00001 end print(string.format('%.4f',lambda(32.90625))) endWorcester
@Worcester That's roughly what I hacked together, with a few caveats. First, it doesn't work to coerce LuaJIT like that because it doesn't handle all precisions the same way, so I coerced all other Luas to be like LuaJIT instead. Also I had to account for not making 0 a negative and rounding negative floats the same direction. Still the result feels dirty as the dregs of an oil pan—hence why I didn't self answer. Surely there is a better way.Metastasize
Hey, a helper function like lambda is used by the giants of science. Einstein do it for example. Therefore i choose this name for it. A more clean way could be to add the math functions as __index metamethod and write a more precise lambda function to add math.f_floor() ? Try to see what i mean: do float=32.90625 debug.setmetatable(float,{__index=math}) print(float:floor()) endWorcester
yea, LuaJIT seems to round if precision is less than 13 else if (prec < 13) { /* Precision is sufficiently low as to maybe require rounding. */ You could floor it, at the required digit, to make them return similar results print( string.format( "%.4f", math.floor( 32.90625 *10000 ) /10000 ) )Mallen
1 followed by four 0's shifts the decimal over four places, floor() drops everything after that, then divide by the same amount to recover your original decimal location.Mallen
@Wolf My personal website is offline for a while for personal reasons. It will come back when it comes back. In the mean time you can see from the SO interface that this question is still open (even if not relevant to me, others have upvoted so you can see it's relevant to others too), and as for which direction I want the round to happen that's already spelled out in the question.Metastasize
@Wolf Check the last sentence, the examples are just a sample. I want to make LuaJIT behave the same way as other Lua implementations do.Metastasize
@Metastasize Interestingly you seem to be absolutely sure that the question has perfect wording. I'm really sorry for the time I wasted here. Too bad!Sirocco
@Sirocco I'm not saying it's perfect, but the info is there and you know what the question is now even if you missed it the first time. If you don't think it is a clear question either downvote it, edit it to make it clear, or maybe just leave it alone for somebody that recognizes the problem and knows a solution to answer.Metastasize

© 2022 - 2024 — McMap. All rights reserved.