rounding-error Questions
4
I'm working with floating point numbers. If I do:
import numpy as np
np.round(100.045, 2)
I get:
Out[15]: 100.04
Obviously, this should be 100.05. I know about the existence of IEEE 754 and t...
Hierogram asked 16/5, 2018 at 15:25
7
Solved
Hello I have this code in C#:
float n = 2.99499989f;
MessageBox.Show("n = " + n.ToString("f2", CultureInfo.InvariantCulture));
And this code in C++:
float n = 2.99499989f;
printf("n = %.2f", n)...
Paunch asked 18/1, 2010 at 10:58
3
Solved
In C# Is there any difference in the accuracy of the two decimal rounding strategies MidpointRounding.ToEven and MidpointRounding.AwayFromZero? I mean do both ensure an even distribution amongst th...
Multiversity asked 9/9, 2011 at 10:41
4
Solved
I am trying to calculate the length of an Integer in Haskell, using the fact that the length is equal to truncate (log10(x)+1).
Using Integers I created:
len :: Integer -> Integer
len i = toIn...
Columbous asked 10/11, 2014 at 12:43
3
Solved
As you all know since it is one of the most asked topic on SO, I am having problems with rounding errors (it isn't actually errors, I am well aware).
Instead of explaining my point, I'll give an ex...
Distract asked 28/1, 2011 at 17:2
4
Solved
I want to emulate this function. I want to round a floating point number down to the nearest multiple of 0.05 (or generally to the nearest multiple of anything).
I want this:
>>> round_nea...
Lisabeth asked 10/2, 2015 at 6:47
6
Solved
I'm using PHP 5.2.13 on my linux server. I'm getting weird error when rounding numbers. This is my test case:
<?php
echo " " . round(1.505, 2) . "\n";
echo " " . round(11.505, 2) . "\n";
echo "...
Kenney asked 7/2, 2011 at 12:48
2
Solved
I am implementing an Hybrid mobile application in which I have to represent our Desktop application written in C#.
When rounding off a value, the value differs between Desktop and Mobile applicati...
Aeonian asked 28/9, 2017 at 13:43
5
Solved
I've a product that costs 4€ and i need to divide this money for 3 departments.
On the second column, i need to get the number of rows for this product and divide for the number of departments.
M...
Diploblastic asked 24/1, 2011 at 17:12
2
Solved
I'm getting a rounding oddity in Delphi 2010, where some numbers are rounding down in roundto, but up in formatfloat.
I'm entirely aware of binary representation of decimal numbers sometimes givin...
Hamhung asked 25/7, 2019 at 2:23
2
I am using Entity Framework 6.x using the Code First approach on an MVC 5 application. In this particular situation my model (among other things) contains two properties named Latitude and Longitud...
Deportee asked 15/10, 2014 at 21:33
5
Solved
I am curious about the existence of any "rounding" standards" when it comes to the calculation of financial data. My initial thoughts are to perform rounding only when the data is being presented t...
Cavil asked 1/10, 2010 at 15:45
1
Solved
I have a program that were giving slithly different results under Android and Windows. As I validate the output data against a binary file containign expected result, the difference, even if ...
Tightrope asked 18/1, 2019 at 8:30
1
Not sure why, but if I add big to small fp numbers it seems that the increment error is bigger:
#include <iostream>
#include <math.h>
int main() {
std::cout.precision(50);
const in...
Forelli asked 4/11, 2018 at 11:0
3
Solved
I was just reading about rounding errors in C++. So, if I'm making a math intense program (or any important calculations) should I just drop floats all together and use only doubles or is there an ...
Incompetent asked 20/7, 2011 at 9:43
4
The code
import numpy as np
a = 5.92270987499999979065
print(round(a, 8))
print(round(np.float64(a), 8))
gives
5.92270987
5.92270988
Any idea why?
Found nothing relevant in numpy sources.
U...
Scop asked 6/2, 2018 at 8:19
1
Solved
I have a number that I have to deal with that I hate (and I am sure there are others).
It is
a17=0.0249999999999999
a18=0.02499999999999999
Case 1:
round(a17,2) gives 0.02
round(a18,2) gives ...
Corroboration asked 4/1, 2018 at 12:34
2
Solved
I understand that the reason why sin(pi) is not equal to zero is because of the fact that there is not enough bits to store all the significant digits of "pi", but what does that have to do with ma...
Luke asked 11/1, 2017 at 7:10
2
I read data stored in the format DECIMAL from a MySQL-Table. I want to do calculations on those numbers within R.
I used to cast them to a numeri represaentation using as.numeric(), but the docume...
Stull asked 6/6, 2013 at 10:48
4
Solved
I'm in the process of converting a program from Scilab code to C++. One loop in particular is producing a slightly different result than the original Scilab code (it's a long piece of code so I'm n...
Floss asked 20/6, 2016 at 17:7
4
Solved
I am attempting to form a double precision floating point number (64-bit) by taking the ratio of one product of integers divided by another product of integers. I wish to do so in a way that reduce...
Alderman asked 16/5, 2016 at 14:46
2
I want to get rid of one pixel issue on my device.I am designing a calendar which looks like below
I am using a UICollectionViewCell for this purpose
- (UIEdgeInsets)collectionView:
(UIColle...
Dialytic asked 29/1, 2016 at 22:32
4
Solved
Given the following issue with rounding milliseconds under R. How do I get around it so that the times are correct?
> options(digits.secs=3)
> as.POSIXlt("13:29:56.061", format='%H:%M:%OS', ...
Paranoiac asked 7/6, 2012 at 12:40
2
Solved
I am working on an application that is converting Float samples in the range of -1.0 to 1.0 to signed 16bit, to ensure the output of the optimized (SSE) routines are accurate I have written a set o...
Joub asked 14/10, 2015 at 1:30
1
I am working on some c++ problem which should output 22258199.5000000 this number, I have stored the result in double datatype and when I tried to print that variable using std::cout it rounds off ...
Seagraves asked 26/8, 2015 at 16:19
1 Next >
© 2022 - 2024 — McMap. All rights reserved.