floating-accuracy Questions
1
Solved
I'm developing on OS portable software that has unit tests that must work on Linux, UNIX, and Windows.
Imagine this unit test that asserts that the IEEE single-precision floating point value 1.267...
Yawl asked 31/5, 2014 at 7:52
2
Suppose the processor has only 'fadd' and 'fmul' operations (no 'dot' or 'fma' instructions) which are IEEE-754 compliant. What is the worst case accuracy that will be achieved by the trivial imple...
Tanguy asked 28/5, 2014 at 6:37
5
Solved
Can someone give me an example of a floating point number (double precision), that needs more than 16 significant decimal digits to represent it?
I have found in this thread that sometimes you need...
Surpass asked 25/5, 2011 at 0:11
3
Solved
I'm running into a baffling issue with a basic MySQL query.
This is my table:
id | rating
1 | 1317.17
2 | 1280.59
3 | 995.12
4 | 973.88
Now, I'm attempting to find all rows where the rating col...
Volny asked 28/4, 2014 at 12:55
1
Solved
I store some numbers in a MySQL using the ORM of SQLAlchemy. When I fetch them afterward, they are truncated such that only 6 significant digits are conserved, thus losing a lot of precision on my ...
Favela asked 24/4, 2014 at 22:19
7
Solved
I have an std::vector of floats that I want to not contain duplicates but the math that populates the vector isn't 100% precise. The vector has values that differ by a few hundredths but should be ...
Fireboard asked 22/4, 2014 at 23:45
3
Solved
Can someone explain why 1.000000 <= 1.0f is false?
The code:
#include <iostream>
#include <stdio.h>
using namespace std;
int main(int argc, char **argv)
{
float step = 1.0f / 10...
Humorous asked 1/4, 2014 at 16:36
5
Solved
I have a large amount of numeric values y in javascript. I want to group them by rounding them down to the nearest multiple of x and convert the result to a string.
How do I get around the ...
Chorus asked 27/7, 2012 at 21:3
5
Solved
I'm putting a float in an Android based SQLite database, like so:
private static final String DATABASE_CREATE =
"create table " + DATABASE_TABLE + " ("
+ KEY_ID + " integer primary key autoinc...
Denude asked 17/7, 2010 at 7:7
2
Solved
Everyone knows sqrt function from math.h/cmath in C/C++ - it returns square root of its argument. Of course, it has to do it with some error, because not every number can be stored precisely. But a...
Dulcy asked 7/3, 2014 at 19:57
2
Solved
Is there a gcc flag to signal a warning/error when I try to put a double value into an int variable? I currently have -Wall -Wextra -Werror set but I still don't get warned when I (for instance) pa...
Cavesson asked 5/3, 2014 at 12:50
4
Solved
Why is the output different when adding same numbers?
public class Test {
public static void main(String a[]) {
double[] x = new double[]{3.9, 4.3, 3.6, 1.3, 2.6};
System.out.println(">>...
Agapanthus asked 27/1, 2014 at 5:50
2
Solved
I'm trying to write unit tests for some simple vector math functions that operate on arrays of single precision floating point numbers. The functions use SSE intrinsics and I'm getting false positi...
Dampen asked 13/1, 2014 at 17:22
1
Solved
I am struggeling with floating point arithmetic, because I really want to understand this topic!
I know that the numbers can be represented in scientific notation.
So for both numbers the exponen...
Royden asked 19/11, 2013 at 7:14
2
Solved
I am having problem with the LISP expression below. There is floating precision error while doing sum for floating point numbers.
CL-USER> (+ -380 -158.27 -35.52)
Actual: -573.79004
Expected: ...
Capet asked 10/12, 2013 at 9:5
3
Solved
I have the following Python code and output:
>>> import numpy as np
>>> s = [12.40265325, -1.3362417499999921, 6.8768662500000062, 25.673127166666703, 19.733372250000002, 21.6495...
Lebrun asked 4/10, 2013 at 22:6
2
Solved
In my c++ app i have a vector of doubles in the range (0,1) and i have to calculate its total as accurately as possible.
It feels like this issue should have been addressed before, but i cant find ...
Fleck asked 2/10, 2013 at 4:34
2
This seems basic but I am having a lot of trouble answering the following question:
Give two numbers X and Y represented in the IEEE754 format such that computing X-Y will result in underflow.
To...
Saval asked 27/9, 2013 at 14:46
5
Solved
This a very simple question, but an important one since it affects my whole project tremendously.
Suppose I have the following code snipet:
unsigned int x = 0xffffffff;
float f = (float)((double)...
Grunberg asked 6/8, 2013 at 16:25
1
Solved
For 1-D numpy arrays, this two expressions should yield the same result (theorically):
(a*b).sum()/a.sum()
dot(a, b)/a.sum()
The latter uses dot() and is faster. But which one is more accurate? ...
Systematics asked 7/8, 2013 at 1:2
1
Solved
Background
Matlab's built-in eps function [1] can take a numeric value X and return "the positive distance from abs(X) to the next larger in magnitude floating point number of the same precision"....
Sokol asked 12/6, 2013 at 15:45
2
Solved
In F#. How to efficiently compare floats for equality that are almost equal? It should work for very large and very small values too. I am thinking of first comparing the Exponent and then the Sign...
Bunn asked 11/6, 2013 at 9:31
1
Solved
I have to put the price of some items inside a mysql table. When creating the table I'm using DECIMAL(10,2) as I don't need more than 2 digits after the comma (for example: 123,45 would be accepted...
Tantalum asked 5/6, 2013 at 12:43
3
Solved
Debugging some finance-related SQL code found a strange issue with numeric(24,8) mathematics precision.
Running the following query on your MSSQL you would get A + B * C expression result to be 0....
Alicealicea asked 24/9, 2008 at 10:31
3
Why does sinl give incorrect results when the argument is near a non-zero multiple of pi? Why does sinl give incorrect results when the argument is large? The following code illustrates that.
Note...
Postcard asked 2/6, 2013 at 6:29
© 2022 - 2024 — McMap. All rights reserved.