Java: Trigonometry and double inaccuracy causing NaN

Question

I have a distance formula using latitude and longitude:

distance = EARTH_MILES_RADIUS
    * Math.acos(Math.sin(lat1 / RADIAN_CONV)
    * Math.sin(lat2 / RADIAN_CONV)
    + Math.cos(lat1 / RADIAN_CONV)
    * Math.cos(lat2 / RADIAN_CONV)
    * Math.cos((lng2 - lng1) / RADIAN_CONV));

lat1,lng1,lat2,lng2 are double primitives. They come to me as double primitives and there is nothing I can do about it.

The problem is that when I have a pair of longitude or latitudes that are the same the formula sometimes returns NaN. I believe this is because I am taking the arc cosine of a number very slightly greater than 1, when in fact it should be exactly 1. I would probably have problems if the points were antipodal as well, where they might be slightly less than -1.

How can I best fix this problem?

Skyscraper

If you want to study a better great-circle distance and bearing calculation implementation in Java, here is the place UNICAR NETCDF library svn.unidata.ucar.edu/repos/common/java/trunk/src/main/java/ucar/… Copyright notice from UNICAR on NetCDF: These applications are copyrighted by UCAR and are intended to be freely available with very minimal restriction. – Purpura

Reformatted code; please revert if incorrect. – Elwandaelwee

@Purpura I see in the linked code the they have if ((lat1 == lat2) && (lon1 == lon2)) in which case the distance is set to zero, so perhaps that is the best solution. Maybe checking for equality within a certain accuracy is unnecessary (puddingfox's solution). This was solution that I tried on my system and it worked great but I wasn't sure if it would work in every case. – Skyscraper

Answer 1

Simply checking for equality was enough to fix my problem:

if (lat1 == lat2 && lng1 == lng2) ...

Skyscraper

Answer 2

Check for two very close (ideally equal) values in your code. For example:

boolean doubleApproxEqual(double a, double b) {
    double PRECISION = 0.000001;
    if (Math.abs(a-b) < PRECISION) //not sure what the name of the function is
                                   //cannot be bothered to check
        return true;
    return false;
}

if you get a True, do cos(1.0) or whatever

Cletuscleve

Personally, I'd use return math.abs(a-b) < PRECISION :-) – Waadt

Answer 3

If you are indeed calculating great circle distance as I think you are, you should use the Vincenty formula instead of what you have.

http://en.wikipedia.org/wiki/Great-circle_distance

Amitosis

Wouldn't the Vincenty formula perform worse? I am processing a very high volume of data. – Skyscraper

Unless you are trying to write a real time application, I don't see how this could be much of an issue. – Amitosis

Performance is a huge issue because it is a website, and the distances are typically pretty close ( < 100 miles ). – Skyscraper

Performance of T.Vincenty formula is not a real issue if correctly implemented...see my comment below your main post on the Java implementation of it which combines Vincenty's method with other methods (Rainsford's + Helmert's elliptical terms) – Purpura