Interpolating data from a look up table

Question

read the look up table

LUT = np.genfromtxt('test.out', delimiter=',', dtype=float)
LUT:
    12, 25, 136, 6743
    13, 26, 139, 6786
    14, 27, 142, 6791
    15, 28, 145, 6789

Values to be read from the LUT are as follows:

x1, x2, x3 = 12.5, 25.5, 137

Reading the neighboring two values in the LUT for each of the given values (3 columns), I have to linearly interpolate the results (4th column in LUT).

The given values (x1, x2, x3) belongs to between 1st and 2nd row of the LUT. Based on this how to read the results between 1st and 2nd row?

Earthshaking

It's unclear what you want here. You have a 3D coordinate space. Do you want 3D linear interpolation? You don't seem to have sufficient data for that, as your data is along the line x3 = 3*x1+100, x2=x1+13, while your desired coordinate is off that line (12.5,25.5,137.5 is on it). If you're only interested in things on that line, then 1D interpolation works; if you're interested in things off it, you'll need data off it. – Dorman

@Dorman Nope, I have to find out the lower and upper indices in the LUT where the x1,x2,and x3 exists. It is between first and second row. But how to know it by program? If it is known the results would be obtained by linear interpolation of 6743 and 6786. – Earthshaking

So what happens if you have x1,x2,x3 = 12.5,26.3,144? Or will they always all be between the same two rows? If so, are the first three columns always strictly increasing? By linear interpolation, do you mean average? Because if you actually want to do a linear interpolation based on the x1,x2,x3 values, it's not at all obvious how to do so. – Dorman

@Dorman They are always between the same two rows. Yes, first three columns are are strictly increasing. Yes it is average. – Earthshaking

@Borys: If that is the case, then why are all of x1, x2 and x3 necessary in the interpolation? If they are all monotonically increasing by the same step value, what is stopping you from using a simple linear interpolant between just x1 and the fourth column? – Malinowski

@Malinowski Aha sorry, three columns are are strictly increasing but the expected results are always between the same two rows – Earthshaking

May I venture a guess that the reason "this question has not received enough attention" is that you have not explained the question in a way many people can understand? I'm certainly not going to waste any time on it, bounty or not. – Laveralavergne

I must unfortunately agree with @MarkRansom, It seems to me like you just need to assign c1 = LUT[:, 0], (c2=, c3= ...) and then x1 = (c1[0] + c1[1]) / 2 (x2=, x3=), which you already pretty much have. – Amylaceous

Answer 1

Given a list of coordinates coords where you want to interpolate, you can use scipy.spatial.cKDTree to obtain the 2 closest entries of your table that are necessary for the linear interpolation. The code below shows an usage example, already vectorized.

import numpy as np
from scipy.spatial import cKDTree

# inputs
LTU = np.genfromtxt('test.txt', delimiter=',')

coords = ((12.5, 25.5, 137),
          (13.5, 26.5, 141),
          (14.5, 25.5, 144))

# querying and interpolating
xyz = LTU[:, :3]
val = LTU[:, 3]

del LTU # attempt to clean up memory

tree = cKDTree(xyz)
dist, ind = tree.query(coords, k=2)

d1, d2 = dist.T
v1, v2 = val[ind].T
v = (d1)/(d1 + d2)*(v2 - v1) + v1

print(v)
#[ 6758.73909236  6789.16987298  6790.03575996]

Tillett

My LUT is around 1GB. Does your code work with big data? Or there are other methods to keep the LUT rather than in text file for faster reading? – Earthshaking

@Earthshaking 1GB should not be a problem... I've added a del LTU command in an attempt to clean up memory – Tillett

very good answer and its useful for me but i dont understand the line of interpolation: v = (d1)/(d1 + d2)*(v2 - v1) + v1 – Spoliate

Answer 2

It is a little unclear - the context you are working with.

Provided this is a more general LUT: find the closest 2 points via the euclidian distance to all points in the the LUT from the provided point. After establishing those 2 points, use bilinear interpolation on the 4th column.

Now if each column increases in lock step w/ (1, 1, 3) and you have some notion of order here, find the upper and lower bounds with python's bisect module of the first column and you're done finding the indices you would interpolate with (bilinearlly?). Since you mention in the comments below the delta is fixed, this makes this far more a 1d LUT problem than a 3d problem - arguably you could use numpy.interp using just the first dimension and 4th dimension.

If they are not in lock step but a similar ordering is preserved, restrict the range of allowed upper and lower indices by producing a cumulative upper/lower bound across columns, then decide what indices you'd like to interpolate with over that range.

For all of these, if you find an exact value in the LUT, do not bother interpolating.

Amylopectin

Each column increases in lock step. Could you provide working code for this purpose. – Earthshaking