Python - ValueError: The length of the input vector x must be greater than padlen, which is 6

Question

I am using the function filtfilt in Python as follows

import numpy as np
from scipy.signal import filtfilt

a = np.array([1, -lambd]).T
b = np.array([-lambd,1]).T
delayed = filtfilt(b,a,sig)

where sig has shape (6,). As a result, I get the following error:

ValueError: The length of the input vector x must be greater than padlen, which is 6.

generated by scipy.signal.

The same code works fine if sig has shape (7,) or longer, while it returns the same error for any shape smaller than (6,). Any idea?

Loveinamist

padlen - int or None, optional - The number of elements by which to extend x at both ends of axis before applying the filter. This value must be less than x.shape[axis] - 1. padlen=0 implies no padding. The default value is 3 * max(len(a), len(b)). (x is third element, that is your sig). In your case default value of padlen is 3*2=6 - but at the same time ` This value must be less than x.shape[axis] - 1`. Source - docs.scipy.org/doc/scipy/reference/generated/… – Mackenzie

try using filtfilt(b,a,sig, axis=0) – Askari

Tried axis=0, same error message – Loveinamist

h4z3 - It looks like that, according to scipy.signal.filtfilt, this error is linked to the filtfilt function itself, and there is no way to get around it (unless I want to use a longer signal). – Loveinamist

I am now implementing scipy.signal.lfilter, which gives no errors. – Loveinamist

signal.filtfilt(b, a, x) if not len(sensor) < len(a)*len(b) else signal.filtfilt(b, a, x, padlen=0) – Luana

@DavideNardone Your comment was perhaps already a valid answer in many cases, but not in all. It does not work on my data if you only take your condition, but you brought the main idea. When I use your conditional filter on len(x) < len(a)*len(b) in a function and in addition constrain the input with [butter_lowpass_filter(x, cutoff, fs, order) for x in df['test'] if len(x) > 0] so that the input is constrained with if len(x) > 0 before reaching your condition, it works. Thus the remaining error was probably caused by some signal lists of length =0 (or strangly also =1, see answer). – Hugibert

Answer 1

If the error is saying:

ValueError: The length of the input vector x must be greater than padlen, which is 6 [or whatever].

Why not filtering the length of the signal sub-elements > 6?

df = df[[len(x) > 6 for x in df['listSignal']]]

For me, it was already enough to filter for a len(x) > 1 to avoid the error. I had quite a few 0-length and 1-length signal lists in the nested 'test' column. I assume that > 1 is needed since only a length of 2 provides one item on the left and one on the right.

The functions that I borrowed from https://github.com/leilamr/LowPassFilter stay untouched:

import pandas as pd
from scipy.signal import butter, freqz, filtfilt

def butter_lowpass(cutoff, fs, order=5):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    b, a = butter(order, normal_cutoff, btype='low', analog=False, fs=None)
    return b, a

def butter_lowpass_filter(x, cutoff, fs, order=5):
    b, a = butter_lowpass(cutoff, fs, order=order)
    return filtfilt(b, a, x)

No error with the filtered df len(x) > 1 (try yourself, yours might be higher):

df = df[[len(x) > 1 for x in df['test']]]
df['testFilt'] = [butter_lowpass_filter(x, cutoff, fs, order) for x in df['test']]

Thanks go to @DavideNardone with his idea of filtering the return value of the function, wheras I am now filtering the main df in advance, the idea is the same.

Hugibert

Answer 2

An explanation as to why this happens:

Applying a filter on a signal is actually convolving the signal with the filter.
The convolution operation is only defined when both the signal and the filter can fully overlap.
Since your filter is of size 6, the signal has to be of at least size 6 for the convolution to be deined.

Dealing with it

One easy way to deal with this is to ignore short signals.
Another is to pad the signal with zeros or otherwise such as here.

---

@questionto42's answer shows another nice way of dealing with the problem.

Horizontal

Answer 3

Use padlen less than length of sig array.

Like following code:

import numpy as np
from scipy.signal import filtfilt

a = np.array([1, -lambd]).T
b = np.array([-lambd,1]).T
delayed = filtfilt(b,a,sig, padlen=len(sig)-1)

Spermary

Answer 4

This may just be my fault, but changing column names did the job for me. I merged 2 dataframes that had the same columns '0', '1' & '2'. So I ended up with repeating names, although it displayed in order and all. I was getting the same error, with the value 33 for some reason. But just renaming the columns made it go away!

Endplay