tail -f over ssh with Paramiko has an increasing delay

Question

I'm trying to check for errors in a log file of a running embedded system.

I already have implemented paramiko in my scripts, as I've been told this is the best way to use ssh in python.

Now when I tail the log file I see that there is a big delay build up. Which increases with about 30 seconds per minute.

I already used a grep to decrease the number of lines which is printed as I thought I was receiving too much input but that isn't the case.

How can I decrease this delay or stop the delay from increasing during runtime. I want to tail for hours...

def mkssh_conn(addr):
    """returns an sshconnection"""
    paramiko.util.logging.getLogger('paramiko').setLevel(logging.WARN)
    sshcon = paramiko.SSHClient()
    sshcon.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    sshcon.connect(addr , username, password)
    return sshcon

while True:
    BUF_SIZE = 1024
    client = mkssh_conn() #returns a paramiko.SSHClient()
    transport = client.get_transport()
    transport.set_keepalive(1)
    channel = transport.open_session()
    channel.settimeout(delta)
    channel.exec_command( 'killall tail')
    channel = transport.open_session()
    channel.settimeout(delta)
    cmd = "tail -f /log/log.log | grep -E 'error|statistics'"
    channel.exec_command(cmd)
    while transport.is_active():
        print "transport is active"
        rl, wl, xl = select.select([channel], [], [], 0.0)
        if len(rl) > 0:
            buf = channel.recv(BUF_SIZE)
            if len(buf) > 0:
                lines_to_process = LeftOver + buf
                EOL = lines_to_process.rfind("\n")
                if EOL != len(lines_to_process)-1:
                    LeftOver = lines_to_process[EOL+1:]
                    lines_to_process = lines_to_process[:EOL]
                else:
                    LeftOver = ""
                for line in lines_to_process.splitlines():
                    if "error" in line:
                        report_error(line)
                    print line
    client.close()          

Hartle

Answer 1

I've found a solution: It seems that if I lower BUF_SIZE to 256 the delay decreases. Obviously. I need to recheck if the delay still increases during runtime or not.

Hartle

Answer 2

BUFFER_SIZE should be on the higher end to reduce the cpu cycles ( and in turn to reduce the overall delay due to network latency ) in case if you are working with high throughput tailed pipe.

Further making the BUFFER_SIZE to higher number, should not reduce the performance. ( if paramiko doesn't wail till the buffer fills out in low throughput pipe )

Contradiction between @studioj 's answer and this, probably due to the upgrade of paramiko ( fixed now )

Hagen