Very slow interaction using Python's telnetlib

Question

I'm writing a Python script that connects to a Linux terminal over Telnet, runs a number of commands and parses the output, then based on the output runs more commands, etc.

This was quite easy to set up using telnetlib. I'm using write(cmd + '\n') to send the command and then read_until(prompt) to read the output. The problem I'm having is that this setup seems to be very slow. Every command takes maybe around 100-200 ms to run. This makes the total run time around half a minute, which I find to be too long.

If I connect to the terminal using a normal Telnet client the commands I'm trying to run return near instantly. I've also made a small bash script that runs ~20 commands that also returns near instantly. Also I tried some of the other read functions in telnetlib (such as read_very_eager()) without any improvement.

Does anyone know why this setup is so slow and if there's anything I can do about it?

Friedrick

I would recommend to use the Python profiler and read the results by pstats so you can see exactly where the script takes most of the time. There can be standard network latencies, DNS resolution or every command is run as new telnet connection? You will see more after you read the results from the profiler. – Ague

All the commands are run in the same Telnet connection, and as I said the commands finish a lot quicker when I run them manually in a normal Telnet client. I tried profiling with cProfile.run() and it seems that nearly all the time is spent on select.select() in telnetlib.py. – Friedrick

pc3e, I know it's has been a long time, but you got any improvements on the timing or found an alternative solution ? – Daudet

Hi Cristiano. Unfortunately I still don't have any solutions to this, and it is still a major inconvenience for me and a number of other people working in the company. A partial solution is to use Linux, which is quite a lot faster, but still not as fast as I think it should be. – Friedrick

Answer 1

I had the same problem you had, i was doing "read_until" and it was working very slow on one machine and fast on the other... I switched my code to "read_very_eager" and put small pauses between request like in example bellow. Now my code works with the same speed everywhere. If you missing some of the responses try to make variable "wait" = bigger.

tn = telnetlib.Telnet(host)
wait=0.1

sleep(wait)              # wait for greeter
tn.read_very_eager();    # optional step
tn.write(PASSWORD+"\n")  # send password
sleep(wait)              # wait for prompt
tn.read_very_eager()     # optional step

tn.write("write some ting\n") # send your request
sleep(wait)                # wait for response to become available
print tn.read_very_eager() # read response IF you need it otherwise skip it
tn.write("write some thing else\n") # send your request
sleep(wait)                # wait for response to become available
print tn.read_very_eager() # read response IF you need it otherwise skip it
tn.write("exit\n")         # optional step
tn.close()                 # close connection

Circumvolution

Sorry about the late answer. This doesn't really work for me, I have to set wait to about 0.3 not to miss any responses. Not trying to exaggerate here, but that's insanely slow! ;) It really is though, as I said the whole script takes half a minute to run. And ping says the round-trip time to the machine is <1ms. Maybe this is somehow a problem with the operating system rather than with Python, I've found out that it runs faster on Linux than on Windows. – Friedrick

Is there a way to set wait dynamically to optimize performance while guaranteeing no missed responses? – Orta

Answer 2

I've used Telnet.expect([b"#", b">"]), which reads until it encounters either "#" or ">" (new line). This could pose an issue if the output you're reading includes either of these two characters. To ensure I capture all the output, I've also used Telnet.read_very_eager().

Here's an example:

tn = telnetlib.Telnet(console_host, console_port)
tn.write(command.encode() + b"\n")
items = tn.expect([b"#", b">"]) # This is a blocking call
print(items[2].decode('utf-8')) 
print(tn.read_very_eager().decode('utf-8'))  # This ensures we capture all the output

Cheshire