Connecting to 'Explicit FTP over TLS' in Python (??)

Question

I cannot figure out how to see the file contents of an FTP site using ftplib.

I can connect to the FTP site using WinSCP just fine, and see the 6 files in the root directory.

In Python 3.4, I am using the following code:

        from ftplib import FTP_TLS
        ftps = FTP_TLS(timeout=100)           
        ftps.connect(ipAddress, 21)
        ftps.auth()
        ftps.prot_p()
        ftps.login('username', 'password')

Which produces:

        Out[72]: '230 User logged in.'

I can then run this:

        ftps.pwd()

...and I see that I am in the root directory:

        Out[73]: '/'

Everything seems to be gravy. BUT, when I try to see what is in the directory, using ftps.dir() or ftps.retrlines('NLST'), or anything else I have tried, I get a timeout:

TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

I have googled around, and found several people saying it could be a 'passive vs active' connection setting problem. However, I asked the administrator of the FTP if it is passive or active, and he acted incredulous, just saying, "It's not SFTP. It is explicit FTP over TLS!!"

I am pretty ignorant of any of this. I am just trying to use ftplib (or any other package!) to connect to his FTP and download some files.

What am I missing??

EDIT: I just figured out WinSCP is using passive mode, so that seems to be what works. As I understand it, ftplib is passive by default, but I went ahead and set the flag to true anyways. This did nothing, and I am still having the same issue.

I also checked my firewall (grabbing at straws), and it is completely turned off, so there should be no issues there.

I also tried sending explicit commands with the '.sendcmd()' method, which seems to be able to send commands, but will not grab any responses (or it cannot see the responses).

Is there some reason why I am able to see the directory I am in, but not any other data? I am obviously connected and talking to the server, and it would seem like the directory being available and viewable to me would imply data can go back and forth. Is the data possibly coming back in some way ftplib doesn't recognize? Any thoughts on where should I begin with troubleshooting this?

Finally, again, it seems like everything is setup correctly on their end, as I can see everything fine in WinSCP. What settings should I check to compare to ftplib?

EDIT 2:

As requested, I set the debug level (thanks for this tip), and below is the output:

ftps = FTP_TLS(timeout=15)
ftps.set_debuglevel(2)                   
ftps.connect(ipAddress, 21)
ftps.set_pasv(True)            
ftps.auth()            
ftps.prot_p()
ftps.login('username', 'password')
ftps.retrlines('NLST')
*get* '220 Microsoft FTP Service\n'
*resp* '220 Microsoft FTP Service'
*cmd* 'AUTH TLS'
*put* 'AUTH TLS\r\n'
*get* '234 AUTH command ok. Expecting TLS Negotiation.\n'
*resp* '234 AUTH command ok. Expecting TLS Negotiation.'
*cmd* 'PBSZ 0'
*put* 'PBSZ 0\r\n'
*get* '200 PBSZ command successful.\n'
*resp* '200 PBSZ command successful.'
*cmd* 'PROT P'
*put* 'PROT P\r\n'
*get* '200 PROT command successful.\n'
*resp* '200 PROT command successful.'
*cmd* 'USER username'
*put* 'USER username\r\n'
*get* '331 Password required for username.\n'
*resp* '331 Password required for username.'
*cmd* 'PASS ****************'
*put* 'PASS ****************\r\n'
*get* '230 User logged in.\n'
*resp* '230 User logged in.'
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A.\n'
*resp* '200 Type set to A.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (10,19,1,137,195,128).\n'
*resp* '227 Entering Passive Mode (10,19,1,137,195,128).'
Traceback (most recent call last):

  File "<ipython-input-13-a79eb3c23dc5>", line 8, in <module>
    ftps.retrlines('NLST')

  File "C:\Anaconda3\lib\ftplib.py", line 467, in retrlines
    with self.transfercmd(cmd) as conn, 
  File "C:\Anaconda3\lib\ftplib.py", line 398, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]

  File "C:\Anaconda3\lib\ftplib.py", line 793, in ntransfercmd
    conn, size = FTP.ntransfercmd(self, cmd, rest)

  File "C:\Anaconda3\lib\ftplib.py", line 360, in ntransfercmd
    source_address=self.source_address)

  File "C:\Anaconda3\lib\socket.py", line 516, in create_connection
    raise err

  File "C:\Anaconda3\lib\socket.py", line 507, in create_connection
    sock.connect(sa)

timeout: timed out

EDIT 3:

I tried to just set the passive flag to false, but when I ask for files, I get the following:

ftps.retrlines('NLST')
*cmd* 'TYPE A'
*put* 'TYPE A\r\n'
*get* '200 Type set to A.\n'
*resp* '200 Type set to A.'
*cmd* 'PORT 10,1,10,100,223,39'
*put* 'PORT 10,1,10,100,223,39\r\n'
*get* '501 Server cannot accept argument.\n'
*resp* '501 Server cannot accept argument.'

EDIT 4:

I enabled logging on WinSCP, and it looks like the helpful Steffen Ullrich was right! This was in the log for WinSCP:

< 2017-05-19 08:44:27.880 227 Entering Passive Mode (10,19,1,137,195,139).
< 2017-05-19 08:44:27.880 Server sent passive reply with unroutable address 10.19.1.137, using host address instead.

So, how do I get ftplib to do the same thing?

EDIT 5:

Found THIS - which is just rewriting the source to make it look to the host. Anyone else got a simpler, less intrusive/possibly-destructive way to do this?

As a second option, since the admin seems less than up to snuff, is there something explicit I can tell him to go do to the host to make this work better (i.e. - have passive return the correct IP)?

Iodine

If what you cite is true then the administrator is clueless, at least in this regard. SFTP is file transfer over SSH and there is no passive vs. active mode there. You are using instead FTPS and there is passive and active similar to plain FTP. But, what is used gets controlled by the client and not by the server. This means you don't need the administrator but need to use FTP.set_pasv in your code. – Dossal

I tried this, but no dice. I checked WinSCP (which is working great), and it was set to passive mode. I set the passive flag in ftplib, but the problem was not resolved. – Iodine

Please enable debugging in ftplib and add the output to your question (not in a comment). It might be that the server includes the wrong IP address in the response to PASV which is often the case with FTP servers behind a firewall. Some FTP clients work around such broken configuration, some don't. – Dossal

Thanks for pointing me to the debug, much appreciated. How do I look for this incorrect IP? And can I work around it in ftplib? See the debug above. – Iodine

Yes, it looks like you are right (see edit 4). The server is replying with a no bueno IP for passive mode. WinSCP seesm to be able to work around it. Can I force ftplib to do the same? – Iodine

Answer 1

*get* '227 Entering Passive Mode (10,19,1,137,195,128).\n'

The problem is that the server is returning the wrong IP address inside the response to the PASV command. This is typical for servers in some internal network behind some firewall. In this case 10.19.1.137 is returned, which is an IP address usable in local networks only.

This is a broken setup of the FTP server. But unfortunately such broken setup is common so many clients work around it by ignoring the given IP address in the response and using instead the IP address of the control connection. ftplib has no support for such workaround. But one can monkey patch it to provide such support:

from ftplib import FTP_TLS

# replace original makepasv function with one which always returns
# the peerhost of the control connections as peerhost for the data
# connection
_old_makepasv = FTP_TLS.makepasv
def _new_makepasv(self):
    host,port = _old_makepasv(self)
    host = self.sock.getpeername()[0]
    return host,port
FTP_TLS.makepasv = _new_makepasv

ftp = FTP_TLS(ipAddress)
ftp.login(...)
ftp.nlst()

This is successfully tested with Python 2.7.12 and Python 3.5.2

Dossal

ERMAHGERD!!!! Works perfectly. I had tried to patch the ftplib itself, and was running into all sorts of issues. I did not think to 'monkey' patch it in my function. Thanks a bunch for walking me though how to diagnose this, and for providing a working solution. – Iodine

Thanks, this helped me patch and FTP TLS upload along with #14659654 – Conspectus

Thanks for this tip! Looking at the Winscp logs I can tell that it does the same thing under the covers: Server sent passive reply with unroutable address 10.6.9.21, using host address instead. – Illinium