Net::SFTP connection not closing

Question

I have a Ruby on Rails (Rails 3.2.14 and ruby 1.9.3) application that uploads 2 files to a remote SFTP server. The SFTP code is:

require 'net/sftp'
Rails.logger.info("Creating SFTP connection")
uri = URI.parse('sftp://'+ host)
Net::SFTP.start(uri.host,'user', :password=>'password',:port=>port) do |sftp|
    Rails.logger.info("SFTP Connection created, uploading files.")
    sftp.upload!("public/file1.txt", "./file1.txt")
    Rails.logger.info("First file uploaded.")
    sftp.upload!("file2.txt", "./file2.txt")
    Rails.logger.info("Both files uploaded, terminating connection.")
end
Rails.logger.info("Connection terminated.")

Both files are uploading properly to the remote server, but the connection doesn't seem to close. I keep getting an error when I execute this function and on analyzing my console, I see that the "Both files uploaded, terminating connection." logger message is running, but nothing after that. I've tried using

sftp.close(:handle)
sftp.close!(:handle)
#and
sftp.close_connection()

but none of them are working. Any idea on why this is happening and how I can rectify it? I'm running this through a single instance Engine Yard cloud server.

EDIT These are the last few lines in my log: Creating SFTP connection SFTP Connection created, uploading files. First file uploaded. Both files uploaded, terminating connection.

After that, nothing. When viewing my log with the 'tail -f' command, the log goes up to that last line, and the app redirects to the internal server error page.

Weanling

The issue is not about connection not getting closed. It closes automatically when the block ends. What is the error that you keep getting? – Monarchist

I get an internal server error that directs me to the 500.html file in my public folder. Is there any way I can verify exactly what this error is or what's causing the error? – Weanling

It is better to look at the logs log/production.log – Monarchist

The log terminates after Rails.logger.info("Both files uploaded, terminating connection."). This is the last message the log shows. After that, the error occurs, I'm redirected to the server error page and the log doesn't show anything. Considering that the log doesn't move on to Rails.logger.info("Connection terminated."), I assumed that it's a connection termination issue... – Weanling

You have post to the controller & action where you are doing this. – Monarchist

I don't quite understand what you're saying. I just tried to figure out the exact error by implementing exception handling, but even that didn't work. Whatever the error is, it completely skips the catch block and goes straight to the internal server error page. – Weanling

There is log file in your Rails.root/log/production.log. You can see the exact error there. If you are not sure post the last few lines here. – Monarchist

I've updated the question with the log – Weanling

Answer 1

Short answer

Net::SFTP.start('host', 'user', password: 'pass', port: 22) do |sftp|

  # Do stuff

end

is equivalent to :

session = Net::SSH.start('host', 'user', password: 'pass', port: 22)
sftp = Net::SFTP::Session.new(session)
sftp.connect!

# Do stuff

sftp.close_channel unless sftp.nil?
session.close unless session.nil?

Better answer

For people that couldn't find a way to actually close the connection, without using auto-closing blocks, here is how I you can achieve this :

require 'net/ssh'
require 'net/sftp'

begin

  # Instance SSH/SFTP session :
  session = Net::SSH.start('host', 'user', password: 'pass', port: 22)
  sftp = Net::SFTP::Session.new(session)

  # Always good to timeout :
  Timeout.timeout(10) do
    sftp.connect! # Establish connection

    # Do stuff

  end

rescue Timeout::Error => e

  # Do some custom logging
  puts e.message 

ensure

  # Close SSH/SFTP session
  sftp.close_channel unless sftp.nil? # Close SFTP
  session.close unless session.nil? # Then SSH

  # If you really really really wanna make sure it's closed,
  # and raise after 10 seconds delay
  Timeout.timeout(10) do
    sleep 1 until (sftp.nil? or sftp.closed?) and (session.nil? or session.closed?)
  end

end

If you don't close the connection before performing some other task, you might sometimes experience errors like this in rails for instance :

IOError (not opened for reading) # Not closed when rendering controller action
ActionView::Template::Error (not opened for reading) # Not closed when rendering template

Debate

Really appreciated the breakdown here of what the Net:SFTP:start do structure is equivalent to. – Barque

While this is a really good breakdown. People need to be warned about Timeout and it's potential implications. jvns.ca/blog/2015/11/27/… – Middlebrooks

@Middlebrooks : Thanks, I didn't know about that. Luckily, I always used the explicit rescue Timeout::Error in the apps I have in production :) – Debate

Note that you don't need to manually create a Net::SSH session. If you are using Net::SFTP.start, you can still close the SSH session using sftp.session.close (first do sftp.close_channel). – Affiliate

Answer 2

The auto-closing block will be able to close if you send an EOF at the end of your block:

Net::SFTP.start(host, user, port: port, password: password) do |sftp|
    # ... do your things
    sftp.channel.eof!
end

Caduceus

One of the servers I was integrating with had this problem after sometime running smoothly and closing the channel fixed it. I'm still not sure why this happened to a single server though – Furze

Worked for me as well, but could someone explain why? – Corrosive

Answer 3

I don't know why, but rewriting the code to include 'net/ssh' and creating the sftp connection explicitly through SSH worked!

require 'net/ssh'
require 'net/sftp'
#upload a file or directory to the remote host
Rails.logger.info("Creating SFTP connection")
session=Net::SSH.start('host','user', :password=>'password',:port=>port)
sftp=Net::SFTP::Session.new(session).connect!
Rails.logger.info("SFTP Connection created, uploading files.")
sftp.upload!("file1.txt", "./file1.txt")
Rails.logger.info("First file uploaded.")
sftp.upload!("file2.txt", "./file2.txt")
Rails.logger.info("Both files uploaded, terminating connection.")
Rails.logger.info("Connection terminated.")

Perhaps someone can shed some light on why this one worked and the other didn't.

Weanling

I had the same issue, and this work around was the only thing that resolved it. – Imparisyllabic

Answer 4

We have had often issues when passing blocks to net-sftp like so:

Net::SFTP.start(...) do |sftp|
...
end

It seems as if there was no clean shutdown of the SFTP-Connection in case of errors when using blocks.

Everything works flawlessly since we use the following pattern:

begin
  sftp = Net::SFTP.start(...)
  ... use sftp.xxx to do something ...
rescue => e
  sftp&.session&.shutdown!
end

Note: we are doing some dedicated error handling. The code presented here is for demonstration purposes and serves as a pattern. So feel free to implement your own requirements as needed.

Tachometer