How to get the content of a remote file without a local temporary file with fabric

Question

I want to get the content of a remote file with fabric, without creating a temporary file.

Fancy

Answer 1

from StringIO import StringIO
from fabric.api import get

fd = StringIO()
get(remote_path, fd)
content=fd.getvalue()

Fancy

I get: _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? as an error, any thoughts? – Roccoroch

Answer 2

With Python 3 (and fabric3), I get this fatal error when using io.StringIO: string argument expected, got 'bytes', apparently because Paramiko writes to the file-like object with bytes. So I switched to using io.BytesIO and it works:

from io import BytesIO

def _read_file(file_path, encoding='utf-8'):
    io_obj = BytesIO()
    get(file_path, io_obj)
    return io_obj.getvalue().decode(encoding)

Ectoderm

Nice find! I was hitting the same issue. Really need to get my mind into bytes and string difference now that I ported my code to PY3. :) – Speciation

Answer 3

import tempfile
from fabric.api import get
with tempfile.TemporaryFile() as fd:
    get(remote_path, fd)
    fd.seek(0)
    content=fd.read()

See: http://docs.python.org/2/library/tempfile.html#tempfile.TemporaryFile

and: http://docs.fabfile.org/en/latest/api/core/operations.html#fabric.operations.get

Fancy

Note that this will, in fact, be backed by a temp file on disk. Just that the temp file will be deleted on context exit – Wether

Answer 4

I also didn't want to store a temp file locally, but had a different approach.

Fabric exposes Secure File Transfer Protocol (SFTP) from the lower level library paramiko.

Following the same strategy as this article, I swapped out parimiko for fabric with a little re-work.

class remote_operations:
  def __init__(self):
    pass

  def create_connection(self, hostname, username, kwarg_password):
    connection = fabric.connection.Connection(
      host=hostname, user=username,
      connect_kwargs=kwarg_password)
    return connection

  def open_remote_file(self, ssh_client:Connection, filename):
    sftp_client = ssh_client.sftp()
    file = sftp_client.open(filename)
    return file

And utilized it like so using a dictionary called values that had my host, username, and password.

test = remote_operations()
client = test.create_connection(
    hostname=values.get('remote_host'),
    username=values.get('ssh_username'),
    kwarg_password={"password": values.get('ssh_password')})

file = test.open_remote_file(client, "/path/to/file.txt")

for line in file:
   print(line)

file.close()
client.close()

Tipi