Transfer large file via SFTP in PHP

Question

I have a large file (200 MB upwards). I need to transfer it via PHP cron job. Using Phpseclib gives the following error:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 4133 bytes) in /app/vendor/phpseclib/phpseclib/phpseclib/Net/SSH2.php

Is there a way I can do this with PHP cron job?

The code is simple one line where $localFile is an already existing CSV file

$sftp->put('/Import/coupons/coupons_import_test.csv', $localFile, NET_SFTP_LOCAL_FILE);

PS. This needs to be done after PHP generates that file in the /tmp folder so timing of the transfer script has to come into play.

[Edit] I do not intend on increasing the memory limit as the files later could be of higher sizes. A solution where I can transfer the file in parts (append mode) or use some shell script with PHP cron could be worthwhile

The file size on remote server is 111.4 MB while the actual file is much larger on local.

[Edit after the fix] The issue disappeared after changing to version 2.0.2 from version 1.0 I had to modify the code for put

$sftp->put('/Import/coupons/coupons_import.csv', $localFile, $sftp::SOURCE_LOCAL_FILE);

Barocchio

Possible duplicate of Allowed memory size of 33554432 bytes exhausted (tried to allocate 43148176 bytes) in php – Medievalist

What does your php.ini have as memory_limit? Usually something like 64M or 128M, you need to increase this. – Offhand

Is there a way to do this without increasing the limit? – Barocchio

What is your version of Phpseclib? – Cercus

you can do it with a shell script in more simple way, you dont need php and it's limitations. if you consider making sh i can help with a proper answer – Firearm

I asked this before and didn't get an answer so I'll ask it again (since you just posted the bounty): can you post some code? Also, how much of the file is uploaded by the time you get the memory error? Also, Pawel Dubiel's idea is a good one too so try that and then, if not, maybe answer my question. – Spaceport

Version of Phpseclib is 1.0. I have edited question for code ( single line of code). – Barocchio

Maybe try open an issue here github.com/phpseclib/phpseclib and post logs over there. – Demodulator

Can you also try to transfer this file to different server to see if it's the same problem? – Demodulator

I think you should see this. Hope this helps unix.stackexchange.com/questions/190537/… – Autocratic

Answer 1

Phpseclib should do just fine transferring big files with no need to increase available memory.

I think you probably hit the old bug "SSH2: don't count data length against window size". You most likely may be using older version of the Phpseclib ( The older faulty version is bundled even with relatively new software like for example Magento 1.9.* )

Check your version if is not the latest redownload it from https://github.com/phpseclib/phpseclib

Demodulator

I am using phpseclib 1.0 by composer require on a heroku set up. – Barocchio

I don't know if it was that bug in version 1.0 which you pointed out which was causing the issue, but when I changed to version 2.0.2 the issue disappeared. I though have to modify the code a bit. – Barocchio

@aeonsleo - What version of 1.0 were you using? Seems like composer update might have also fixed the issue if 2.0.2 did.. – Spaceport

Answer 2

In lieu of seeing the code that you're using my guess is that you're trying to load the 200mb file as a string and you're trying to upload it as a string. eg. $sftp->put('filename.remote', file_get_contents('filename.local'));. If so then try this instead:

$sftp->put('filename.remote', 'filename.local', NET_SFTP_LOCAL_FILE);

Spaceport

I am doing exactly the way you have mentioned. – Barocchio

To comment further I'd need to see some code. You can edit it into your orig post. Also, how much of the file is uploaded by the time you get the memory error? – Spaceport

Answer 3

This actually changed a bit with phpseclib version >= 2.0.5. You should include the namespace use phpseclib\Net\SFTP; , then do something like this:

$sftp->put('filename.remote', 'filename.local', SFTP::SOURCE_LOCAL_FILE);

Rabe