Piping local stdin to 'heroku run' - mustn't write to stdout? 4k stdout buffer?

Question

I'm using thor to write command-line tasks (in a rails 3 app) and trying to pipe input to them over heroku, e.g. cat somefile.csv | heroku run thor sometask

works great on my local machine cat somefile.csv | thor sometask but can fail on heroku. Some versions of sometasks work occasionally, other ones always fail at a specific point -- 1 issue is that the cat somefile.csv is being echoed to the console while it is being consumed by the thor sometask (this only happens over heroku, so I presume this is internal to heroku CLI).

I have found a workaround, which is to ensure that I'm writing nothing to the console (stdout) in the thor tasks & the functions they call, but it's frustrating to have to do that (no progress indication etc).

Another "workaround" is to ensure that the file being sent is under 4k (4095bytes) - I haven't been able to make it fail with this

Any ideas why this is happening (and why the cat is being echoed to stdout)? Is there a 4k stdout buffer limit for example somewhere in communication process? Any way to make the heroku CLI less verbose?

Ceric

Answer 1

If you use your workaround of "writing nothing to the console" and instead of cat use pv, you'll be able to see a progress indicator.

This is what the output might look like:

$ pv somefile.csv | heroku run thor sometask
3.12MiB 0:00:00 [25.9MiB/s] [===============================================================================================================>] 100%            

Forelimb