How to send a csv attachment with lines longer than 990 characters?

Question

Alright. I thought this problem had something to do with my rails app, but it seems to have to do with the deeper workings of email attachments.

I have to send out a csv file from my rails app to a warehouse that fulfills orders places in my store. The warehouse has a format for the CSV, and ironically the header line of the CSV file is super long (1000+ characters).

I was getting a line break in the header line of the csv file when I received the test emails and couldn't figure out what put it there. However, some googling has finally showed the reason: attached files have a line character limit of 1000. Why? I don't know. It seems ridiculous, but I still have to send this csv file somehow.

I tried manually setting the MIME type of the attachment to text/csv, but that was no help. Does anybody know how to solve this problem?

Some relevant google results : http://www.google.com/search?client=safari&rls=en&q=csv+wrapped+990&ie=UTF-8&oe=UTF-8

update

I've tried encoding the attachment in base64 like so:

    attachments['205.csv'] = {:data=> ActiveSupport::Base64.encode64(@string), :encoding => 'base64', :mime_type => 'text/csv'}

That doesn't seem to have made a difference. I'm receiving the email with a me.com account via Sparrow for Mac. I'll try using gmail's web interface.

Serviette

Are you able to post the code for your ActionMailer (and the CSV data generation)? I've tried using CSV.generate to build lines > 1000 characters, added the resulting data to the attachments part of an ActionMailer and sent it, but it didn't reproduce your problem. – Delilahdelimit

Unfortunately I still can't reproduce the problem, even with your code. If you just save the CSV instead of emailing it are the linebreaks still there? Also, what mail server, mail client and CSV viewer are you using? – Delilahdelimit

I'm using SendGrid services through Heroku. In terms of viewing the csv file, I've used Numbers as well as simple text editors. The line break is definitely there in the file. – Serviette

If I write the csv to a file directly from my local rails console, the output is fine. So it seems there's not a problem with the CSV generation itself. – Serviette

Answer 1

This seems to be because the SendGrid mail server is modifying the attachment content. If you send an attachment with a plain text storage mime type (e.g text/csv) it will wrap the content every 990 characters, as you observed. I think this is related to RFC 2045/821:

6. Content-Transfer-Encoding Header Field

   Many media types which could be usefully transported via email are represented, in their "natural" format, as 8bit character or binary
   data. Such data cannot be transmitted over some transfer protocols.
   For example, RFC 821 (SMTP) restricts mail messages to 7bit US-ASCII
   data with lines no longer than 1000 characters including any trailing CRLF line separator.

   It is necessary, therefore, to define a standard mechanism for
   encoding such data into a 7bit short line format. Proper labelling
   of unencoded material in less restrictive formats for direct use over less restrictive transports is also desireable. This document
   specifies that such encodings will be indicated by a new "Content-
   Transfer-Encoding" header field. This field has not been defined by
   any previous standard.

If you send the attachment using base64 encoding instead of the default 7-bit the attachment remains unchanged (no added line breaks):

attachments['file.csv']= { :data=> ActiveSupport::Base64.encode64(@string), :encoding => 'base64' }

Delilahdelimit

Right, I did come across that documentation. Let me try your solution… – Serviette

I've used the line you suggested to add the attachment, but the resulting file isn't decoded when I receive it – it's still a sequence of base64 values. – Serviette

What email server are you sending to, and are you using OSX mail.app to read the emails? It sounds like mail.app has issues decoding some base64 attachments but I don't have access to OSX at the moment to test it. Before posting I tested sending the attachment with ActionMailer via heroku/SendGrid to gmail/hotmail/yahoo/smartermail accounts and they all decoded the attachment correctly. Are you sending text in the email or just the attachment? Also did you definitely add the :encoding => 'base64' part? – Delilahdelimit

Don't add the :mime_type => 'text/csv' (as you've done in the question update), that will set the :encoding value back to 7-bit so the attachment won't get decoded. – Delilahdelimit

There it is, baby. Thank you! The attachment is coming through fine now! (I'll award the bounty once the time limit is hit. 3 hrs from now) – Serviette

Amazing! I had the exact same problem. Also emailing CSV files on Heroku to our warehouse using SendGrid. Seems to work great now. Thanks! – Appreciative

@Delilahdelimit , Here, we create the emails manually, we manually add the content boundary and the encodings/content types, what I've tried is to encode the csv content with base-64 and Content-type => application/octet-stream and the ;charset => base-64 but yet its not working, now the email's attachment is encoded and the csv file is also encoded. do you have any other hints that could help, Thanks alot – Taoism

I'm having @Serviette issue where it mails but it returns encoded base64 and its illegible. I'm having the CSV generated as a file, and then I encode it. – Frigidarium

I've tried hundreds of different ways of trying to send a large CSV attachment and none of them worked.. do you guys know of any working examples of such besides the one listed here? I'm only returned a blank email with a noname'd file attachment that reads This is a multi-part message in MIME format... – Frigidarium

Answer 2

Could you have newlines in your data that would cause this? Check and see if

csv_for_orders(orders).lines.count == orders.count

If so, a quick/hackish fix might be changing where you call values_for_line_item(item) to values_for_line_item(item).map{|c| c.gsub(/(\r|\n)/, '')} (same for the other line_item calls).

Unitarian

I tried something along these lines already, but let me try your specific examples and see if they come up with anything. Thanks! – Serviette

Yeah, it looks like the csv checks out fine. The line count is correct. – Serviette