wkhtmltopdf error when generating pdf with lots of pages with header / footer

Question

I am using pdfkit (which uses wkhtmltopdf under the hood) to generate PDFs in my rails app. Following the guide here i've gotten it mostly working for basic cases of PDFs. I am now running into an issue when attempting to generate a PDF with lots of pages thats also has headers/footers. The error I see from wkhtmltopdf in console when attempting to generate the PDF is:

QEventDispatcherUNIXPrivate(): Unable to create thread pipe: Too many open files
QEventDispatcherUNIXPrivate(): Can not continue without a thread pipe

A minimal example of the html which can be used to recreate the error:

<!-- content of pdf_header_url is the character "h" -->
<meta content="<%= pdf_header_url %>" name="pdfkit-header-html"/>
<!-- content of pdf_footer_url is the character "f" -->
<meta content="<%= pdf_footer_url %>" name="pdfkit-footer_html"/>
<% [*1..3].each do |j|%>
  <h1><%= j %></h1>
  <ul>
    <% [*1..1000].each do |i|%>
      <li><%= i %></li>
    <% end %>
  </ul>
<% end %>

Note that removing the of headers/footers tags allows the pdf to render fine.

The actual ruby code to generate the PDF is:

def view_report
  html = render_to_string(:template => 'pdf/pdf_body.html', :layout => false)
  kit = PDFKit.new(html)
  pdf = kit.to_pdf
  send_data pdf, :type => 'application/pdf', :disposition => 'inline', :filename => 'foo.pdf'
end

Visiting this controllers route will generate the PDF. And then lastly, I also have a controller for the header/footer, since those "partials" need to be fetchable via url:

class PdfController < ApplicationController

  def header
    render :layout => false
  end

  def footer
    render :layout => false
  end

end

The values of pdf_header_url and pdf_footer_url are literally just "h" and "f" for the sake of a minimal reproducible example.

Does anyone familiar with wkhtmltopdf have any recommendations on furthur debug steps to take to get around this issue?

Endways

Could you show all the code that you are using to generate the PDF? As well as the values of pdf_header_url and pdf_footer_url. I want to try to reproduce this. – Antimere

Answer 1

I was getting the same error message today and I solved the problem with a very easy solution. The problem was that my header and footer needed to be complete html documents with the html, head and body tags. Also, see if you can get the generated html output of your header and footer and validate them.

Jasik

Curious, any idea why this would fix the problem? – Bataan

I wasn't able to look into the source code, but my assumption is that any of the HTML entry points need to be complete, correct HTML documents. The body of the document is an HTML entry point and needs to be a complete well formatted HTML doc, if the header and footer behave the same way, they should be complete HTML docs also. Like I said, haven't confirmed in the source, just a strong hunch and worked on testing. – Jasik

Thanks for the response. So, I was running into this issue and both the header and footer files were complete HTML files. My issue to get passed this was upping the "ulimit -n" on the servers, similar to what @ashkulz was referring to. I'll post an answer with some more details to help others running into this problem. – Bataan

Answer 2

Open Files Limit.

I ensured my header and footer files were complete HTML documents as Tom Hirschfeld suggested, but I was still getting this error of too many open files.

After scouring the internets I discovered that you need to UP your limit of the number of files a single process is allowed to have open. In my case, I was generating PDFs with hundreds to thousands of pages. It worked fine without headers and footers but when incorporating headers and footers it seems to hit this open file ceiling.

There are different ways to adjust this setting based on the system you are running but here is what worked for me on an Ubuntu server:

Add the following to the end of /etc/security/limits.conf:

# Sets the open file maximum here.
# Generating large PDFs hits the default ceiling (1024) quickly. 
*    hard nofile 65535
*    soft nofile 65535
root hard nofile 65535 # Need these two lines because the wildcards
root soft nofile 65535 # (the * above) are not applied to the root user.

A good reference for the ulimit command can be found here.

I hope that puts some people on the right track.

Bataan

Perhaps my limit of 1024 was coming from /etc/systemd/system.conf:#DefaultLimitNOFILE=1024:524288. For me, ulimit -n 10001 was sufficient. Thanks for setting me on the right track. – Stupid

Answer 3

wkhtmltopdf uses 2 file descriptor per page (one each for header and footer) which are required for generating the per-page custom variables. You'll have to edit /etc/security/limits.conf to set nofile (i.e. maximum no of open files) to a suitably high number -- some experimentation may be required to find a value which works for you.

Selfinduced

Thanks for the pointer on this. I added another answer with some more details on how I solved it specifically. Just wanted to give a quick thumbs up to you. – Bataan

Answer 4

Here is a solution for Mac: https://github.com/uncss/uncss/issues/106#issuecomment-64768667

it works for me.

Theological