Config rsyslog with rails elastic beanstalk

Question

I config remote loging for rails application with elastic beanstalk. I want to get logs from /var/log/puma/puma.log but only get some kernel and system informations. This is my config .ebextensions/papertrail.config

packages:
  yum:
    rsyslog: []
    rsyslog-gnutls: []
files:
  "/etc/rsyslog.d/01-udp.conf":
    mode: "000640"
    owner: root
    group: root
    content: |
      $ModLoad imudp
      $UDPServerRun 514
  "/etc/rsyslog.d/02-papertrail-tls.conf":
    mode: "000640"
    owner: root
    group: root
    content: |
      $DefaultNetstreamDriverCAFile /etc/papertrail-bundle.pem # trust these CAs
      $ActionSendStreamDriver gtls # use gtls netstream driver
      $ActionSendStreamDriverMode 1 # require TLS
      $ActionSendStreamDriverAuthMode x509/name # authenticate by hostname
      $ActionSendStreamDriverPermittedPeer *.papertrailapp.com
  "/etc/rsyslog.d/03-logfile-config.conf":
    mode: "000640"
    owner: root
    group: root
    content: |
      $ModLoad imfile
      $InputFileName /var/log/puma/puma.log
      $InputFileTag api
      $InputFileStateFile api-staging
      $InputFileSeverity error
      $InputFileFacility local3
      $InputRunFileMonitor
  "/etc/rsyslog.d/04-papertrail.conf":
    mode: "000640"
    owner: root
    group: root
    content: |
      $LocalHostName api-staging

container_commands:
  01_copy_ca_certs:
    command: 'cp ./.ebextensions/papertrail-bundle.pem /etc/papertrail-bundle.pem'
  02_install_rsyslog_config:
    command: '/bin/echo "*.* @${SYSLOG_HOST}" >> /etc/rsyslog.d/04-papertrail.conf'
  03_restart_rsyslog:
    command: 'sudo service rsyslog restart'

But I only get some info on papertrail like these:

Nov 03 21:28:00 api-staging kernel:  imklog 5.8.10, log source = /proc/kmsg started.
Nov 03 21:28:00 api-staging rsyslogd:  [origin software="rsyslogd" swVersion="5.8.10" x-pid="32340" x-info="http://www.rsyslog.com"] start
Nov 03 23:50:41 api-staging kernel:  Kernel logging (proc) stopped.
Nov 03 23:50:41 api-staging rsyslogd:  [origin software="rsyslogd" swVersion="5.8.10" x-pid="32340" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 04 00:51:56 api-staging kernel:  imklog 5.8.10, log source = /proc/kmsg started.
Nov 04 00:51:56 api-staging rsyslogd:  [origin software="rsyslogd" swVersion="5.8.10" x-pid="15883" x-info="http://www.rsyslog.com"] start
Nov 04 00:53:42 api-staging kernel:  Kernel logging (proc) stopped.

Please help me if you have experience for this issue. Thank you!

Oke

I'm experiencing this too, and I have no idea why! Very frustrating. I've put a bounty on it, hopefully we can get an answer, or if you found one @Oke then you can maybe update. – Paleography

@KevinWillock Thank you for your bounty. I will update and notice you when get or find a solutions. – Oke

Have you tried making it work with unencrypted logging first? – Bilbao

Answer 1

There are a number of potential problems you could be encountering, too many to enumerate here. Your approach is complicated significantly by the inclusion of file access in the logging system. Simply trying to scrape a log file (puma.log) is fragile at best and depending on your deployment environment could result in many possible obscure failures.

My suspicion is that the logging data is not being delivered to the logging daemon at all, therefore never getting delivered to PaperTrail.

I suggest you reconfigure your application to communicate directly with the logging daemon via UDP as explained here:

https://www.thoughtworks.com/mingle/infrastructure/2015/06/10/simple-solution-for-papertrail-on-elasticbeanstalk.html

Once you have made this change I suspect your problem will resolve. At the very least it will be come easier to troubleshoot, and will be more reliable in the future.

Populous