How do I set up PHP Logging to go to a remote server?

Question

We are going to a deployment setup where we will have many servers, most of which auto-added to the load balancer when the traffic goes up. The trouble with this setup is that if an individual developer needs to tail logs to troubleshoot something, he will have to open a console on each server, which is made complicated by the fact that the developer often doesn't know HOW many servers we might have in function at that time.

If the developer can find all the logs in one server - say our deploy server, then the troubleshooting becomes easier.

Towards this, I was thinking of setting up a push from each FE machine to the deploy server using a cron which will replicate the logs on our deploy server. There are two problems with this approach:

• There is a lag of 1 minute because crons can't be run more frequently.
• The cron on each FE machine will have to set up to sync to a specific location on the deploy server, but up front, I don't know how many such FE servers will exist.

To solve this problem, I am looking at a way to connect error_log, or PEAR Log to send the logs directly to our deploy server, which will log it in real time to it's local locations at /var/log/..

Anybody knows how I can configure this? Or perhaps a service which accomplishes this?

Our servers are all Ubuntu 10.04 LTS and we are running these servers on EC2 instances in AWS. Our PHP version is 5.3.

Epidemic

I don't know how many such FE servers will exist. But you have to put your PHP code on them. So, when you do that, set up the cron. – Totalitarianism

There is a lag of 1 minute because crons can't be run more frequently. Is that really a problem? Is your developer going to need to see the logs within a minute of the problem occuring? Is he even ever going to be able to react that quickly? – Totalitarianism

The servers are auto created from an EC2 AMI that we have and are spawned and added to the load balancer automatically. If we have to stop for a manual step in between, that makes the process error-prone and slow. – Epidemic

The PHP code itself is fetched by each of these auto-created servers onto themselves from the deploy server. The AMI has been pre-created to have the right ssh keys to communicate to the deploy server. – Epidemic

Have the auto-created servers fetch the cron config, too. – Totalitarianism

@LightnessRacesinOrbit - Imagine this - I know each server has to get PHP code from /usr/lib/php5 and dump it on their own /usr/lib/php5. But when s1, s2, s3 want to push their /var/log/nginx/*.log to deploy, where do they push it to? deploy:/var/log/nginx/? No, because each server will override other server's logs. – Epidemic

So, s1 perhaps needs to log to deploy:/var/log/nginx/s1, then "s1" becomes a config specific to s1. When s2 is deployed, it has to be manually configured to go to /var/log/nginx/s2 and so on. This breaks the fact that all servers can be exact mirror copies of one another. – Epidemic

Well of course not. Why not have a directory based on the server's hostname? I find it difficult to believe that there isn't some unique identifier that you can use. – Totalitarianism

let us continue this discussion in chat – Epidemic

Answer 1

The answer from @MichelFeldheim was the genesis, but we improved upon it to handle multiple applications writing to multiple log files.

---

Central Logging Server

In the central logging server, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf:

destination d_php { file("$PROGRAM" owner(www-data) group(www-data) perm(0644)); };
filter f_php { program("^\/var\/log\/"); };
log { source(s_all); filter(f_php); destination(d_php); flags(final); };

source s_all {
        # ....
        # .... LET THE PREVIOUS CONTENT STAY - add the following line
        tcp(port(5140) keep_alive(yes));
};

restart syslog service:

sudo service syslog-ng restart

---

On FE Servers

On each of the FE Servers, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf on each of the FE servers:

destination php { tcp("log.example.com" port(5140)); };
log { source(s_all); filter(f_php); destination(php); };
filter f_php { facility(user); };

restart syslog servers:

sudo service syslog-ng restart

---

Application Code Changes:

Now, the application code can be changed thus. Suppose each of the application have code like this writing to a separate file and you want the same structure to be reflected in the central log server:

// PREVIOUS CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('file', '/var/log/nginx/xxx.log', '', array(), PEAR_LOG_INFO);
// PREVIOUS CODE: Using error_log
ini_set('error_log' , '/var/log/nginx/xxx.log');

The new code should look like:

// NEW CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('syslog', LOG_USER, '/var/log/nginx/xxx.log', array(), PEAR_LOG_INFO);
// NEW CODE: Using error_log
ini_set(‘error_log’, ‘syslog’);
openlog('/var/log/nginx/xxx.log', LOG_NDELAY, LOG_USER);

If your FE servers and the Logging servers are all within the same EC2 security group, then there is no need to open the ports, since within the groups, all ports can be accessed freely, so long as a service is listening to it.

This approach allows your each of your apps, modules to decide whether they want central logging or not.

Epidemic

Hey you stole my green tick :D thanks for the precise implementation example – Barony

Answer 2

What you could do is to log into a custom syslog channel which writes into a central logging server.

php.ini

error_log = syslog

syslog-ng.conf on the php boxes

destination php { tcp("10.10.10.10" port(5140)); };
log { source(src); filter(f_php); destination(php); };

This would send all php logging to a box 10.10.10.10 where syslog-ng is listening on port 5140.

On your logging box you have to open the port 5140 in the ec2 security group

Here's a good tutorial on how to setup a syslog server

http://praxis.edoceo.com/howto/syslog-ng

EDIT: This of course would also make it possible to log other important log sources of your php boxes to the log server as well.. Thinking of traffic logs, system logs etc. etc.

Barony

Did you have success with this approach? – Barony

Sorry, got pulled out for an offsite event for past few days. Let me try this today and post an update. – Epidemic

We have a basic prototype working, so that's good news. I want to transfer the ini_get('error_log') setting at the time of logging to syslog-ng and onto the central logging server - so that I reproduce the same file structure in logging server. Any ideas how to do it? – Epidemic

We have read that $PROGRAM can be a security threat, but we open up the syslog-ng port only internally and hence I believe it is low risk. Any comments on this? – Epidemic

@Epidemic suppose you are talking about the syslog-ng macro $PROGRAM. Can't tell you about security. What I know, is that this is set by the logging program rather than the syslog server itself, so theoretically the logging program could write what ever it wants. you control the writing program (e.g. php) so this should not be an issue - haven't used $PROGRAM by myself yet. Concerning your edit above: thanks, but I think this will be rejected(by other voters:3/5 rejected it), add another answer instead. – Barony

@MichelFeldheim: Thanks for your comment as well as the suggestion to add a new answer. I will do that. – Epidemic