Is my mysql.general_log table getting too big?

Question

I just recently upgraded to MySQL 5.1.6 in order to take advantage of the ability to save the general log to a table -> i.e. mysql.general_log. Once i did this I was immediately surprised how many queries are actually hitting our system. I have about 40,000 rows in this general log table from the first hour. I haven't found it written on the MySQL docs about if there is a general log table size limit.

Is there a problem to letting this general log grow at this rate?

If there is a size problem, how to deal with it?

Are there some accepted practices how to deal with a size problem if there is one?

Should I make an event to purge the table and save the data to a file every so often?

Thanks a lot for the help!

Reservation

One thing you should be aware of, is that running general query log in production environment is not recommended. The performance impact can be pretty high, especially in high concurrency situations. – Enroll

Answer 1

The general_log table by default uses the CSV engine, which is literally just a full-blown CSV file on your drive, but can be accessed via SQL. This means its size limit is the size limit of files on your file system.

Parallel

Answer 2

I do something like this for my log file. I'm only interested in keeping the last 24 hours, but you could tweak the event to create archive tables, etc. It won't log for the few seconds it takes the event to run, but I don't mind.

CREATE EVENT `prune_general_log` ON SCHEDULE
EVERY 1 DAY STARTS '2013-10-18'
ON COMPLETION NOT PRESERVE
ENABLE
COMMENT 'This will trim the general_log table to contain only the past 24 hours of logs.'
DO BEGIN
  SET GLOBAL general_log = 'OFF';
  RENAME TABLE mysql.general_log TO mysql.general_log2;
  DELETE FROM mysql.general_log2 WHERE event_time <= NOW()-INTERVAL 24 HOUR;
  OPTIMIZE TABLE general_log2;
  RENAME TABLE mysql.general_log2 TO mysql.general_log;
  SET GLOBAL general_log = 'ON';
END

Tannenwald

Answer 3

You should use some utility like mysql-log-rotate http://dev.mysql.com/doc/refman/5.0/en/log-file-maintenance.html for rotating log file.

Terle

Unfortunately I am running on a Windows server, it says in the documentation this is included on Linux (Red Hat) installations. – Reservation

Answer 4

Not sure if this is best practice but this was my solution:

DATE=$(date +"%Y%m%d%H%M")
mv general_log.CSV general_log.${DATE}.csv  # move the log table file
sudo -u mysql -g mysql touch general_log.CSV  # create a new log table file with correct owner and group
mysql -u root -e "FLUSH TABLE mysql.general_log"  # flush the log table

Cannibal