Sitecore EventQueue Table growing out of control

Question

We are having an issue with the EventQueue table growing very fast at times, up to 3k records a second, and never clearing records (30 million as of right now). Our environment has the following set up:

Sitecore 7.2

• 4 CD servers and 1 CM server
• All four CD servers are load balanced.
• CD1 and CD2 are pointed to DB1 server CD3 and CD4 are pointed to DB2
• server There are 2 Publishing targets (one for each DB) Merge
• Replication is setup for the Core db across all servers (CM, CD's)
• EventQueue is enabled

I have a few questions so I will break them down into separate line items.

1. When a publish is issued for all CD servers is the updated content sent directly from the CM db to the CD db's (all of the correct tables) or is it sent to the EventQueue table in the CD db and the CD server has a job/task that looks at the table and updates as needed.

2. Depending on answer to the first question, if there are 2 CD servers pointing to the same DB how do they know if they should process the EventQueue table (wont they each process the table and be duplicating efforts)

3. Why isn't the EventTable table cleared? How is is cleared, when is it cleared?

Delly

Answer 1

1. On CM publish, the publish request is sent to the EventQueue table on the CD db where it is processed as per the instance's publishing schedule.
2. The InstanceName column in the EventQueue table stores the unique name of each Sitecore instance (by default this is Machine Name + IIS Instance Name, but can be set in web.config). This enables events to be picked up by an individual CD instance in a load balanced environment.

3. The EventQueue table is cleared by a Sitecore task defined in the <scheduling> element in the web.config, although I've seen this misbehave in the past. By default, it is set as follows:

   <agent type="Sitecore.Tasks.CleanupEventQueue, Sitecore.Kernel" method="Run" interval="04:00:00"> <DaysToKeep>1</DaysToKeep> </agent>

I've previously run into high loads on the EventQueue and PublishQueue tables and would recommend trying the following (some of which were suggested from Sitecore support):

• Reduce the interval of the CleanupEventQueue agent (above)
• Reduce the DaysToKeep setting on the CleanupEventQueue (also CleanupPublishQueue wouldn't hurt)
• Create a scheduled SQL job to run the clean up script outlined in the CMS Tuning Guide (Page 10: http://sdn.sitecore.net/upload/sitecore7/70/cms_tuning_guide_sc70-usletter.pdf)

Finally, from Sitecore support:

Sitecore recommends that the number of rows (entries) in the History, PublishQueue, and EventQueue tables would be less than 1000.

Albarran

This is explains a lot, thanks. One thing I still don't understand is if there are 2 CD servers pointing to one DB and a publish occurs, will there be and entry in the EventQueue table for each CD server only differing by the InstanceName field? If thats the case, wont they both process the same data (which would be duplicate) as they both have an action item in the table? – Delly

Sorry - I've edited my answer. The request for publish is sent to the EventQueue table on CD, with details of item/s to publish and publish type, not the item content itself. But yes, each instance gets its own row in the table IIRC. – Albarran

Thanks for this answer. The gripe I have with the EventQueue table is that it can fill up a LOT if you have scripts that run on a schedule to create/update content. The cleanup agent has a "daystokeep" setting with the finest level of granularity as a single day. There could be ~100k records added in a day. You should be able to specify a TimeToKeep setting down to minutes. – Mishmash

@DavidMasters that's fantastic news. Just gotta wait until I upgrade Sitecore now :) – Mishmash