NserviceBus scheduler stops executing job

Question

One of our agents has a job that runs every 5 seconds, which seems to stop for no apparent reason. There are no exceptions being thrown, another scheduled task, a heartbeat that confirms that the agent is running correctly, still works, and there are no error messages in the windows event log. The scheduled task runs correctly for a few days, but then stops for no apparent reason, as if the scheduler forgets to start the task again.

I have no debug information, since there are none, so my question is if anyone else has experienced this? Can it be caused by a scheduled task running for too long, or taking up too many resources? As in, the scheduled tasks don't terminate correctly (for some reason) and then the scheduler being unable to allocate a thread to start a new task?

There is a parallel discussion about this on the NServiceBus Google groups: https://groups.google.com/forum/#!topic/particularsoftware/0GAI9oqmp74

Unfinished

I have experienced the same problem twice in the last couple of months. Would appreciate if someone comments on possible causes. – Farly

One possible cause is that the system time is set, either manually or automatically. If you change the time on the system to beyond a scheduled task, then it does not correctly trigger the task (since that time never occurred on the system). – Unfinished

So as I understand, if this happens once, the task will not be executed any more until endpoint is restarted? Still I think this is likely not the cause of our problem. At least on our system, the probability of significant time adjustment was close to 0. – Farly

I ran an experiment on the server but could not reproduce it. But we have seen a problem when the clock is manually adjusted (for testing purposes), but it might not be related. – Unfinished

Marius, I understand that it's not throwing an exception or terminating.. but rather just not doing what it's supposed to.. Would you be able to increase your logging level in hopes of getting some insight of what it is doing? docs.particular.net/nservicebus/logging-in-nservicebus Joe – Bonnybonnyclabber

looking at the source code if something goes wrong there should be an exception logged, one or more, at the end of the task execution, so as Joe said can we start trying to increase the log information that the endpoint outputs. In order to dig deeper into the issue can you supply more information regarding the environment? * The Transport used * if the "Schedule" is stored in memory or in a persistent storage, in this second case which one; Thanks in advance, .m – Repertoire

Answer 1

A bug has been filed in the NServiceBus repo: https://github.com/Particular/NServiceBus/issues/2133

Unfinished