How to resolve this error? "RestartFreqExceeded: 5 in 1s" in django+celery+rabbitmq+mysql+redis

Question

So I am using django with celery. rabbitmq is the broker. redis is the cache. mysql is the db. (everything in localhost)

1. I am using python2.7 and using virtualenv based virtual environment
2. I start the redis server (local) at default port

3. In a new terminal, I run

   python manage.py runserver
   

4. In a new terminal I start celery like so

   celery -A ds_django worker -B -l warning
   

And this procedure used to work. But now when I run the celery command, I get the following

    [2016-07-12 09:15:20,113: CRITICAL/MainProcess] Frequent restarts detected: RestartFreqExceeded('5 in 1s',)
    Traceback (most recent call last):
        File "/Users/user/Desktop/ds-django/ds_django/newDs/lib/python2.7/site- packages/celery/worker/consumer.py", line 285, in start
            self._restart_state.step()
        File "/Users/user/Desktop/ds-django/ds_django/newDs/lib/python2.7/site-packages/billiard/common.py", line 130, in step
            raise self.RestartFreqExceeded("%r in %rs" % (R, self.maxT))
     RestartFreqExceeded: 5 in 1s

I have absolutely no clue why this is happening or how to fix this. I have searched Google and StackOverFlow over and over again, but no use.

One reason why I think it might have happened is, sometime back I had run 4 celery workers at-a-time and then cold-shut them all down. Maybe because of that it is saying restart frequency exceeded, but not sure. And no idea how to fix this. Probably some config file setting, but which setting and where also no clue.

If any other info is required, I will provide. Any help is appreciated. Thank you.

Selfstyled

I suspect that this is because of the RabbitMQ connectivity issue ? Can you check accessing RabbitMQ – Trisyllable

I used to access rabbitmq console using localhost:15672/#/queues. Now when I try it says "undefined: There is no template at js/tmpl/login.ejs" @Backtrack – Selfstyled

which means, Your RabbitMQ is down. Please start it and give a try – Trisyllable

Any update ? Did you try after starting RabbitMQ ? – Trisyllable

Answer 1

To answer your question, Your RabbitMQ is down due to that consumer started to reconncet and it went to contineous loop and it created an exception RestartFreqExceeded. Please try starting your RabbitMQ server and run your celery.

Trisyllable

I have been trying that @Backtrack. Same error still. I uninstalled and installed rabbitmq (using homebrew). No use. Still the same error – Selfstyled

@RajJain, Are you able to access RabbitMQ ? – Trisyllable

@RajJain, Are you giving localhost to connect of remote ip in the consumer ? – Trisyllable

I get this when I run rabbitmqctl status [{pid,3641}, {running_applications,[{os_mon,"CPO CXC 138 46","2.4"}, {ssl,"Erlang/OTP SSL application","7.3"}, {mnesia,"MNESIA CXC 138 12","4.13.3"}, {amqp_client,"RabbitMQ AMQP Client","3.6.1"}, {public_key,"Public key infrastructure","1.1.1"}, {asn1,"The Erlang ASN1 compiler version 4.0.2", "4.0.2"}, {syntax_tools,"Syntax tools","1.7"}, and so on @Backtrack – Selfstyled

How are you connecting from consumer ? Are you using localhost or remote IP ? – Trisyllable

i am using localhost – Selfstyled

@RajJain, Are you running the celery task from same machine ? – Trisyllable

yes @Backtrack. everything is local and at localhost – Selfstyled

update: I run brew services start rabbitmq, it says already started. when I run rabbitmqadmin list queues it says connection refused. when I go to management console, it says unable to connect to it – Selfstyled

rabbitmq-server ERROR: node with name "rabbit" already running on "localhost" and then localhost:15672/#/queues, it says "undefined: There is no template at js/tmpl/login.ejs" again – Selfstyled

@RajJain Which means you are still having RabbitMQ connection issue. Which OS are you suing ? – Trisyllable

Mac OS X @Backtrack – Selfstyled

@RajJain - Can you run this : rabbitmqctl status and check the status. Please follow this :rabbitmq.com/install-standalone-mac.html – Trisyllable

I did that. i reinstalled again. And then when i run rabbitmq-server stops at Starting broker... for almost forever @Backtrack – Selfstyled

Please check the above link – Trisyllable

The answer is correct. there was some problem with my rabbitmq. I uninstalled rabbitmq, deleted ALL files regarding it (with the help of my senior), then reinstalled and it worked :) but the above link was not so useful :) @Backtrack – Selfstyled

@RajJain, Consider upvoting :) glad it could help – Trisyllable

i did! but apparently due to less than 15 rep, it won't show in public lol. when my rep becomes more than 15, it will show. So fingers-croosed :) – Selfstyled

@RajJain, :). lol. Plz follow this 2 links, We are using celery in PRD and the follwoing links will be useful. denibertovic.com/posts/celery-best-practices blog.balthazar-rouberol.com/celery-best-practices – Trisyllable

Nice!! Thanks @Backtrack – Selfstyled

I'm seeing this error and my RabbitMQ server is just fine. – Weeds

@Weeds - When you say the Rabbitmq is running fine - How did you verify that ? Are you running both Celery and Rabbit on the same server ? – Trisyllable

I saw a recent down vote - dear down voters please leave your message. I think i have helped OP to resolve this issue. If my answer did not help you please create one. I am happy to help you. – Trisyllable