How to implement paho python client in Django 3.1

Question

from this SO question, I implemented a subscriber client in my Django project as following:

in mqtt.py, I create a client and connect to a local broker and subscribe to a topic.

#myapp/mqtt.py:
import paho.mqtt.client as paho
import json
import django
django.setup()
from .models import Temperature, Turbidity, Refined_fuels, Crude_oil
from datetime import datetime

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("CONNACK received with code %d." % (rc))
    client.subscribe("sensor/temp", qos=0)


def on_subscribe(client, userdata, mid, granted_qos):
    print("Subscribed: "+str(mid)+" "+str(granted_qos))

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    print(msg.topic+", "+'QOS: ' +str(msg.qos)+",\n"+str(msg.payload, 'utf-8'))
    message_dict = json.loads(msg.payload)
    
    now = datetime.now()
    captured_timestamp = datetime.utcfromtimestamp(int(message_dict['Timestamp'])).strftime('%Y-%m-%d %H:%M:%S')
    print('timestamp: ', captured_timestamp )
    if message_dict['Temperature'] and message_dict['D850'] and message_dict['D280']:
            
        temp = Temperature(captured_timestamp=captured_timestamp, data=message_dict['Temperature'], received_timestamp=now)
        temp.save()
        refined_fuels = Refined_fuels(captured_timestamp=captured_timestamp, data=float(message_dict['D850']), received_timestamp=now)
        refined_fuels.save()
        crude_oil = Crude_oil(captured_timestamp=captured_timestamp, data=float(message_dict['D280']), received_timestamp=now)
        crude_oil.save()

# defining client
client = paho.Client(client_id="testSubscriber",
                     clean_session=True, userdata=None,
                     protocol=paho.MQTTv311)

# adding callbacks to client
client.on_connect = on_connect
client.on_subscribe = on_subscribe
client.on_message = on_message

client.connect(host="localhost", port=1883, keepalive=60,
               bind_address="" )

in __init__.py I call the loop_start() of client:

# myapp/__init__.py
from . import mqtt

mqtt.client.loop_start()

For publisher client I used paho C client. and for broker I used hivemq-4.5.1 enterprise trial version. And, I'm running the project on ubuntu 18.04.

now, When I run the Django server(python manage.py runserver), it keeps calling the on_connect() method and continues on_connect() method, But the server is not running and I cant access the project from localhost:8000.

this is the django error before the on_connect() and on_subscribe() methods keep printing their messages:

Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/shahriar/webapp/venv/lib/python3.8/site-packages/django/apps/registry.py", line 94, in populate
    raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: graphs

and this is the broker log that is ok and shows no error:

2021-03-01 10:06:38,059 INFO  - Sent PUBLISH to client 'testSubscriber' on topic 'sensor/temp': Payload: '{ "Timestamp": 1609446782, "Temperature": "30.13", "D850": "102.48", "D280": "4845.83" }', QoS: '0', Retained: 'false'
2021-03-01 10:06:38,075 INFO  - Received CONNECT from client 'testSubscriber': Protocol version: 'V_3_1_1', Clean Start: 'true', Session Expiry Interval: '0'
2021-03-01 10:06:38,207 INFO  - Sent CONNACK to client 'testSubscriber': Reason Code: 'SUCCESS', Session Present: 'false'
2021-03-01 10:06:38,209 INFO  - Received SUBSCRIBE from client 'testSubscriber': Topics: { [Topic: 'sensor/temp', QoS: '0'] }
2021-03-01 10:06:38,209 INFO  - Sent SUBACK to client 'testSubscriber': Suback Reason Codes: { [Reason Code: 'GRANTED_QOS_0'] }
2021-03-01 10:06:39,056 INFO  - Received PUBREL from client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:39,057 INFO  - Sent PUBCOMP to client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:39,131 INFO  - Received CONNECT from client 'testSubscriber': Protocol version: 'V_3_1_1', Clean Start: 'true', Session Expiry Interval: '0'
2021-03-01 10:06:39,158 INFO  - Received PUBLISH from client 'testPublisher' for topic 'sensor/temp': Payload: '{ "Timestamp": 1609446783, "Temperature": "30.13", "D850": "102.48", "D280": "4845.83" }', QoS: '2', Retained: 'false'
2021-03-01 10:06:39,161 INFO  - Sent PUBREC to client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:39,200 INFO  - Sent CONNACK to client 'testSubscriber': Reason Code: 'SUCCESS', Session Present: 'false'
2021-03-01 10:06:39,201 INFO  - Received SUBSCRIBE from client 'testSubscriber': Topics: { [Topic: 'sensor/temp', QoS: '0'] }
2021-03-01 10:06:39,203 INFO  - Sent SUBACK to client 'testSubscriber': Suback Reason Codes: { [Reason Code: 'GRANTED_QOS_0'] }
2021-03-01 10:06:40,162 INFO  - Received PUBREL from client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:40,162 INFO  - Sent PUBCOMP to client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:40,262 INFO  - Received PUBLISH from client 'testPublisher' for topic 'sensor/temp': Payload: '{ "Timestamp": 1609446784, "Temperature": "30.13", "D850": "102.48", "D280": "4845.83" }', QoS: '2', Retained: 'false'
2021-03-01 10:06:40,263 INFO  - Sent PUBREC to client 'testPublisher': Reason Code: 'NO_MATCHING_SUBSCRIBERS'
2021-03-01 10:06:41,262 INFO  - Received PUBREL from client 'testPublisher': Reason Code: 'SUCCESS'
2021-03-01 10:06:41,263 INFO  - Sent PUBCOMP to client 'testPublisher': Reason Code: 'SUCCESS'

now the question is that, is the implementation of paho python client in Django wrong?

Since I wanted to store messages in DB, I imported the models in mqtt.py and that caused an error:

raise AppRegistryNotReady("Apps aren't loaded yet.") django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

to over come that. I put

import django
django.setup()

before importing models in mqtt.py and that solved the problem. to the best of my knowledge, it starts the app which mqtt.py is inside of it before the main app. So the django see two instance of that app and throws this error:

raise ImproperlyConfigured( django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: graphs

So, I understand that my implementation is wrong. but how to correct it?

Ceballos

This is the 3rd time in about 24hrs you've asked exactly the same question (#66408004 & #66408004). Please delete the other 2 and in future update the original question rather than reposting. – Trichloride

Answer 1

I had a similar problem after following a similar setup as to what you have, here: https://mcmap.net/q/677111/-how-to-use-paho-mqtt-client-in-django

The resolution for the repeated connection was due to a setting on the MQTT server where the user name is set as the clientid.

mosquitto.conf

# Set use_username_as_clientid to true to replace the clientid that a client
# connected with with its username. This allows authentication to be tied to
# the clientid, which means that it is possible to prevent one client
# disconnecting another by using the same clientid.
# If a client connects with no username it will be disconnected as not
# authorised when this option is set to true.
# Do not use in conjunction with clientid_prefixes.
# See also use_identity_as_username.
use_username_as_clientid true

You may want to check that setting. and make sure that you are using different usernames for each mqtt client.

Appeal

Thank you friend. I've overcame this problem by calling the loop function in urls.py file. Using init.py was wrong at first place. And there is only one client and one username, so that look not the problem. BTW, thank you for your time – Ceballos

@Ceballos Can you share the code? – Drown

Answer 2

I stumbled over the same issue and I found out two things which might be interesting also for this problem here:

1. Django development server starts two threads for auto-reload. Many files (I am not sure if all) are loaded twice, like the __init__.py. If you specifiy a client_id like you did (client_id="testSubscriber"), the client connects to the broker using this name, if now the second thread starts it also connects the mqqt client to the broker with this exact id. The broker notices that this client is already connected and deletes the old connection. As the loop_start() function auto-reconnects, this is an infinit loop. You can fix this by using the --noreload flag when starting the development server like:

   python .\manage.py runserver --noreload

   this will start only one thread, or by specifying a unique client id e.g. like:

   client_id = f"testSubscriber_{datetime.datetime.now()}"

2. Regarding AppRegistryNotReady("Apps aren't loaded yet."):

   To avoid this place the code in apps.py and override the ready function. This will execute the code when the application is loaded and ready.

# example_app/apps.py
from django.apps import AppConfig


class ExampleAppConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'example_app'

    def ready(self):
        from example_app.mqtt import mqtt_client      
        mqtt_client.loop_start()

Ahab