Daemon Threads Explanation

Question

In the Python documentation it says:

A thread can be flagged as a "daemon thread". The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread.

Does anyone have a clearer explanation of what that means or a practical example showing where you would set threads as daemonic?

Clarify it for me: so the only situation you wouldn't set threads as daemonic, is when you want them to continue running after the main thread exits?

Volding

Answer 1

Some threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it's okay to kill them off once the other, non-daemon, threads have exited.

Without daemon threads, you'd have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.

Clack

So if I have a child thread that is performing a file write operation which is set to non-deamon, Does that mean I have to make it exit explicitly ? – Inflorescence

@san What does your writer thread do after it's finished writing? Does it just return? If so, that's sufficient. Daemon threads are usually for things that run in a loop and don't exit on their own. – Clack

It do nothing, neither returns, its sole purpose to perform file write operation – Inflorescence

@san If it falls off the bottom of the thread function, it returns implicitly. – Clack

It returns None in that case, but it doesn't matter, the return value isn't used. – Clack

Let us continue this discussion in chat. – Inflorescence

@Ciastopiekarz If you want to communicate with a child thread, you need to do it via some shared structure. Depending on how you do it, you need to consider concurrency issues. But in the simplest case, the child could write an attribute in an object that is reachable from both the main thread and the child, and the main thread could make sure not to read it until it has join()ed the child. – Endamage

1. If I quite the program but not kill the no-daemon threads, then how can I manage the no-demon threads? Does this situation exist? 2. If I kill the no-daemon threads before quite the program, then what are differences between daemon and no-daemon? – Carcajou

upvoted! so going by this definition should a non blocking threaded version of a redis subscriber that listens to messages infinitely be daemon or non daemon, I am assuming it should not be daemon since it is like the primary thing you want to do inside a thread? – Ralina

The documentation (docs.python.org/3/library/threading.html#thread-objects) says that "Daemon threads are abruptly stopped at shutdown. " - Doesn't this mean that - when the main program finishes the daemon threads are shutdown? – Odoriferous

Answer 2

Let's say you're making some kind of dashboard widget. As part of this, you want it to display the unread message count in your email box. So you make a little thread that will:

1. Connect to the mail server and ask how many unread messages you have.
2. Signal the GUI with the updated count.
3. Sleep for a little while.

When your widget starts up, it would create this thread, designate it a daemon, and start it. Because it's a daemon, you don't have to think about it; when your widget exits, the thread will stop automatically.

Arteriovenous

Answer 3

I will also add my few bits here, I think one of the reasons why daemon threads are confusing to most people(atleast they were to me) is because of the Unix context to the word dameon.

In Unix terminology the word daemon refers to a process which once spawned; keeps running in the background and user can move on to do other stuff with the foreground process.

In Python threading context, every thread upon creation runs in the background, whether it is daemon or non-daemon, the difference comes from the fact how these threads affect the main thread.

When you start a non-daemon thread, it starts running in background and you can perform other stuff, however, your main thread will not exit until all such non-daemon threads have completed their execution, so in a way, your program or main thread is blocked.

With daemon threads they still run in the background but with one key difference that they do not block the main thread. As soon as the main thread completes its execution & the program exits, all the remaining daemon threads will be reaped. This makes them useful for operations which you want to perform in background but want these operations to exit automatically as soon as the main application exits.

One point to keep note of is that you should be aware of what exactly you are doing in daemon threads, the fact they exit when main thread exits can give you unexpected surprises. One of the ways to gracefully clean up the daemon threads is to use the Threading Events to set the event as an exit handler and check if the event is set inside the thread and then break from the thread function accordingly.

Another thing that confused about daemon threads is the definition from python documentation.

The significance of this flag is that the entire Python program exits when only daemon threads are left

In simple words what this means is that if your program has both daemon and non-daemon threads the main program will be blocked and wait until all the non-daemon have exited, as soon as they exit main thread will exit as well. What this statement also implies but is not clear at first glance is that all daemon threads will be exited automatically once the main threads exits.

Overwhelming

I like how you went to explain what 'daemon' means. It was exactly what I was looking for. – Nakamura

Only person that can tell us what it its :) – Synonymous

Answer 4

A simpler way to think about it, perhaps: when main returns, your process will not exit if there are non-daemon threads still running.

A bit of advice: Clean shutdown is easy to get wrong when threads and synchronization are involved - if you can avoid it, do so. Use daemon threads whenever possible.

Brunei

Another comment has this: Other posters gave some examples for situations in which you'd use daemon threads. My recommendation, however, is never to use them. Is his suggestion or your suggestion more correct for Python 3? – Monahan

Answer 5

Other posters gave some examples for situations in which you'd use daemon threads. My recommendation, however, is never to use them.

It's not because they're not useful, but because there are some bad side effects you can experience if you use them. Daemon threads can still execute after the Python runtime starts tearing down things in the main thread, causing some pretty bizarre exceptions.

More info here:

https://joeshaw.org/python-daemon-threads-considered-harmful/

https://mail.python.org/pipermail/python-list/2005-February/343697.html

Strictly speaking you never need them, it just makes implementation easier in some cases.

Creekmore

Still this issue with python 3 ? There is no clear information regarding these "bizarre exceptions" in the documentation. – Omniscience

From Joe's blog post: "Update June 2015: This is Python bug 1856. It was fixed in Python 3.2.1 and 3.3, but the fix was never backported to 2.x. (An attempt to backport to the 2.7 branch caused another bug and it was abandoned.) Daemon threads may be ok in Python >= 3.2.1, but definitely aren’t in earlier versions." – Endamage

I'd like to share here my experience: I had a function spawned as Thread multiple times. Inside of it, I had an instance of Python logging and I expected that, after finishing the Thread, all objects (File Descriptors for each Thread/Function), would be destroyed. At the end of my program, I saw many outputs like IOError: [Errno 24] Too many open files:. With lsof -p pid_of_program, I discovered that the FDs were open, even tough the Thread/Functions have finished their jobs. Workaround? Removing the log handler at the end of the function. So daemonic Threads, are untrustworthy... – Saguache

That's odd. If I don't use daemon=True and if I interrupt the process which spawned the Threads, I see that the Threads are still running. The same doesn't happens when this flag is set, the main program and the Threads are all terminated. How would you explain this? If I need to interrupt a program, I want all Threads to terminate as well. Do you know a better approach than this? Thanks. Here's the doc, for further reference: docs.python.org/3/library/threading.html#thread-objects – Saguache

@JoeShaw another comment has this: A bit of advice: Clean shutdown is easy to get wrong when threads and synchronization are involved - if you can avoid it, do so. Use daemon threads whenever possible. https://mcmap.net/q/99825/-daemon-threads-explanation lol whose suggestion is more correct? – Monahan

Answer 6

Quoting Chris: "... when your program quits, any daemon threads are killed automatically.". I think that sums it up. You should be careful when you use them as they abruptly terminate when main program executes to completion.

Deliquesce

Answer 7

Chris already explained what daemon threads are, so let's talk about practical usage. Many thread pool implementations use daemon threads for task workers. Workers are threads which execute tasks from task queue.

Worker needs to keep waiting for tasks in task queue indefinitely as they don't know when new task will appear. Thread which assigns tasks (say main thread) only knows when tasks are over. Main thread waits on task queue to get empty and then exits. If workers are user threads i.e. non-daemon, program won't terminate. It will keep waiting for these indefinitely running workers, even though workers aren't doing anything useful. Mark workers daemon threads, and main thread will take care of killing them as soon as it's done handling tasks.

Dissent

Be careful with that! If a program submits an important task (e.g., update some file "in the background") to a daemon task queue, then there's a risk that the program could terminate before performing the task, or worse, in the middle of updating that file. – Bernard

Answer 8

When your second thread is non-Daemon, your application's primary main thread cannot quit because its exit criteria is being tied to the exit also of non-Daemon thread(s). Threads cannot be forcibly killed in python, therefore your app will have to really wait for the non-Daemon thread(s) to exit. If this behavior is not what you want, then set your second thread as daemon so that it won't hold back your application from exiting.

Betweenwhiles

Answer 9

Create a Daemon thread when:

• You want a low-priority thread
• Your Thread does background-specific tasks, and more importantly,
• When you want this thread to die as soon as all user threads accomplish their tasks.

Some Examples of Daemon Thread Services: Garbage collection in Java, Word count checker in MS Word, Auto-saver in medium, File downloads counter in a parallel file downloads application, etc.

Lashoh

How is that "low priority" handled? What if I want it to be as fast as any other? – Lipps