Thread count increases a lot, even when deleting the threads

Question

Have an application where I have QOBJects which all contain an QNetworkAccessManager. I am aware of that it's suggested to only have on per application but since I'm making a lot more that 6 calls at the same time, I needed to have it like this. So, this is how I start the threads.

FileUploader *fileUploader = new FileUploader(_fileList);
QThread *fileUploaderThread = new QThread();
fileUploader->moveToThread(fileUploaderThread);

// uploader > model
connect(fileUploader, SIGNAL(progressChangedAt(int)), _model, SLOT(reportProgressChanged(int)), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(statusChangedAt(int)), _model, SLOT(reportStatusChanged(int)), Qt::QueuedConnection);
// uploader > its thread
connect(fileUploader, SIGNAL(canceled()), fileUploaderThread, SLOT(quit()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finished()), fileUploaderThread, SLOT(quit()), Qt::QueuedConnection);
// uploader > this
connect(fileUploader, SIGNAL(canceled()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finished()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finishedCurrentUpload()), this, SLOT(uploadNextFileOrFinish()), Qt::QueuedConnection);
// thread > this
connect(fileUploaderThread, SIGNAL(finished()), this, SLOT(checkIfAllThreadsAreFinished()), Qt::QueuedConnection);
connect(fileUploaderThread, SIGNAL(finished()), this, SLOT(deleteFinishedThread()), Qt::QueuedConnection);
// this > uploader
connect(this, SIGNAL(cancel()), fileUploader, SLOT(cancel()), Qt::QueuedConnection);

fileUploaderThread->start();
QMetaObject::invokeMethod(fileUploader, "init", Qt::QueuedConnection);
QMetaObject::invokeMethod(fileUploader, "uploadAt", Qt::QueuedConnection, Q_ARG(int, startIndex));

QMutexLocker locker(&_mutex);
_threadCount++;

Every thread starts out with an index to a list so that they can fetch the thing they need to upload and proceed about 5 steps (calls with the QNetworkAccessManager). When there's no more items to upload, the fileUploader signals "finished()" which calls the deleteFinishedThread and deleteFinishedUploader where I do:

QThread *thread = qobject_cast<QThread*>(sender());

if(thread != NULL) thread->deleteLater();

or

FileUploader *fileUploader = qobject_cast<FileUploader*>(sender());

if(fileUploader != NULL) fileUploader->deleteLater();

These are suppose to delete the threads when they are done.

The issue is that every time I start (for example) 3 threads that have 1 file to upload and handle each, the thread count gets increased by 8-10. This means that the thread count goes from about 5 to 100 if I restart the uploading process a few times.

What am I doing wrong? Or is my biggest issue that I use "Windows Task Manager" to control this? I am handeling all replies from the QNAM which I delete and everything seems to get deleted, but still I scratching my head when the threadcount keeps increasing...

EDIT: In my fileuploader I create an object(Manager) on the heap which has a QNetworkAccessManager on the stack. When the fileuploader gets deleted it calls "deleteLater()" on the Manager but it never gets deleted. We tried to delete the Manager and set it to NULL but that gave us an access violation since the Manager wasn't done yet (the QNetwork.dll reported the issue so it must be something inside the QNAM that is running still). The times when we didnt get access violation, the object was deleted and the thread count went back to normal. What can live inside the QNAM and prevent me from deleting it when it goes out of scope? Should I create the QNAM on the heap instead? At this stage non of the destructors are being called even when calling deleteLater()...

Also, how do I reduce the handle-count?

Caroche

added a bounty. your question need some love... – Ornithopod

one comment: if you still use them, call QMetaObject::invokeMethod( before starting fileUploaderThread. It is the best way to ensure that these slots are called first when the thread loop start. – Ornithopod

if i call the "init" before the thread is started, all my heap alloocated objects will be locateded in the main thread – Caroche

no QMetaObject::invokeMethod will put stack up "init" in the even loop and will be executed first. this is what happens in QApplication a(argc, argv); MainWindow w; w.show(); return a.exec();. show() is invoked as soon as the app a even loop starts. (see invoke method doc) – Ornithopod

still not too sure what you mean. my approach works like i want it to but if there's any issue i would be greatful if you sent me a link or a longer comment explaining why this isnt working. atm the costructor is called when the object is created. the object is moved to a thread and all the heap allocations etc gets created when the object is in the thread (which means when i create them using "this" as parent, "this" will be in the thread, not the main – Caroche

the doc states "The child of a QObject must always be created in the thread where the parent was created. This implies, among other things, that you should never pass the QThread object (this) as the parent of an object created in the thread (since the QThread object itself was created in another thread)." so I want to allocate an object I cannot do that in the constructor since "this" is still in the mainthread there – Caroche

would still love if you could explain " It is the best way to ensure that these slots are called first when the thread loop start. " to me since I might not get at all what you are talking about :) – Caroche

Answer 1

After a lot of "almost giving up" I came up with a solution for the threads. It is true what Synxis said about the order of the slots.

I still however have some issue with the filehandles, and if someone has a better answer that mine I'm happy to accept that.

I changed my code to:

...
// uploader > this
connect(fileUploader, SIGNAL(canceled()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finished()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);
// uploader > its thread
connect(fileUploader, SIGNAL(destroyed()), fileUploaderThread, SLOT(quit()));

This means that the thread gets stopped (quit()) when the object is getting deleted. This actually works even though the documentation states:

This signal is emitted immediately before the object obj is destroyed, and can not be blocked.

All the objects's children are destroyed immediately after this signal is emitted.

Which means that this signals gets emitted just BEFORE anything gets destroyed (which would mean that I would quit the thread before the uploader in it got deleted)? Not really good enough and it might be a better way. HOWEVER, atm, my thread count goes down quite a bit every time the uploader finishes and back to normal after a 20sec or so (a few "watcher-threads" must be killed by windows etc).

Caroche

Nice solution for the slots. How are file handles managed/created/deleted ? What is the problem with them ? – Appointee

im not too sure. have some experience with c# and how you do then but all i can see now is that the number of handles are increasing but not decreasing – Caroche

the problem is that i'm not too sure if deleating the thread before the objects inside of it gets destroyed.. – Caroche

If you mean deleted with 'deleteLater()', no, because it needs an event loop to work. You have either to delete objects before deleting the thread or move objects to another thread. – Appointee

not too sure what u mean. it currently works and everything in the uploader gets deleted, aswell as the thread that it is in. This is possibly since the baseclass QOBject (of the uploader) is the last thing to be called and when that emits "destroyed" (which will destroy the thread) theres really nothing else left that could be deleted. Yeah, thers still things left to do in ~QObject that might be really important so I will move the object to the main thread. – Caroche

Answer 2

I might be wrong, but I think there is a problem with your signals:

// uploader > its thread
connect(fileUploader, SIGNAL(canceled()), fileUploaderThread, SLOT(quit()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finished()), fileUploaderThread, SLOT(quit()), Qt::QueuedConnection);
// uploader > this
connect(fileUploader, SIGNAL(canceled()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);
connect(fileUploader, SIGNAL(finished()), this, SLOT(deleteFinishedUploader()), Qt::QueuedConnection);

Remember that when multiple slots are connected to the same signal, they're executed in the order of connection. Here, when the fileUploader will be finished, it will call finished() which will first call the quit() method of the thread, and then the deleteFinishedUploader() method. Same for the canceled() signal. But, meanwhile, the thread was finished, so no event processing for the fileUploader can be done (consequence of the moveToThread(...)). deleteLater() needs event processing, thus your fileUploader will never be deleted...

I am not 100% that arranging your connections in the other way will get things working: the deleteLater() can be called and the thread exited immediatelly after, without event processing.

The solution might be to re-moveToThread() the fileUploader to the main thread, or to a thread which still processes its event loop.

Appointee

Answer 3

Not an answer but :

    fileUploaderThread->start();
    QMetaObject::invokeMethod(fileUploader, "init", Qt::QueuedConnection);
    QMetaObject::invokeMethod(fileUploader, "uploadAt", Qt::QueuedConnection, Q_ARG(int, startIndex));

means you start the even loop, and then you queue up slots or signals to be executed.. Assume (in general) that there are other QObject in this thread. It might be possible that these get their slots or signals executed because the event loop has already started. If you want the "init" and "uploadAt" to be the first methods to be invoked when the event loop run, you queue them before starting the event loop (If the thread is not started they are never going to be executed).

From QMetaObject::invokeMethod:

If type is Qt::QueuedConnection, a QEvent will be sent and the member is invoked as soon as the application enters the main event loop.

In this case the event is sent to the thread event loop.

Ornithopod

oh okay. really good to know thank you! however, in this example i might just stick to what ive got since the uploader will be the only item in that thread :) – Caroche