QNetworkAccessManager: post http multipart from serial QIODevice

Question

I'm trying to use QNetworkAccessManager to upload http multiparts to a dedicated server.

The multipart consists of a JSON part describing the data being uploaded.

The data is read from a serial QIODevice, which encrypts the data.

This is the code that creates the multipart request:

QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);

QHttpPart metaPart;
metaPart.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
metaPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"metadata\""));
metaPart.setBody(meta.toJson());
multiPart->append(metaPart);

QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant(fileFormat));
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"file\""));
filePart.setBodyDevice(p_encDevice);
p_encDevice->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
multiPart->append(filePart);

QNetworkAccessManager netMgr;
QScopedPointer<QNetworkReply> reply( netMgr.post(request, multiPart) );
multiPart->setParent(reply.data()); // delete the multiPart with the reply

If the p_encDevice is an instance of QFile, that file gets uploaded just fine.

If the specialised encrypting QIODevice is used (serial device) then all of the data is read from my custom device. however QNetworkAccessManager::post() doesn't complete (hangs).

I read in the documentation of QHttpPart that:

if device is sequential (e.g. sockets, but not files), QNetworkAccessManager::post() should be called after device has emitted finished().

Unfortunately I don't know how do that.

Please advise.

EDIT:

QIODevice doesn't have finished() slot at all. What's more, reading from my custom IODevice doesn't happen at all if QNetworkAccessManager::post() is not called and therefore the device wouldn't be able to emit such an event. (Catch 22?)

EDIT 2:

It seems that QNAM does not work with sequential devices at all. See discussion on qt-project.

EDIT 3:

I managed to "fool" QNAM to make it think that it is reading from non-sequential devices, but seek and reset functions prevent seeking. This will work until QNAM will actually try to seek.

bool AesDevice::isSequential() const
{
    return false;
}

bool AesDevice::reset()
{
    if (this->pos() != 0) {
        return false;
    }
    return QIODevice::reset();
}

bool AesDevice::seek(qint64 pos)
{
    if (this->pos() != pos) {
        return false;
    }
    return QIODevice::seek(pos);
}

Exemplificative

I think the appropriate signal is QIODevice::readChannelFinished(). Basically QIODevice::bytesAvailable() must return the correct value for it to work. – Nail

Have you resolved the issue since then, matejk? – Jeffiejeffrey

I managed to resolve it, but not in a clean way. See my comment below. – Exemplificative

Answer 1

From a separate discussion in qt-project and by inspecting the source code it seems that QNAM doesn't work with sequential at all. Both the documentation and code are wrong.

Exemplificative

so how to solve this use case? We need to compromise with large data in memory, or you start mmap'ing it instead? – Jeffiejeffrey

@LaszloPapp I created the encryption device in such a way that it declares to be non-sequential, but read() and seek() functions verify current position to make sure that the data is read in sequential manner. QNAM is reading the data sequentially. It is not nice, but it works. – Exemplificative

Answer 2

You'll need to refactor your code quite a lot so that the variables you pass to post are available outside that function you've posted, then you'll need a new slot defined with the code for doing the post inside the implementation. Lastly you need to do connect(p_encDevice, SIGNAL(finished()), this, SLOT(yourSlot()) to glue it all together.

You're mostly there, you just need to refactor it out and add a new slot you can tie to the QIODevice::finished() signal.

Heterosexual

Nicolas, thanks. Does this effectively mean that all of the data from incoming p_encDevice will be read into QNetworkAccessManager's internal buffer before the post is called? If it is so then it is much easier if I read the data into QByteArray and pass it to QHttpPart::setBody. – Exemplificative

It won't read into the QNAM buffer, essentially you're going to continue reading into the filePart as you do now but you'll need filePart to be a class member so the slot you create can access it. – Heterosexual

QIODevice doesn't have finished slot at all. What's more, reading from my custom IODevice doesn't happen at all if QNetworkAccessManager::post is not called and therefore the device wouldn't be able to emit such an event. – Exemplificative

@matejk: Indeed, there is no QIODevice::finished(). I wonder what Nicholas meant to write instead... – Jeffiejeffrey

Answer 3

I've had more success creating the http post data manually than with using QHttpPart and QHttpMultiPart. I know it's probably not what you want to hear, and it's a little messy, but it definitely works. In this example I am reading from a QFile, but you can call readAll() on any QIODevice. It also is worth noting, QIODevice::size() will help you check if all the data has been read.

QByteArray postData;
QFile *file=new QFile("/tmp/image.jpg");
if(!(file->open(QIODevice::ReadOnly))){
    qDebug() << "Could not open file for reading: "<< file->fileName();
    return;
}
//create a header that the server can recognize
postData.insert(0,"--AaB03x\r\nContent-Disposition: form-data; name=\"attachment\"; filename=\"image.jpg\"\r\nContent-Type: image/jpeg\r\n\r\n");
postData.append(file->readAll());
postData.append("\r\n--AaB03x--\r\n");
//here you can add additional parameters that your server may need to parse the data at the end of the url
QString check(QString(POST_URL)+"?fn="+fn+"&md="+md);
QNetworkRequest req(QUrl(check.toLocal8Bit()));
req.setHeader(QNetworkRequest::ContentTypeHeader,"multipart/form-data; boundary=AaB03x");
QVariant l=postData.length();
req.setHeader(QNetworkRequest::ContentLengthHeader,l.toString());
file->close();
//free up memory
delete(file);
//post the data
reply=manager->post(req,postData);
//connect the reply object so we can track the progress of the upload        
connect(reply,SIGNAL(uploadProgress(qint64,qint64)),this,SLOT(updateProgress(qint64,qint64)));

Then the server can access the data like this:

<?php
$filename=$_REQUEST['fn'];
$makedir=$_REQUEST['md'];
if($_FILES["attachment"]["type"]=="image/jpeg"){
if(!move_uploaded_file($_FILES["attachment"]["tmp_name"], "/directory/" . $filename)){
    echo "File Error";
    error_log("Uploaded File Error");
    exit();
};
}else{
print("no file");
error_log("No File");
exit();
}
echo "Success.";
?>

I hope some of this code can help you.

Chekiang

I currently read complete data and post it, however I can't afford this because the files can be very large (a few GB). – Exemplificative

@matejk: exactly, readAll would not actually need anything more than readAll and post from QNAM, but people cannot do this when there are big files. – Jeffiejeffrey

Answer 4

I think the catch is that QNetworkAccessManager does not support chunked transfer encoding when uploading (POST, PUT) data. This means that QNAM must know in advance the length of the data it's going to upload, in order to send the Content-Length header. This implies:

1. either the data does not come from sequential devices, but from random-access devices, which would correctly report their total size through size();
2. or the data comes from a sequential device, but the device has already buffered all of it (this is the meaning of the note about finished()), and will report it (through bytesAvailable(), I suppose);
3. or the data comes from a sequential device which has not buffered all the data, which in turn means
   1. either QNAM reads and buffers itself all the data coming from the device (by reading until EOF)
   2. or the user manually set the Content-Length header for the request.

(About the last two points, see the docs for the QNetworkRequest::DoNotBufferUploadDataAttribute.)

So, QHttpMultiPart somehow shares these limitations, and it's likely that it's choking on case 3. Supposing that you cannot possibly buffer in memory all the data from your "encoder" QIODevice, is there any chance you might know the size of the encoded data in advance and set the content-length on the QHttpPart?

(As a last note, you shouldn't be using QScopedPointer. That will delete the QNR when the smart pointer falls out of scope, but you don't want to do that. You want to delete the QNR when it emits finished()).

Alain

Thank you for additional options to try. I set DoNotBufferUploadDataAttribute and set content length explicitly, because I know the size in advance, but it really did not help. Input stream is read to the EOF, but then everything stops. Documentation is mentioning signal finished(), which QIODevice doesn't emit at all. – Exemplificative

I use ScopedPointer intentionally, because the code that follows is waiting for the QNR to finish. – Exemplificative

@matejk: yes, I am using QScopedPointer as well in a main function where QNetworkReply cannot be deleted by it since the Qt event loop defends that. Yeah, there is no finished() signal, unfortunately. – Jeffiejeffrey