Proper way to convert (many!) numbers to strings without allocations in Qt

Question

tl;dr

I want to call QString::number(int) many times per second. It is very slow: seems like it allocates a new string each time. Tried to use setNum on same string instead, still no joy.

---

^{Original, long question:}

The problem

I have a big array of numbers (say, integers) and I want to format them into text, that will be then (maybe not immediately) written to file. Naive way looks approximately¹ like this:

QString allData;
foreach(const int & value, values) {
    allData += QString::number(value);
    allData += '\n';
}

It takes about 280ms for 150000 integers on my machine which seems to much for me. I suppose that this is because QString::number is called 150000 times and each time allocates new string. This is someway confirmed to be the root of the problem when I try to use itoa (which does not allocate memory) instead.

Possible, but not-Qt [not-cute] solution

QString allData;
char buffer[100];                               // <-------
foreach(const int & value, values) {
    _itoa_s(value, buffer, sizeof(buffer), 10); // <-------
    allData += buffer;
    allData += '\n';
}

This takes about 70ms for the same 150000 integers (about 4x faster) which is by now acceptable for me (I think I can do something with string concatenation as well, but let's leave this outside this question)

But I don't like that I have to use some unstandard, probably deprecated, probably unportable² function (not to say that this just looks ugly).

Then I remembered that there is also an instance method: QString::setNum. I hoped I could use the same pattern as with itoa: have only one string allocated and modify it each time.

Desirable, but not working solution

QString allData;
QString number;                       // <-------
foreach(const int & value, values) {
    number.setNum(value);             // <-------
    allData += number;
    allData += '\n';
}

Unfortunately, this doesn't make big difference from QString::number: again about 280ms, well, maybe 250ms but still too much.

So, congrats if you reached here :) and finally...

The Question(s)

1. What would Qt experts advise me to do? Shut up and use itoa despite the distinct smell of C in otherwise fragrant C++/Qt code?
2. Or can I somehow say "C'mon, Qstring, just eat this number into you" ?
3. I wonder why setNum did not do the trick?

---

Footnotes:

¹ In actual code i have not just 150000 integers, but 50000 triples of integers which I also add '\t' between them. This is the only difference from my actual code and I guess it is not important: here I'm interested only in performance of QString::number vs itoa.

² Actually, I was surprised that MinGW also has _itoa_s that behaves just like as Visual Studio's, but I still have some awkward feeling that using such a dirty function in my polished Qt code reduces its portability. Correct me if I'm wrong.

Sos

I vote for 2). If you are using Qt and QString, I would not complain about performance since it is inherently a slower framework than a performance-optimized one, at least most of the time. I would never use Qt in performance critical systems. As for setNum, have you tried reserve? – Thevenot

@LaszloPapp, as I hoped, herer it actually appeared that not Qt is slow, but I used wrong class for this task: should be using QByteArray instead of QString (thanks to @SalvatoreAvanzo). As to your advice: I tried playing with reserve, but it didn't have any impact. Seems like the implementation of QString's setNum is just slower than QByteArray's one. Probably because QByteArray is just kinda wrapper over char*, while QString is much more complicated (encodings, etc). – Sos

well, you were asking about QString, so people like me assumed you insert other non-numbers, in which case QByteArray is kind of moot. The thing still stands, I really would not personally use Qt for performance critical things, especially if you are after miliseconds. It just feels bad idea to me. – Thevenot

Have you done a QString::reserve() prior to adding all those numbers? You know you'll need at least two characters per number, at most 10 (?), and you probably have a reasonable estimate how many digits you have on average. – Prosthesis

@Prosthesis yes, of course :) But I insist: problem was not with not-enough-bytes initially allocated. I added some notes to @SalvatoreAvanzo's answer to explain what was the problem with QString::setNum. – Sos

If you need speed, then creating a custom helper function, which first uses QString::resize() (if necessary), then does the integer to text conversion, writing unicode characters directly to QString with myqstr[index] = QChar(0x0030 + digitvalue); would probably be fastest (U+0030 is Unicode code point for '0'). This is basically micro-optimization, but it allows taking shortcuts not possible in general purpose library code, and enables better optimizations (esp. inlining). – Canaveral

Answer 1

You can try with QByteArray that shares same QString's interface but is more suitable for performance issues.I obtain 36 ms (qt 5.2 clang) vs. your original 57 ms (on my machine) with this code:

QByteArray allDatab;
foreach(const int & value, values) {
    allDatab += QByteArray::number(value);
    allDatab += '\n';
}
QString result(allDatab);

and 29 ms with this version (that maybe confirm your assumptions about setNum):

QByteArray allDatad;
QByteArray number;                       
foreach(const int & value, values) {
    number.setNum(value);             
    allDatad += number;
    allDatad += '\n';
}

Loire

Whoa, didn't even except that QByteArray bursts so much! O_o On my machine I got 57 ms with the second variant, so it is even faster that QString+itoa. Great, solves my problem. – Sos

Keep that in mind, this would not work so well if you had to insert other things before and/or after that is QString'ish. Hopefully, that is not your use case. – Thevenot

@LaszloPapp , important comment, thanks. In my case it was enough to convert QByteArray to QString as soon as I finished adding numbers, and only then prepend header, etc: all subsequent manipulation of this string are little, so they didn't have impact on performance. – Sos

Well, a complete answer would explain why it was slow, which is that QString is more than just a byte array. It deals with utf, locale, and all that. – Thevenot

@LaszloPapp: Qt is a well documented framework. There you can find an explanation of the differences between QString and QByteArray better than I do. However I will be happy if you would edit my post to add some insights to this solution. – Loire

@SalvatoreAvanzo , well, I tried to suggest edit with more info to your answer but it did not work for some reason. You can read it here: (link). If you wish, you can edit your answer yourself (using source of this gist). – Sos

Answer 2

How about using STL?

I tested your code (modifying the loop to simplify)

int main() {
  stringstream ss;
  for(int i=0; i<2000000; ++i) {
     ss << i << "\n";
  }
}

And I obtain

time ./ss_test
  real  0m0.146s
  user  0m0.139s
  sys   0m0.006s

With Qt version (in my machine)

int main() {
  QString allData;
  for(int i=0; i<2000000; ++i) {
    allData += QString::number(i);
    allData += '\n';
  }
}

I obtain

time ./qtstring_test 
   real 0m0.516s
   user 0m0.508s
   sys  0m0.008s

Comfort

Thank you for you idea! While @SalvatoreAvanzo's solution with QByteArray has better overall performance in my situation (probably because it doesn't require inter-framework conversions), but still good idea that I may use somewhere else. – Sos