Copy a std::vector to a repeated field from protobuf with memcpy

Question

At first I have this simple protobuf file

message messagetest
{
    ...
    repeated float samples = 6;
    ....
}

Which creates a headerfile with this methods

    //repeated float samples = 6;
      inline int samples_size() const;
      inline void clear_samples();
      static const int kSamplesFieldNumber = 6;
      inline float samples(int index) const;
      inline void set_samples(int index, float value);
      inline void add_samples(float value);
      inline const ::google::protobuf::RepeatedField< float >&  samples() const;
      inline ::google::protobuf::RepeatedField< float >* mutable_samples();

What I'm basically doing is to copy all data one by one in a for loop.

int main(int argc, char** argv)
{    
    messagetest fMessage;
    
    vector<float> fData (1000, 0);

    // Create 1000 random values
    for (int i = 0; i < fData.size(); i++)
    {
        fData[i] = rand() % 1001;
    }
    
    for (int j = 0; j < fData.size(); j++)
    {
        fMessage.add_samples(fData[j]);    
    }

    return 0;
}

But I want to use a method like memcpy to accelerate the copy process. It is just an idea that comes to my mind. If it's completely wrong correct me. The last declaration in the headerfile is:

inline ::google::protobuf::RepeatedField< float >* mutable_samples();

I have no idea what this method does (lack of skill). But it kind of looks like a vector. Maybe that's the solution for my problem. If so, I have no idea how to implement it.

Avon

Have you profiled it, and seen that it's slow? Have you inspected the compiled code, and seen that it's poorly optimized? – Metronome

Answer 1

Since this isn't here yet and I like one-liners:

*fMessage.mutable_samples() = {fData.begin(), fData.end()};

Syllabify

Note that it works with c++11 and later, because reassignment using initializer list was not available before c++11. – Shiverick

Does this assignment copy, or move the used memory? – Precursory

@DavidTóth it copies the range into a temporary and then the temporary is move assigned to the pbf repeated field – Interlocutory

I would greatly appreciate if someone linked to actual documentation as I don't even have keywords to search. I am talking about this two-iterator braces initializer. – Cretic

@Cretic en.cppreference.com/w/cpp/language/list_initialization – Syllabify

But how do you know RepeatedField is std::vector behind the scene? list_initialization is something for standard library stuffs. – Spathic

Note that the reverse is also possible with the same one liner: std::vector<float> samples = {fMessage.samples().begin(), fMessage.samples().end()}; – Tiffany

@Spathic If a list_initialization constructor is not found, it will try to find a constructor that fits the data values in the list. In this case, there is a template constructor RepeatedField(Iter begin, Iter end) which is used. So no need to use std::vector behind the scenes for it to work. – Stupefaction

Answer 2

I found the shortest way to copy vector into repeated field as this:

google::protobuf::RepeatedField<float> data(fData.begin(), fData.end());
fMessage.mutable_samples()->Swap(&data);

It is probably also faster than yours since it avoids initial iteration and setting values to 0.

Decrescent

Answer 3

@mgild's answer will implicitly call the RepeatedField(Iter begin, Iter end) constructor to create a temporary to be move assigned. Same with @nazgul's answer, which explicitly creates a temporary RepeatedField and swaps it.

Much simpler, avoiding the creation of a new object would be:

fMessage.mutable_samples()->Add(fData.begin(), fData.end())

If samples field is not empty already, you can call Clear method prior.

Internally, this uses std::copy (so long as fData is a forward iterator), so will be just as fast as any memcpy implementation you come up with.

Stupefaction

Answer 4

As an alternative to the excellent answer posted by @mgild, Assign can also be used in this situation, so the data would not be copied to a temporary before being moved.

fMessage.mutable_samples()->Assign(fData.begin(), fData.end())

Credit to Marek R's answer

Genovese

Answer 5

fMessage.mutable_samples()

return an array of pointer of samples : [*sample1, *sample2, sample3, ...].

&fData[0]

is the address of first element of fData.

memcpy(fMessage.mutable_samples()->mutable_data(),
     &fData[0],
     sizeof(float)*fData.size());

So I do not think the code above can successfully fill data from fData to fMessage. It's totally wrong!

Cns

No it doesn't. mutable_samples() points to a RepeatedField<float>, and RepeatedField<float>::mutable_data returns float*. Have some documentation. – Spigot