How to turn only member variables into byte array in C++?

Question

TL;DR

I have a struct with more than just member variables (e.g. they contain functions), and I want to convert only the member variables into a byte array(/vector), such that I can upload data to the graphics card in Vulkan. How do I get only that part of the struct, which represents the member variables?

My concrete approach

I have a setup where I use an empty ParamsBase struct and from that inherit ParamsA, ParamsB ... structs which actually contain member variables. I use this such that I can keep members of ParamsBase in a Container class without actual knowledge of the concrete implementation.

I want to turn the Params instance in the Container class into a byte buffer.

Since I need the size of the actual ParamsA/ParamsB/... struct, I have used a generic subclass for these when creating instances that allows me to use a single getSize() function, rather than overwriting this in every substruct.

// =============================
// === Define Params structs ===
// =============================
struct ParamsBase
{
    virtual size_t getSize() const noexcept = 0;
};

struct ParamsA : public ParamsBase
{
    float vec[3] = { 2.3f, 3.4f, 4.5f };
};

// ===============================================================
// === enable sizeof query for derived structs from ParamsBase ===
// ===============================================================
template<class T>
struct GetSizeImpl : T
{
    using T::T;
    size_t getSize() const noexcept final { return sizeof(T); }
};

template<class T, class... Args>
std::unique_ptr<T> create(Args&&... args)
{
    return std::unique_ptr<T>(new GetSizeImpl<T>(std::forward<Args>(args)...));
}

// ============
// === Util ===
// ============
template<typename T>
std::vector<uint8_t> asByteVector(T* t, size_t size)
{
    std::vector<uint8_t> byteVec;
    byteVec.reserve(size);
    uint8_t* dataArr = std::bit_cast<uint8_t*>(t);
    byteVec.insert(byteVec.end(), &dataArr[0], &dataArr[size]);
    return byteVec;
}

// ============================================
// === Use Params struct in container class ===
// ============================================
class Container
{
public:
    Container(ParamsBase* params) : Params(params) {}

    const std::vector<uint8_t>& getParamsAsBuffer()
    {
        ParamsAsBuffer = asByteVector(Params, Params->getSize());
        return ParamsAsBuffer;
    }

    size_t getParamsSize() const { return Params->getSize(); }

private:
    ParamsBase* Params = nullptr;

    std::vector<uint8_t> ParamsAsBuffer;
};

Using all this, the sizes of my Params structs is too large and starts of with two bytes containing some garbage(?) data. I assume it has to do with the getSize() function, since even a non-templated struct with a function has this problem, but I cannot be sure if this assumption is correct.

This little comparison shows differences between what I get and what I want:

// ================================
// === Define comparison struct ===
// ================================
struct ParamsCompareA
{
    float vec[3] = { 2.3f, 3.4f, 4.5f };
};

int main()
{
    // create instances
    auto pA = create<ParamsA>();
    Container cA(pA.get());
    std::vector<uint8_t> vecA = cA.getParamsAsBuffer();

    // comparison
    ParamsCompareA pcA;

    size_t sizeCompA = sizeof(ParamsCompareA);
    std::vector<uint8_t> compVecA = asByteVector(&pcA, sizeof(ParamsCompareA));

    std::cout << "ParamsA size: " << std::to_string(pA->getSize()) 
        << "; CompParamsA size: " << sizeof(ParamsCompareA) << std::endl;

    float* bufAf = reinterpret_cast<float*>(vecA.data());
    float* bufCompAf = reinterpret_cast<float*>(compVecA.data());
}

The compVecA contains 12 entries (i.e. the struct is 12 bytes large), reinterpreting them as floats shows the correct values. The vecA is reported to have 24 entries and the actual data I want is at (0 based) bytes 2 to 14.

I could hardcode an offset of 2 (which is the same as sizeof(GetSizeImpl)), but I'm pretty sure this is not the correct way to handle this. So, is there a way I can get to only the data part I want?

The purpose of the Param sub structs is to make it as easy as possible for a user to add their own Parameter struct and upload it to a Vulkan Buffer (/Shader), i.e. they should worry only about the data they actually need and I can do all the handling and conversion elsewhere.

Placement

Polymorphism and virtual functions are usually implemented using a hidden private member variable, whose size is included in the sizeof of the object. If you want to use polymorphism and virtual functions, you can't just do a raw byte-wise copy an object into a byte array. You need to serialize the data. – Bolero

I only read TL;DR part. If you need to std::memcpy your struct, then just make it POD and define methods manipulating is somewhere else: in another struct by composition, or as free functions. – Cipolin

Just as a side note: Structs only contain the data, function code is always separate. However if there are virtual functions involved there's an additional, hidden v-table pointer (or several ones on more complex inheritance structures). – Oscoumbrian

@Cipolin Thanks, but I need to get the size and a byte array copy of the struct without knowing its exact type (or I'll have to think of a different system for my problem entirely). – Placement

Oki I see what you might need, but your approach with truct GetSizeImpl : T is both overcomplicated and wrong - because it contains vtable.. Just impl virtual std::vector<uint8_t> Params::serialize(); functions and you are good to go. It can be trivial memcpy of members, or something more sophisticated – Cipolin

You are missing calls to std::start_lifetime_as_array to allow bufAf to be read from. – Rocky

@Rocky I only added those arrays for debugging where the actual data would start, they wouldn't be used in the real program. Thanks for the hint though! – Placement

Answer 1

Don't derive ParamsA from anything. Have a wrapper type that provides the behaviours shared between each param type.

struct ParamsA // No inheritance!
{
    float vec[3] = { 2.3f, 3.4f, 4.5f };
};

struct BaseParameters {
    virtual std::span<std::byte> as_bytes() = 0;
};

template <typename T>
struct Parameters : BaseParameters {
    T data;
    /// ... other data members as you need
    std::span<std::byte> as_bytes() {
        return std::span(&data, 1).as_bytes();
    }
};

struct ParamsCompareA
{
    float vec[3] = { 2.3f, 3.4f, 4.5f };
};

int main()
{
    // create instances
    auto * pA = new Parameters<ParamsA>{};
    auto vecA = pA->as_bytes();

    // comparison
    ParamsCompareA pcA;

    std::cout << "ParamsA size: " << std::to_string(vecA.size()) 
        << "; CompParamsA size: " << sizeof(ParamsCompareA) << std::endl;

    float* bufAf = reinterpret_cast<float *>(vecA.data());
    float* bufCompA = pcA.vec;
}

Rocky

Just a few annotations, since there are some syntax errors. It needs to be struct BaseParameters { virtual std::span<const std::byte> as_bytes() = 0; }; and template <typename T> struct Parameters : BaseParameters { T data; /// ... other data members as you need std::span<const std::byte> as_bytes() { return std::as_bytes(std::span(&data, 1)); } }; Works like a charm though, thank you. – Placement

Answer 2

By adding a couple of conventions and layouts this is possible to do with your current setup.

The first modification is to declare the member variables inside a struct within the Param structs.

struct ParamsA : public ParamsBase
{
    struct Members {
        float vec[3] = { 2.3f, 3.4f, 4.5f };
    } m;
};

Then add another virtual function to ParamBase

virtual const void* data() const = 0;

Now we can change GetSizeImpl::getSize to return sizeof(T::Members) instead, and also override the other function

const void* data() const override { return &this->T::m; }

Now you have both the correct byte size of the data and a pointer to the POD data, which you legally can memcpy as much as you want to.

Trixi

This worked nicely. I opted for the second answer, since that is a little less overhead for anyone creating their own parameters, but this is till a nice way to do it, thank you. – Placement