libpqxx v12 - dynamically bind vector-values to prepared statement (alternative to invocation)

Question

I would like to use prepared statements to insert thousands of rows at once into my postgres database. The data to insert is stored in a vector of structs.

By reading the answers to How to prepare statements and bind parameters in Postgresql for C++ I thought I found the way how to do it.

Unfortunately, the current version of libpqxx I am using doesn't support pqxx::prepare::invocation anymore and I wasn't able to find any alternative in the docs/internet.

The code I tried out for my purpose is the following:

//pqxx header
#include "pqxx/connection.hxx" 
#include "pqxx/transaction.hxx"
#include "pqxx/nontransaction.hxx"
#include "pqxx/compiler-public.hxx"
#include "pqxx/result.hxx"
#include "pqxx/prepared_statement.hxx"
#include "pqxx/transaction_base.hxx"
#include "pqxx/internal/statement_parameters.hxx"

...
int main(){

    struct testData {
       string uuid;
       float rndNo;
       string timestamp;
    };

    vector<testData> dataBuffer;
    testData dbdata;

    string queryStr;
    const char* query;

    stringstream connStream;
    stringstream queryStream;

    string DB_NAME = "dbname";
    string DB_USER = "postgres";
    string DB_PASSWORD = "password";
    string DB_HOSTADDR = "127.0.0.1";
    string DB_PORT = "1234";
    string DB_SCHEMA = "test_schema";
    string tableName = "test_table";

    //Create Random data 

    for (int i = 0; i < 100000; i++) {

        dbdata.uuid = "fe2b22ba-6ce7-4bbc-8226-d7696fe7a047";
        dbdata.rndNo = i / 3.123;
        dbdata.timestamp = systemDateTimeSQLFormat(0);

        dataBuffer.push_back(dbdata);
    }

    stringstream connStream;
    connStream << "dbname = " << DB_NAME << " user = " << DB_USER << " password = " << DB_PASSWORD << " hostaddr = " << DB_HOSTADDR << " port = " << DB_PORT;

    connection dbConn(connStream.str());
    if (dbConn.is_open()) {
        cout << "Opened database successfully: " << dbConn.dbname() << endl;
    }
    else {
        cout << "Can't open database" << endl;
        return;
    }


    pqxx::nontransaction W(dbConn);
    std::string m_insertCommand = "INSERT INTO test_schema.test_table (column1, column2, column3) VALUES";

    int buffSize = dataBuffer.size();

    for (size_t i = 0; i < buffSize; i++)
    {
        unsigned int countOf$ = i * 3;
        for (unsigned int i = 0; i < 3; ++i)
        {
            if (i == 0)
            {
                m_insertCommand += "(";
            }
            else
            {
                m_insertCommand += ", ";
            }
            m_insertCommand += "$";
            std::stringstream ss;
            ss << countOf$ + i + 1;
            m_insertCommand += ss.str();
        }
        if (i < buffSize - 1)
            m_insertCommand += ") ,";
    }
    m_insertCommand += ")";


    //FOLLOWING CODE NOT POSSIBLE WITH libpqxx v12

    dbConn.prepare("insert_into_db", m_insertCommand);
    pqxx::prepare::invocation = W.exec_prepared("insert_into_db");  //prepare doesn't have a member "invocation"


    for (size_t i = 0; i < buffSize; i++)
    {
        inv(dataBuffer.at(i).rndNo)(dataBuffer.at(i).timestamp)(dataBuffer.at(i).uuid); //inv not defined
    }

    inv.exec();

return 1;
}

Wie

Is C++ a requirement? Or can you use another language? – Blades

No, C++ is a requirement. – Wie

The documentation says on the "depracation list": Class pqxx::prepare::invocation - As of 6.0, use transaction_base::exec_prepared and friends. Does that help? – Brockington

@LaurenzAlbe thanks - i found this one as well. But I couldn't find an example on how to use it in my case.. Couldn't manage to make it run. – Wie

Answer 1

pqxx::prepare::make_dynamic_params will probably solve your problem. It's solved my problem. Use this way:

for (size_t i = 0; i < buffSize; ++i)
{
    auto element = dataBuffer.at(i);
    vector<string> vect;
    vect.reserve(3);
    vect.push_back(pqxx::to_string(element.rndNo));
    vect.push_back(element.timestamp);
    vect.push_back(element.uuid);

    work.exec_params(m_insertCommand, pqxx::prepare::make_dynamic_params(vect));
}

From the version 7.6.0 dynamic_params are deprecated. params can be used instead. Here is the new solution:

for (size_t i = 0; i < buffSize; ++i) 
{
    auto element = dataBuffer.at(i);
    pqxx::params;
    params.reserve(4);
    params.append(pqxx::to_string(element.rndNo));
    params.append(element.timestamp);
    params.append(element.uuid);
    params.append(); // For example insert null variable

    work.exec_params(m_insertCommand, params);
}

Meredi