Multi-row insert with pg-promise
Asked Answered
T

3

59

I would like to insert multiple rows with a single INSERT query, for example:

INSERT INTO tmp(col_a,col_b) VALUES('a1','b1'),('a2','b2')...

Is there a way to do this easily, preferably for an array of objects like these:

[{col_a:'a1',col_b:'b1'},{col_a:'a2',col_b:'b2'}]

I might end up with 500 records in one chunk, so running multiple queries would be undesirable.

So far I have been able to do it for a single object only:

INSERT INTO tmp(col_a,col_b) VALUES(${col_a},${col_b})

As a side question: Are insertions using ${} notation protected against SQL injections?

Tamera answered 18/5, 2016 at 13:24 Comment(0)
H
128

I'm the author of pg-promise.

In older versions of the library this was covered by simplified examples within the Performance Boost article, which is still a good read when writing high-performance database applications.

The newer approach is to rely on the helpers namespace, which is ultimately flexible, and optimised for performance.

const pgp = require('pg-promise')({
    /* initialization options */
    capSQL: true // capitalize all generated SQL
});
const db = pgp(/*connection*/);
const {ColumnSet, insert} = pgp.helpers;

// our set of columns, to be created only once (statically), and then reused,
// to let it cache up its formatting templates for high performance:
const cs = new ColumnSet(['col_a', 'col_b'], {table: 'tmp'});
    
// data input values:
const values = [{col_a: 'a1', col_b: 'b1'}, {col_a: 'a2', col_b: 'b2'}];
    
// generating a multi-row insert query:
const query = insert(values, cs);
//=> INSERT INTO "tmp"("col_a","col_b") VALUES('a1','b1'),('a2','b2')
    
// executing the query:
await db.none(query);

See API: ColumnSet, insert.

Such an insert doesn't even require a transaction, because if one set of values fails to insert, none will insert.

And you can use the same approach to generate any of the following queries:

  • single-row INSERT
  • multi-row INSERT
  • single-row UPDATE
  • multi-row UPDATE

Are insertions using ${} notation protected against sql injection?

Yes, but not alone. If you are inserting schema/table/column names dynamically, it is important to use SQL Names, which in combination will protect your code from SQL injection.


Related question: PostgreSQL multi-row updates in Node.js


extras

Q: How to get id of each new record at the same time?

A: Simply by appending RETURNING id to your query, and executing it with method many:

const query = insert(values, cs) + ' RETURNING id';
    
const res = await db.many(query);
//=> [{id: 1}, {id: 2}, ...]

or even better, get the id-s, and convert the result into array of integers, using method map:

const res = await db.map(query, undefined, a => +a.id);
//=> [1, 2, ...]

To understand why we used + there, see: pg-promise returns integers as strings.

UPDATE-1

For inserting huge number of records, see Data Imports.

UPDATE-2

Using v8.2.1 and later, you can wrap the static query-generation into a function, so it can be generated within the query method, to reject when the query generation fails:

// generating a multi-row insert query inside a function:
const query = () => insert(values, cs);
//=> INSERT INTO "tmp"("col_a","col_b") VALUES('a1','b1'),('a2','b2')
    
// executing the query as a function that generates the query:
await db.none(query);
Hyperaesthesia answered 18/5, 2016 at 14:28 Comment(34)
Thanks you for the assistance. I only use ${} for inserting data, however using your method that isn't even needed.Tamera
@Tamera I've made some changes, so you know you can generate multi-row updates and other query types in the same way ;)Hyperaesthesia
Nice improvements. Btw, would you happen to know what causes ETIMEDOUT errors? I get them occasionally (this is in an aws lambda connecting to redshift). The query returns this error after about 2 minutes & I'm not sure how to troubleshoot it further.Tamera
@vitaly-t: Great answer, thanks! Is there a way to return ids (or other info) of newly inserted records in the same query? You suggested something similar for batched queries in Inserting multiple records with pg-promise.Overthrust
@IvanSchwarz of course, simply by appending ` RETURNING id` to your generated query, and executing it with many instead of none.Hyperaesthesia
@Hyperaesthesia is it possible to implement a bulk upsert helper? basically along the lines of: insert into table(col1, col2) values ('a1','b1'),('a2','b2') ON CONFLICT (col1) DO UPDATE SET (col1, col2) = ('a1','b1'),('a2','b2')Pointblank
@Pointblank I think so, you can append ON CONFLICT part with just references (not values) to the end of the update multi-update query.Hyperaesthesia
Remember to make all columns as lowercase, otherwise you may face errors.Zemstvo
@Zemstvo why is that?Hyperaesthesia
@Hyperaesthesia here is the problem I facedZemstvo
@Zemstvo that's not really an issue, and type ColumnSet wraps columns in double quotes. And if you want to read them back the same - check out: coderwall.com/p/irklcq/…. That by the way comes from the example I provided for event receive.Hyperaesthesia
@Hyperaesthesia I agree that there may have workaround, but in case to prevent any unnecessary obstacles, to use lowercase will be more efficient for development effectsZemstvo
@Zemstvo best for development is what that article suggests - use low-case with underscores for column names in the database, and then use that automatic conversion to camel-case with pg-promise, as camel-case is the best notation for JavaScript apps.Hyperaesthesia
@Hyperaesthesia to use lower case for columns is what I suggest based on my painful experience, and upper case without double quote may face issue, that's all I want to mention.Zemstvo
How would you do insert into "table_name" as "alias" ... with this helper?Cason
@Cason What would be your use-case scenario to need an alias in this case? Anyway, this is worth a separate question, can't do it in the comments. Describe your example in a new question, and I will answer it ;)Hyperaesthesia
@Hyperaesthesia #44525279Cason
Is it possible use ` ST_SetSRID(ST_MakePoint(${Lat}, ${Lng})` on your module when i am insert like this pgp.helpers.insert(values, cs) ? @HyperaesthesiaThereupon
@sayreskabir Yes. You can either make use of property init within Column, and set mod: ':raw', or use Custom Type Formatting, which helpers respect also.Hyperaesthesia
Do you have any sample that how can i use that? I am new @HyperaesthesiaThereupon
@sayreskabir The official documentation, to which I gave you links, provides the samples.Hyperaesthesia
the latest example just doesn't work. it requires a lot of modifications.Zehe
column set and table must be declared properly.Zehe
@AlexeySh. ColumnSet is declared in the code above; and table - what table?Hyperaesthesia
@Hyperaesthesia What happens if one of the inserts fail? Does the helper method rely on transactions? I need to insert 10-20 elements at each run, but some of the inserts may fail due to constraints in the table. I still want the rest to be inserted. Any tip?Libb
@Hyperaesthesia ...if one set of values fails to insert, none will insert. is there any way to avoid this? I still want all of the good records to get inserted even if a couple fail.Rattrap
@Rattrap See this question. Or you can append ON CONFICT DO NOTHING.Hyperaesthesia
Dang, that doesn't seem to work if the value is invalid. @Libb maybe this will work in your case if still needed.Rattrap
@Rattrap If one query simply fails, the whole thing fails, because multi-row insert is an atomic operation.Hyperaesthesia
Yes I understand. I was trying to find a way to still insert the good rows and drop the bad rows.Rattrap
@vitaly-t, I'm using "pg-promise": "^11.0.2" however I don't see the db.none(query) Screenshot - ctrl.vi/i/HrU9KUq5aIntrinsic
@Intrinsic I don't think it's the version, it's your initialization is wrong somehow. Check proper initialization code ;) See pg-promise-demo.Hyperaesthesia
@vitaly-t, this is how my configuration looks like codeshare.io/r9o8AzIntrinsic
@Hyperaesthesia FYI, the same configuration is functioning okay at other places than pgpIntrinsic
F
1

Try https://github.com/datalanche/node-pg-format - e.g.

var format = require('pg-format');

var myNestedArray = [['a', 1], ['b', 2]];
var sql = format('INSERT INTO t (name, age) VALUES %L', myNestedArray); 
console.log(sql); // INSERT INTO t (name, age) VALUES ('a', '1'), ('b', '2')

works similarly with array of objects.

Feliks answered 31/7, 2021 at 16:22 Comment(1)
The accepted answer allows easy way of conflict resolution, plus the same approach for multi-row updates. This pg-format approach doesn't do either of those. It is neither flexible no expandable. And that's even omitting that the answer relies on the same query library, while this brings in an extra library.Hyperaesthesia
I
-3
CREATE TABLE "user"
(
    id         BIGINT GENERATED BY DEFAULT AS IDENTITY NOT NULL,
    first_name VARCHAR(255),
    last_name  VARCHAR(255),
    email      VARCHAR(255),
    password   VARCHAR(60),
    role       VARCHAR(255),
    enabled    BOOLEAN                                 NOT NULL DEFAULT FALSE,
    CONSTRAINT pk_user PRIMARY KEY (id)
);

INSERT INTO "user" (id,
                    first_name,
                    last_name,
                    email,
                    password,
                    role,
                    enabled)
VALUES (generate_series(1, 50),
       substr(gen_random_uuid()::text, 1, 10),
        substr(gen_random_uuid()::text, 1, 10),
        substr(gen_random_uuid()::text, 2, 5 )
            || '@' ||
        substr(gen_random_uuid()::text, 2, 5)
            || '.com',
        substr(gen_random_uuid()::text, 1, 10),
        (array['ADMIN', 'MANAGER', 'USER'])[floor(random() * 3 + 1)],
        (array[true, false])[floor(random() * 2 + 1)]
       );
Inchworm answered 3/10, 2022 at 8:14 Comment(1)
It's nice and all, but the question pertains to inserting actual values interpolated from application, not generated by database.Older

© 2022 - 2024 — McMap. All rights reserved.