How to Batch INSERT SQL Server?

Question

I am trying to batch inserting rows from one table to another.

DECLARE @batch INT = 10000;

WHILE @batch > 0
BEGIN
BEGIN TRANSACTION

    INSERT into table2
    select top (@batch) *
    FROM table1

    SET @batch = @@ROWCOUNT

COMMIT TRANSACTION

END

It runs on the first 10,000 and inserts them. Then i get error message "Cannot insert duplicate key" which its trying to insert the same primary key so i assume its trying to repeat the same batch. What logic am i missing here to loop through the batches? Probably something simple but i cant figure it out.

Can anyone help? thanks

Bruxelles

Another alternative is this – Hygrometric

Answer 1

Your code keeps inserting the same rows. You can avoid it by "paginating" your inserts:

DECLARE @batch INT = 10000;
DECLARE @page INT = 0
DECLARE @lastCount INT = 1

WHILE @lastCount > 0
BEGIN
BEGIN TRANSACTION

    INSERT into table2
    SELECT col1, col2, ... -- list columns explicitly
    FROM    ( SELECT    ROW_NUMBER() OVER ( ORDER BY YourPrimaryKey ) AS RowNum, *
              FROM      table1
            ) AS RowConstrainedResult
    WHERE   RowNum >= (@page * @batch) AND RowNum < ((@page+1) * @batch)
    SET @lastCount = @@ROWCOUNT
    SET @page = @page + 1

COMMIT TRANSACTION

END

Anjelicaanjou

Answer 2

You need some way to eliminate existing rows. You seem to have a primary key, so:

INSERT into table2
   SELECT TOP (@batch) *
   FROM table1 t1
   WHERE NOT EXISTS (SELECT 1 FROM table2 t2 WHERE t2.id = t1.id);

Retentive

Answer 3

How about using the designated set of keywords for this problem : OFFSET - FETCH

DECLARE @batch INT = 10000;
DECLARE @lastCount INT = 1;
DECLARE @totalCount INT = 0;

WHILE @lastCount > 0
BEGIN
    BEGIN TRANSACTION

    INSERT into table2
    select *
    FROM table1
    ORDER BY <id>
    OFFSET @totalCount ROWS
    FETCH NEXT @batch ROWS ONLY

    SET @lastCount = @@ROWCOUNT
    SET @totalCount += @lastCount

    COMMIT TRANSACTION

END

Abuttals

Look at this link : mssqltips.com/sqlservertip/5636/… – Christiano