SQL NVARCHAR and VARCHAR Limits
Asked Answered
G

6

109

All, I have a large (unavoidable) dynamic SQL query. Due to the number of fields in the selection criteria the string containing the dynamic SQL is growing over 4000 chars. Now, I understand that there is a 4000 max set for NVARCHAR(MAX), but looking at the executed SQL in Server Profiler for the statement

DELARE @SQL NVARCHAR(MAX);
SET @SQL = 'SomeMassiveString > 4000 chars...';
EXEC(@SQL);
GO

Seems to work(!?), for another query that is also large it throws an error which is associated with this 4000 limit(!?), it basically trims all of the SQL after this 4000 limit and leaves me with a syntax error. Despite this in the profiler, it is showing this dynamic SQL query in full(!?).

What exactly is happening here and should I just be converting this @SQL variable to VARCHAR and get on with it?

Thanks for your time.

Ps. It would also be nice to be able to print out more than 4000 chars to look at these big queries. The following are limited to 4000

SELECT CONVERT(XML, @SQL);
PRINT(@SQL);

is there any other cool way?

Grekin answered 28/9, 2012 at 12:22 Comment(2)
MAX is not a synonym for the 4000 limit, its 1..4000 or MAXTest
PRINT will concatenate at 4000 characters (for unicode) or 8000 chars (for single byte encodings). I suspect that is the source of the confusion here.Awning
D
252

I understand that there is a 4000 max set for NVARCHAR(MAX)

Your understanding is wrong. nvarchar(max) can store up to (and beyond sometimes) 2GB of data (1 billion double byte characters).

From nchar and nvarchar in Books online the grammar is

nvarchar [ ( n | max ) ]

The | character means these are alternatives. i.e. you specify either n or the literal max.

If you choose to specify a specific n then this must be between 1 and 4,000 but using max defines it as a large object datatype (replacement for ntext which is deprecated).

In fact in SQL Server 2008 it seems that for a variable the 2GB limit can be exceeded indefinitely subject to sufficient space in tempdb (Shown here)

Regarding the other parts of your question

Truncation when concatenating depends on datatype.

  1. varchar(n) + varchar(n) will truncate at 8,000 characters.
  2. nvarchar(n) + nvarchar(n) will truncate at 4,000 characters.
  3. varchar(n) + nvarchar(n) will truncate at 4,000 characters. nvarchar has higher precedence so the result is nvarchar(4,000)
  4. [n]varchar(max) + [n]varchar(max) won't truncate (for < 2GB).
  5. varchar(max) + varchar(n) won't truncate (for < 2GB) and the result will be typed as varchar(max).
  6. varchar(max) + nvarchar(n) won't truncate (for < 2GB) and the result will be typed as nvarchar(max).
  7. nvarchar(max) + varchar(n) will first convert the varchar(n) input to nvarchar(n) and then do the concatenation. If the length of the varchar(n) string is greater than 4,000 characters the cast will be to nvarchar(4000) and truncation will occur.

Datatypes of string literals

If you use the N prefix and the string is <= 4,000 characters long it will be typed as nvarchar(n) where n is the length of the string. So N'Foo' will be treated as nvarchar(3) for example. If the string is longer than 4,000 characters it will be treated as nvarchar(max)

If you don't use the N prefix and the string is <= 8,000 characters long it will be typed as varchar(n) where n is the length of the string. If longer as varchar(max)

For both of the above if the length of the string is zero then n is set to 1.

Newer syntax elements.

1. The CONCAT function doesn't help here

DECLARE @A5000 VARCHAR(5000) = REPLICATE('A',5000);

SELECT DATALENGTH(@A5000 + @A5000), 
       DATALENGTH(CONCAT(@A5000,@A5000));

The above returns 8000 for both methods of concatenation.

2. Be careful with +=

DECLARE @A VARCHAR(MAX) = '';

SET @A+= REPLICATE('A',5000) + REPLICATE('A',5000)

DECLARE @B VARCHAR(MAX) = '';

SET @B = @B + REPLICATE('A',5000) + REPLICATE('A',5000)


SELECT DATALENGTH(@A), 
       DATALENGTH(@B);`

Returns

-------------------- --------------------
8000                 10000

Note that @A encountered truncation.

How to resolve the problem you are experiencing.

You are getting truncation either because you are concatenating two non max datatypes together or because you are concatenating a varchar(4001 - 8000) string to an nvarchar typed string (even nvarchar(max)).

To avoid the second issue simply make sure that all string literals (or at least those with lengths in the 4001 - 8000 range) are prefaced with N.

To avoid the first issue change the assignment from

DECLARE @SQL NVARCHAR(MAX);
SET @SQL = 'Foo' + 'Bar' + ...;

To

DECLARE @SQL NVARCHAR(MAX) = ''; 
SET @SQL = @SQL + N'Foo' + N'Bar'

so that an NVARCHAR(MAX) is involved in the concatenation from the beginning (as the result of each concatenation will also be NVARCHAR(MAX) this will propagate)

Avoiding truncation when viewing

Make sure you have "results to grid" mode selected then you can use

select @SQL as [processing-instruction(x)] FOR XML PATH 

The SSMS options allow you to set unlimited length for XML results. The processing-instruction bit avoids issues with characters such as < showing up as &lt;.

Duet answered 28/9, 2012 at 12:23 Comment(18)
Thanks. There is a wealth of documentation and I have obviously got confused here. I am getting an error for this dynamic SQL query exactly on 4000 chars and there is nothing else I can find that is wrong with it... Thanks for your time.Grekin
@Killercam - You might be getting an implicit cast to nvarchar(4000) along the way. If a string literal is less than 4,000 characters then it is treated as nvarchar(x). Concatenating to it another nvarchar(x) value will truncate rather than upcast to nvarchar(max)Duet
Perhaps. This error is a strange one - and is taking time to debug. Thanks again for your time... This clearly is the correct answer nad this was a foolish question - I am at the stage where I am doubting my beliefs!Grekin
this is the same as what I have i.e. SELECT CONVERT(XML, @SQL);. Thanks again...Grekin
@Killercam - It's different if your text contains any characters such as < > & . SELECT CONVERT(XML, '> <'); gives an error. select '> <' as [processing-instruction(x)] FOR XML PATH works fine.Duet
I have copied the statement to the question. I am at a loss, it is no doubt some stupid error but I can't see it for the life of me...Grekin
@Killercam - You are probably getting truncation as per my first comment. Try changing the assignment to DECLARE @SQL NVARCHAR(MAX) = ''; SET @SQL = @SQL + so that an NVARCHAR(MAX) is involved in the concatenation.Duet
Adding SET @SQL = N'...' + N'...'; has fixed this, I was not aware that you had to use N prefixes for declaration strings. Again, thanks very much for your time...Grekin
@Killercam - Probably you have a string between 4,000 and 8,000 characters. With the N prefix that will be treated as nvarchar(max) without it it will be treated as varchar(n) then implicitly cast to nvarchar(4000) when you concatenate to an nvarcharDuet
i am enlightened by this answerGaiter
really helpful and doubt clearing answer.it helped me. thanks a lotSweepings
I can't believe after a few years at T SQL I still meet varchar(max) truncate case that I can't explain. I hope this will be the last explanation I read up on this!!Artefact
I had a table with varchar 200. Then later, I've modified to nvarchar(max). But When i insert the string, it allows only for 100 characters. What will be the issue?Noll
When debugging, keep in mind that PRINT will also truncate nvarhcar(max) at 4000 chars! That one cost me some time. Use DATALENGTH() (like Mr. Smith does above)Sever
This answer is very informative, things you don't learn in a class - but should! Laying it out and clearly defining the do's and don't's and why's .Thank you!!Orchard
I added screenshots for the XML PATH trick at the end, see "Avoiding truncation when viewing", but the edit was taken back, see my screenshots at stackoverflow.com/posts/12639972/revisions.Charentemaritime
Since the nvarchar(max) trick does not work for my dynamic query, the only way to see the output is by the XML output in SSMS. I therefore wanted to know how I can run that output again as SQL, but it was closed as a duplicate, see Run long SQL code that is inside a tiny XML frame and that is bigger than the 4000 characters max. What shall I do if the nvarchar(max) trick does not avoid cutting the EXEC input?Charentemaritime
You need to read this answer to understand the cases that truncation occurs. As I can't see your code I can't tell you the issue with itDuet
F
8

Okay, so if later on down the line the issue is that you have a query that's greater than the allowable size (which may happen if it keeps growing) you're going to have to break it into chunks and execute the string values. So, let's say you have a stored procedure like the following:

CREATE PROCEDURE ExecuteMyHugeQuery
    @SQL VARCHAR(MAX) -- 2GB size limit as stated by Martin Smith
AS
BEGIN
    -- Now, if the length is greater than some arbitrary value
    -- Let's say 2000 for this example
    -- Let's chunk it
    -- Let's also assume we won't allow anything larger than 8000 total
    DECLARE @len INT
    SELECT @len = LEN(@SQL)

    IF (@len > 8000)
    BEGIN
        RAISERROR ('The query cannot be larger than 8000 characters total.',
                   16,
                   1);
    END

    -- Let's declare our possible chunks
    DECLARE @Chunk1 VARCHAR(2000),
            @Chunk2 VARCHAR(2000),
            @Chunk3 VARCHAR(2000),
            @Chunk4 VARCHAR(2000)

    SELECT @Chunk1 = '',
           @Chunk2 = '',
           @Chunk3 = '',
           @Chunk4 = ''

    IF (@len > 2000)
    BEGIN
        -- Let's set the right chunks
        -- We already know we need two chunks so let's set the first
        SELECT @Chunk1 = SUBSTRING(@SQL, 1, 2000)

        -- Let's see if we need three chunks
        IF (@len > 4000)
        BEGIN
            SELECT @Chunk2 = SUBSTRING(@SQL, 2001, 2000)

            -- Let's see if we need four chunks
            IF (@len > 6000)
            BEGIN
                SELECT @Chunk3 = SUBSTRING(@SQL, 4001, 2000)
                SELECT @Chunk4 = SUBSTRING(@SQL, 6001, (@len - 6001))
            END
              ELSE
            BEGIN
                SELECT @Chunk3 = SUBSTRING(@SQL, 4001, (@len - 4001))
            END
        END
          ELSE
        BEGIN
            SELECT @Chunk2 = SUBSTRING(@SQL, 2001, (@len - 2001))
        END
    END

    -- Alright, now that we've broken it down, let's execute it
    EXEC (@Chunk1 + @Chunk2 + @Chunk3 + @Chunk4)
END
Furfur answered 28/9, 2012 at 12:41 Comment(0)
S
3

You mus use nvarchar text too. that's mean you have to simply had a "N" before your massive string and that's it! no limitation anymore

DELARE @SQL NVARCHAR(MAX);
SET @SQL = N'SomeMassiveString > 4000 chars...';
EXEC(@SQL);
GO
Sollars answered 18/6, 2015 at 7:32 Comment(1)
This is not the entire picture... If you use the N prefix and the string is <= 4,000 characters long it will be typed as nvarchar(n) where n is the length of the string. So N'Foo' will be treated as nvarchar(3) for example. If the string is longer than 4,000 characters it will be treated as nvarchar(max). If you don't use the N prefix and the string is <= 8,000 characters long it will be typed as varchar(n) where n is the length of the string. If longer as varchar(max). For both of the above if the length of the string is zero then n is set to 1.Grekin
P
1

The accepted answer helped me but I got tripped up while doing concatenation of varchars involving case statements. I know the OP's question does not involve case statements but I thought this would be helpful to post here for others like me who ended up here while struggling to build long dynamic SQL statements involving case statements.

When using case statements with string concatenation the rules mentioned in the accepted answer apply to each section of the case statement independently.

declare @l_sql varchar(max) = ''

set @l_sql = @l_sql +
case when 1=1 then
    --without this correction the result is truncated
    --CONVERT(VARCHAR(MAX), '')
 +REPLICATE('1', 8000)
 +REPLICATE('1', 8000)
end

print len(@l_sql)
Pooh answered 9/4, 2019 at 13:51 Comment(0)
H
0
declare @p varbinary(max)
set @p = 0x
declare @local table (col text)

SELECT   @p = @p + 0x3B + CONVERT(varbinary(100), Email)
 FROM tbCarsList
 where email <> ''
 group by email
 order by email

 set @p = substring(@p, 2, 100000)

 insert @local values(cast(@p as varchar(max)))
 select DATALENGTH(col) as collen, col from @local

result collen > 8000, length col value is more than 8000 chars
Hypnos answered 23/3, 2017 at 8:18 Comment(0)
C
0

This came up when searching for a solution to Azure Data Studio (ADS). ADS also defaults to limiting each text column to about 65K.

Last year, @alanrenmsft provided the ability to adjust this setting in ADS. Increasing this setting now allows full JSON (or text) to be returned accordingly.

Setting in ADS

Reference: https://github.com/microsoft/azuredatastudio/issues/392#issuecomment-1239773578

-- MUST SET the following to pull back the full JSON payload.
-- "mssql.query.maxCharsToStore": 2147483647
Checani answered 12/9, 2023 at 15:52 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.