SSIS Convert Between Unicode and Non-Unicode Error

Question

I have an ssis package where I am using an OLEDB source linking to SQL Server 2005 table. All columns except a date column are NVARCHAR(255). I am using an Excel destination and using a SQL statement to create the sheet in the Excel workbook, the SQL is in the excel connection manager (effectively a create table statement that creates a sheet) and is derived from the mapping of the columns from the DB.

No matter what I have done I keep getting this unicode --> non-unicode conversion error between my source and destination. Tried conversion to string[DT_STR] between S > D, removed it, changed SQL Table VARCHAR to NVARCHAR and still get this flippin error.

Because I am creating the sheet in Excel with a SQL statement I do not see any way to actually pre-define what the data types of the columns will be in the Excel sheet. I imagine it would be a default meta data but I do not know.

So between my SQL table destination and the creation of my Excel sheet with this SSIS sql statement how can I stop this error coming up?

My error is:

Error at Data Flow Task [OLE DB Source [1]]: Column "MyColumn" cannot convert between unicode and non-unicode string data types.

And for all nvarchar columns.

Appreciate any help

Thanks

Andrew

Realism

you need to use Data Conversion to convert non-unicode to unicode because excel recognizes only unicode data – Gable

Deleted the conversion part of the SSIS package, deleted the destination and excel connection, including deleting the sheet created by the SQL in the workbook and started again. now it loads ok with on erros – Realism

As I said removing the conversion worked fine now. From what I understand the SQL data column type is NVARCHAR a unicode type so wont need the conversion. – Realism

Yes my mistake .Just didn't consider that ur having NVARCHAR in source but what i said holds true excel recognizes only unicode data – Gable

Deleted the OLE DB Source and recreated it. Worked for me. – Bradybradycardia

Check the "Unicode" checkbox in the Flat File wizard from inside the Source Assistant or Destination Assistant. This will automatically set string data types to Unicode string (DT_WSTR). If you have other data types besides strings, set up a Derived Column transformation to convert the columns from Unicode (in source) to Unicode (in destination). Also, the order of workflows in Visual Studio matters in the Data Flow activity. Always create the source and update data types, then move to Derived Column transformation, then create the target and update data types. – Harrisonharrod

Answer 1

The below Steps worked for me:

1. right click on source task.

2. click on "Show Advanced editor".

3. Go to "Input and Output Properties" tab.

4. select the output column for which you are getting the error.

5. Its data type will be "String[DT_STR]".

6. Change that data type to "Unicode String[DT_WSTR]".

7. save and close.

Simonnesimonpure

@asmgx try changing the input/output file connection to use Unicode (checkbox) – Lillalillard

@asmgx actually, for me the solution was to go into the Advanced Editor for the source file, and changing the column definition, not on the destination table. – Lillalillard

I tried this but now i am getting the same conversion error in source editor (sql server) – Antihistamine

I am trying to move data from a table in one server to a table in the next. I have tried every variation of the above and I am still getting the error – Faux

Answer 2

Add Data Conversion transformations to convert string columns from non-Unicode (DT_STR) to Unicode (DT_WSTR) strings.

You need to do this for all the string columns...

Seeley

Works but there should be a "Carpal tunnel" warning/disclaimer attached to the answer. – Townsman

Biml would be handy in this case @pokerplayer23 – Marni

I'm getting this error in vs 2017 ent with an ms sql input and an excel output. They say AI is going to take all our jobs. For those of you who haven't done etl long, copying character fields from a source to a destination used to be the easy part. And this is Microsoft to Microsoft. We are a long way from any intelligence whatsoever, let alone artificial intelligence. – Erlin

Answer 3

The missing piece here is Data Conversion object. It should be in between OLE DB Source and Destination object.

Mark

Answer 4

1. First, add a data conversion block into your data flow diagram.

2. Open the data conversion block and tick the column for which the error is showing. Below change its data type to unicode string(DT_WSTR) or whatever datatype is expected and save.

3. Go to the destination block. Go to mapping in it and map the newly created element to its corresponding address and save.

4. Right click your project in the solution explorer.select properties. Select configuration properties and select debugging in it. In this, set the Run64BitRunTime option to false (as excel does not handle the 64 bit application very well).

Freeforall

Answer 5

Instead of adding an earlier suggested Data Conversion you can cast the nvarchar column to a varchar column. This prevents you from having an unnecessary step and has a higher performance then the alternative.

In the select of your SQL statement replace date with CAST(date AS varchar([size])). For some reason this does not yet change the output data type. To do this do the following:

1. Right click your OLE DB Source step and open the advanced editor.
2. Go to Input and Output Properties
3. Select Output Columns
4. Select your column
5. Under Data Type Properties change DataType to string [DT_STR]
6. Change Length to the length you specified in your CAST statement

After doing this your source data will be output as a varchar and your error will disappear.

Source

Silvey

That's a good idea, and I tried that but still it is showing a conversion error (note that the Preview Query in the OLE DB Source is showing correct results), but it is showing a red "X" and the package won't run. Conversion happens from varchar(x) to nvarchar(x). – Smilax

Try to load (fname, lname, emp_id) from Pubs (Employee) into Northwind (Employee) database. Both have SQL_Latin1_General_CP1_CI_AS collation, both are set up for SQL 2016. – Smilax

Modifying the data type in advanced editor results in "Property value is not valid" error – Smilax

Answer 6

I have been having the same issue and tried everything written here but it was still giving me the same error. Turned out to be NULL value in the column which I was trying to convert.

Removing the NULL value solved my issue.

Cheers, Ahmed

Copepod

What did you do with nulls to fix it in ssis? Was there another data conversion needed? I can't remove nulls/blank in my spreadsheet I'm feeding to my db. – Guildroy

Answer 7

No-one seems to mention this but, converting varchar to nvarchar in the source query also solves the issue.

Oloughlin

Zafer, I'm pretty sure I found that an Oracle cast to nvarchar2 didn't work for me. I had to declare the ms sql output column nvarchar to not have to do data conversion. – Erlin

This saved me so much time and effort. Thanks! – Arthropod

Answer 8

On the above example I kept losing the values, I think that delaying the Validation will allow the new data types to be saved as part of the meta data.

On the connection Manager for 'Excel Connection Manager' set the Delay Validation to False from the Properties.

Then on the data flow Destination task for Excel set the ValidationExternalMetaData to False, again from the properties.

This will now allow you to right click on the Excel Destination Task and go to Advanced Editor for Excel Destination --> far right tab - Input and Output Properties. In the External Columns folder section you will be able to now change the Data Types and Length values of the problematic columns and this can now be saved.

Good Luck!

Engage

Answer 9

I experienced this condition when I had installed Oracle version 12 client 32 bit client connected to an Oracle 12 Server running on windows. Although both of Oracle-source and SqlServer-destination are NOT Unicode, I kept getting this message, as if the oracle columns were Unicode. I solved the problem inserting a data conversion box, and selecting type DT-STR (not unicode) for varchar2 fields and DT-WSTR (unicode) for numeric fields, then I've dropped the 'COPY OF' from the output field name. Note that I kept getting the error because I had connected the source box arrow with the conversion box BEFORE setting the convertion types. So I had to switch source box and this cleaned all the errors in the destination box.

Tachygraphy

Answer 10

When creating table in SQL Server make your table columns NVARCHAR instead of VARCHAR.

Marinemarinelli

Answer 11

I think people are missing this. In my case I had 100 character columns to convert between Oracle and MS Sql. All this stuff about Data Conversion and Advanced Editor is incredibly tedious if you have a 100 separate character columns to assign. Plus SSIS being SSIS, it will sometimes reset all your 100 advanced editor changes even if you set VALIDATEEXTERNALMETADATA to false, incredibly obnoxious. I wouldn't mind doing the Data Conversion if there was some value to it but 20 years ago ETL tools used to take oracle character to ms sql characters without fussing. What Bakalolo and Zafer say is the answer if you have a lot of character columns and you can live with nvarchar, just declare all your output ms sql columns (nvarchar) and your data task will automatically assign your oracle fields into ms sql fields with no manual overrides. I have also found that the new Oracle Source (2021) doesn't complain about a unicode conversion to varchar in ms sql. A colleague just told me that the ssis wizard (it may be only in vs 2019+) to assign oracle character to ms sql varchar will do the assignments automatically with no override, but I haven't tried that personally.

2022 update - I think this is just vs 2019 created packages and later. An ado.net task reading a varchar ms sql table going to oledb (and ado.net I think) ms sql varchar will throw the unicode error. If you switch the input task to oledb reading ms sql varchar table you won't have to do the advanced editor overrides for the varchar fields. If you don't want to do advanced editor overrides (who does?) try different tasks and more oledb tasks.

Erlin

Answer 12

I just encounter same issue, I solve it in my SQL request : using convert directly

CONVERT(NVARCHAR(50),'') AS MyVarName

I need to put an empty (or fix size string) into excel file. Converting force type of MyVarName from DT-STR to DT-WSTR (unicode)

Wadewadell

Answer 13

I know this is a very old post but I ran into the same issue and found that I had to manually select the conversion component output alias as the mapping in the excel destination component. Since the names of the OLE DB Source match the excel column names it was mapping it to the OLE DB and not to the Output Alias. Such as SourceID column from the OLE DB component being named Copy of SourceID after conversion. I don't see the original question saying they specifically selected the new alias name just that they mapped to DB columns. @Serge Voloshenko post comes the closest but also does not mention to make sure the mapping happens. To a new SSIS user this might be overlooked.

Witte