SSIS Flat file could not deal with NUL (\x00) value?
Asked Answered
U

2

6

I am trying to load data from text files to database. My source files contain null character NUL somehow (Picture1).

Picture1

I just make all the fields as one column (delimited with {CR}{LF}). Then I do the preview of the data.

Picutre2

The data is just what we need. But then when I run the package, the data changed, not like what I see in data preview. I added a data viewer to see the data.

Picture3

Picture4

The number 1 disappear in the first row (see the red). It seems that flat file reading ends at NUL character. But my Row delimiter is {CR}{LF}, it doesn't make sense the number 1 in the end disappear. Can anyone tell me why is that?

Unscramble answered 27/6, 2013 at 7:36 Comment(10)
Have you tried importing as fixed width? It actually looks like a fixed width file as I don't see any tab characters in your screen grabJochbed
A sample file (not an image) will greatly help us here, as then we can test against that and try to replicate the the problem and see how we get around it. Those images look like BIDS as well; are you using SQL Server 2008(R2)? Certainly your screenshots looks like Windows XP; which has been out of support for years.Continuate
Oh wait.. This question was asked in 2013. Perhaps @Yahfoufi would have been better asking a new question.Continuate
@Larnu I will try to update the question in a while and to provide a sample file. Also i didn't asked a new question since I am facing the same problem but the file content is differentSlabber
Why update someone else's question, @Slabber ? Should this not be a new one? Editing this question (extensively) will hide the original question the initial OP had.Continuate
@Larnu If I posted a new question then certainly it will be marked as duplicate.Slabber
Of what, @Yahfoufi? This question isn't a duplicate candidate, as there are no upvoted answers. if you're saying there is a duplicate out there, then you already have the answer. If, however, you've reviewed other questions that are similar but didn't answer your question, then reference those in the question and explain why those solutions didn't work; so that we know what you've already tried and tested, and failed to implement (or doesn't work).Continuate
@Larnu Thanks for the info. I didn't knew that duplicates are considered based on answers. If the bounty ended with no solutions I will ask a new question.Slabber
A question with no answers hardly offers a good duplicate candidate, when the (other) question remains unanswered, @Slabber :)Continuate
@Larnu Lucky me:) All what I needed where mentioned in the answer below!! Now, there is no need for a new question.Slabber
J
4

Reproducing the error

First of all, I would like to show the steps to reproduce this error using Notepad++ editor.

I created a text file called TestNUL that contains data similar to the screenshot posted in the question (commas are placed where NUL objects should be):

enter image description here

Now, Go To Edit menu strip >> Character Panel

enter image description here

Now the ASCII character panel is shown, double click on the NULL value in order to add it to the text:

enter image description here

Now the text file will looks like:

enter image description here

You can use the following link to download the file:

Removing NUL character using Notepad++

To remove this character you can simply open Notepad++, Click Ctrl + H to open the Find and Replace dialog. Then select to use Regular Expressions and replace \x00 with an empty string:

enter image description here

All NUL characters are removed:

enter image description here

Find and replace in multiple file

If you are looking to find and replace this character in multiple files, then you can use notepad++ to do this using Find in Files feature:

Automating the process Within SSIS

Since the issue occurs at run-time not while previewing data, you can simply add a Script Task before the data flow task to replace all \x00 values with an empty string. You can read the text file path from the flat file connection manager or you can store it in a variable. You can use a similar C# code:


public void Main()
{
    string FilePath = Dts.Connections["SourceConnection"].ConnectionString;

    string text = System.IO.File.ReadAllText(FilePath);
    text = text.Replace(Convert.ToChar(0x0).ToString(), "");
    System.IO.File.WriteAllText(FilePath, text);

    Dts.TaskResult = (int)ScriptResults.Success;
}

If you are working with large text files then you can use System.IO.StreamReader and System.IO.StreamWriter classes to read the file line by line using ReadLine() function.

Experiments

I created a package and added two flat file connection manager, the source reads from TestNUL.txt file and the destination create a new TestNUL_edited.txt file with the same structure. I added a Script Task with the code above and added a data viewer in the Data Flow Task, the following screenshot shows how the rows are not corrupted:

enter image description here

enter image description here

Also the following screenshot shows how the NUL values are removed from the source file after running the Script Task:

enter image description here

References

Jessy answered 22/8, 2019 at 20:57 Comment(0)
K
0

Map every field in his column (using Tab {t} as column delimiter I suppose, and {CR}{LF} as row delimiter) and try again.

Kasiekask answered 27/6, 2013 at 7:44 Comment(2)
column delimiter is not necessary, because I don't need to break the rows to a few of columns. I just need it as one columnUnscramble
in your Flat File Source try to check or uncheck option "Retain null values from the source as null values in the data flow"Kasiekask

© 2022 - 2024 — McMap. All rights reserved.