Strange Notepad++ HEX-editor plugin

Question

The goal is to write byte array to file. I have byte array fits[] with some bytes and then:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace _32_to_16
{
    class Program
    {
        static void Main(string[] args)
        {
            byte[] fits = File.ReadAllBytes("1.myf");
            byte[] img = new byte[fits.Length / 2];
            for (int i = 0; i < fits.Length; i += 4) //Drops 2 high bytes
            {
                img[i/2] = fits[i + 2];
                img[i/2 + 1] = fits[i + 3];
            }
            File.WriteAllBytes("new.myf", img);
        }
    }
}

Before writing to the file img[] has same values:

• img[0]=0x31
• img[1]=0x27
• img[2]=0x31
• img[3]=0xe2
• and so on...

After writing to file, in HEX editor i see

• 00000000: 31 27 31 3f and other wrong values.

Sometimes, with other fits[] values, img[] array write correctly to file. What I`m doing wrong?
File for test 1.myf (which makes wrong results) https://www.dropbox.com/s/6xyf761oqm8j7y1/1.myf?dl=0 File for test 2.myf (correct writes to file) https://www.dropbox.com/s/zrglpx7kmpydurz/2.myf?dl=0

I simplified the code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace _32_to_16
{
    class Program
    {
        static void Main(string[] args)
        {
            byte[] img_correct = new byte[8] { 0xbd, 0x19, 0xbd, 0x72, 0xbd, 0x93, 0xbd, 0xf7 };
            File.WriteAllBytes("img_correct.myf", img_correct);

            byte[] img_strange = new byte[8] { 0x33, 0x08, 0x33, 0xac, 0x33, 0xe3, 0x33, 0x94 };
            File.WriteAllBytes("img_strange.myf", img_strange);
        }
    }
}

in HEX-editor img_correct.myf looks like this: bd 19 bd 72 bd 93 bd f7

in HEX-editor img_strange.myf looks like this: 33 08 33 3f 3f 3f

Stokehole

You're not swallowing exceptions, are you? Maybe the file is locked and not actually being (over)written? – Sapienza

No any exception. No locking file (create new one). I tried to use BinaryWriter — same result. dropbox.com/s/k4zctcy9v2744ke/bw.JPG?dl=0 and dropbox.com/s/xex5m5gzm2aswnu/notepad%2B%2B.JPG?dl=0 – Stokehole

Well, uh, that's weird. Can you try to make an minimal reproducible example so that we can try it out? – Sapienza

I cut out all extra code and left only the important. Edited the startup message. I have no more ideas where to go. – Stokehole

After the call of WriteAllBytes, try reading the new.myf file to another byte array and then compare img with the new array byte by byte. They should be the same. – Jazzy

Is that file being written to with Unicode format? – Helldiver

Your code will throw an IndexOutOfRangeException if fits.Length is not a multiple of 4 (you're checking if i is less than fits.Length but you're going beyond that with fits[i + 3]). Other than that however, it should work as expected - I cannot reproduce the problem. As Martin said, what result do you get when you read that file back with File.ReadAllBytes? Are you sure you're checking the same file, given those relative paths? – Erectile

@MartinStaufcik, yes, the comparison gave the result that this arrays are the same. The problem was in HEX-editor (Notepad++ plugin). Marcio Rinaldi gave advice in answers to try in different editor. – Stokehole

Answer 1

You're using the HEX-Editor plugin from Notepad++, which seems to have a problem reading binary files.

Try with another hex editor and it should display the correct values.

Here's a screenshot of HxD and HEX-Editor displaying the same file

Fatidic

Yes that's right, I'm using Notepad++ plugin. In HxD all looks correct! But this problem occurs not only whith N++, but with some different programms too. I translate 32bit astronomical image to 16bit and third party software has the same problems in visualisation images after this convertions. I will try different way to translate 32bit to 16bit. This way seems fast, but unsuccessful. Thanks for answer! – Stokehole

Holy shit!!! This was holding up our project because we thought our data logging was broken. I googled "writeallbytes 3f" in desperation and found this. Thank you so much! – Theolatheologian

Just in case somebody else is likewise desperate... Disable 'Autodetect character encoding' in Preferences/MISC menu of NPP and set encoding to ansi or something like that. Hopefully NPP won't play with bytes shown in HEX mode! – Nutrilite

Answer 2

For Full Width Colon "：" the correct Unicode format is : U+EF1A

But in NotePad ++ the "：" in Hex Editor show "EFBC9A" and not "EF1A".

Because this is an UTF8 Encoding & this is not in the Unicode format.

If I put "EFBC9A" in another editor, it show the Korean character "벚".

When you direct type in Hex Editor, make sure to use UTF8 encoding, but when you are not in Hex Editor, make sue to use Unicode Format and not UTF8 encoding.

So people is confusing with UTF8 Encoding and Unicode format.

By the way: the U+EF1A --> "：" can be put in folder name in Windows System.

Eileen

Answer 3

Is your source file size divisible by 4? If not, then any remaining bytes will be ignored at the end of the operation. The i += 4 is going to skip over them. You'll need to handle those at the end, after your for loop, if the source (fits) file isn't perfectly divisible by 4.

Matthias

Answer remains the same. You're potentially dropping additional bytes by jumping over them. The code, as it stands, is intended to jump these bytes, as it's grabbing the top two bytes from each set of four and copying them to your smaller byte array. I would suggest doing the numbering logic a little differently, from the point of view of img, rather than from the point of view of fits. You're making your indexing unnecessarily complex that way. Output each pass of the for loop and all your bytes to the console and see what's going on. – Matthias

I simplified the code and corrected startup message. – Stokehole

I see. There was no real problem at all, just the hex editor. You could have tried reading in the newly written file again and checking the byte values in Visual Studio or from console output. You don't even need a hex editor tbh. Thanks :) – Matthias