How does Windows Notepad interpret characters?

Question

I was wondering how Windows interprets characters.

I made a file with a hex editor with the 3 bytes E3 81 81.
Those bytes are the ぁ character in UTF-8.

I opened the notepad and it displayed ぁ. I didn't specify the encoding of the file, I just created the bytes and the notepad interpreted it correctly.

Is notepad somehow guessing the encoding?
Or is the hex editor saving those bytes with a specific encoding?

Brockway

Answer 1

If the file only contains these three bytes, then there is no information at all about which encoding to use.

A byte is just a byte, and there is no way to include any encoding information in it. Besides, the hex editor doesn't even know that you intended to decode the data as text.

Notepad normally uses ANSI encoding, so if it reads the file as UTF-8 then it has to guess the encoding based on the data in the file.

If you save a file as UTF-8, Notepad will put the BOM (byte order mark) EF BB BF at the beginning of the file.

Snapdragon

Yeah but i didnt save the file as UTF-8 or whatever, ive created the bytes. Thank you – Brockway

@Guffa, This answer is wrong dude. He created the file with HxD. The file is only 3 bytes without BOM. Yet notepad didn't open it with ascii text, it does it in UTF8 text? How did notepad know this? – Foolscap

Answer 2

Notepad makes an educated guess. I don't know the details, but loading the first few kilobytes and trying to convert them from UTF-8 is very simple, so it probably does something similar to that.

Diller

Answer 3

...and sometimes it gets it wrong... https://ychittaranjan.wordpress.com/2006/06/20/buggy-notepad/

Importance

Answer 4

There is an easy and efficient way to check whether a file is in UTF-8. See Wikipedia: http://en.wikipedia.org/w/index.php?title=UTF-8&oldid=581360767#Advantages, fourth bullet point. Notepad probably uses this.

Wikipedia claims that Notepad used the IsTextUnicode function, which checks whether a patricular text is written in UTF-16 (it may have stopped using it in Windows Vista, which fixed the "Bush hid the facts" bug): http://en.wikipedia.org/wiki/Bush_hid_the_facts.

Nitroso

Bush hid the facts don't seem to work with me here... win 8.1. Yet the questioner's code works. – Foolscap

Answer 5

how to identify the file is in which encoding ....?

Go to the file and try to Save As... and you can see the default (current) encoding of the file (in which encoding it is saved).

Telega