Delphi 7 - Why does Windows 7 change encoding of characters in runtime?

Question

I have a delphi 7 form:

and my code:

when I run this form in Windows 7, I see:

In design time, form had polish letters in first label, but it doesn't have them in runtime. It looks ok on Vista or Windows XP. When I set caption of second label in code, everything works fine and characters are properly encoded.

First 5 codes of top label on Windows 7: 65 97 69 101 83

First 5 codes of top label on Windows Vista/XP: 165 185 202 234 140

First 5 codes of bottom label on every system: 165 185 202 234 140

Windows 7 changes encoding, why? My system settings seem to be ok. I have proper language set for non-unicode applications in control panel.

EDIT

This problem is not only related with labels on forms, but also with FastReport (where switching to EASTERN_CHARSET resolves the problem) or with accesing Microsoft Excel through COM interface.

Trilateration

I'm not familiar with Delphi, but is it possible that the Form Designer allows specifying a different encoding than your default codepage (i.e. Windows-1250)? And what version of Delphi is this? IIRC Unicode support has been added only recently... – Helwig

@0xA3 This is Delphi 7 and it is not unicode application, but I set polish language for non-unicode applications in regional settings. That is why Label2 displays correctly, but first one still doesn't work. – Trilateration

Did you check how Label1 and Label2 are defined in the DFM? How is Label1.caption saved in the DFM? – Stpierre

@Ken Bourassa DFM stores #260#261#280#281#346#347#262#263#323#324. This looks like Unicode. Letters match unicode codes, so there is nothing wrong with dfm. – Trilateration

Does it work if you run the app in XP or Vista compatibility mode (right-click the .exe, Properties -> Compatibility)? – Helwig

@0xA3 No, it doesn't work in compatibility mode. – Trilateration

Answer 1

Answers to this question solve my problem:

GetThreadLocale returns different value than GetUserDefaultLCID?

One solution:

The strange thing we found is that switching to a different regional settings via Control Panel and then switching back to NZ resolves the issue. I'd be curious to know if the same workaround resolves it for you just to verify that we're seeing the same phenomenon.

and second:

initialization
  SetThreadLocale(LOCALE_USER_DEFAULT);
  GetFormatSettings;

Both solutions work great and problem with application disappears.

Trilateration

Answer 2

I did reproduce the behavior in Delphi 2010 in win XP.

procedure Button1Click(Sender : TObject);
begin
  ShowMessage(AnsiString(Label1.Caption));
end;

In this situation, the conversion of Label1.Caption to AnsiString is done through WideCharToMultiByte (Windows API).

The API has the following note :

The ANSI code pages can be different on different computers, or can be changed for a single computer, leading to data corruption. For the most consistent results, applications should use Unicode, such as UTF-8 or UTF-16, instead of a specific code page, unless legacy standards or data formats prevent the use of Unicode. If using Unicode is not possible, applications should tag the data stream with the appropriate encoding name when protocols allow it. HTML and XML files allow tagging, but text files do not.

So, my best guess is that the difference in behavior come from the fact that the version of Windows 7 you have has a different active CodePage than on your vista/XP stations.

I still have to find how to get the active codepage on a system... My best guess is that it is defined in the regional settings in the control panel. But I still need to verify this...

Stpierre

Control Panel settings seem to be ok. I am sure that this is problem of Windows 7 having another approach to string convertion, but how do I resolve it? – Trilateration

Try calling GetThreadLocale from your application on the various system, just to make sure the same value is returned before we investigating further... – Stpierre

Good idea. I'll do that tomorrow. – Trilateration

Answer 3

You ran into what I consider a "bug" in the TWriter.WriteString and TWriter.ReadString methods. Those two methods are internally used by Delphi to move your TLabel.Caption from the actual live object at design time into the DFM file and then back into the live object at run time.

If you look at the code for the mentioned two routines, you'll notice (in shock I assume) that the actual stuff that goes into the stream is converted to Unicode using the operating system's default code page! That's fine and dandy as long as the code page used on the development machine exactly matches the code page used on the test machine, and they probably don't match, and that's most likely why you get the error. Please note the EASTEUROPEAN_CHARSET you're setting for the Caption on the form has absolutely no value, because the TWriter.WriteString method has no idea about it!

I've got an bug report on this issue on QC, it's been there for many years... They probably think it's "by design", but I don't think it's an very good design.

The solution I'd recomand is a fast switch to Delphi 2010. I'm an Delphi developer in Romania, and I've had lots and lots of problem with this kind of stuff, but now it's all in the past because Delphi 2010 is UNICODE so I no longer need to worry about code page conversions.

If you can't switch to Delphi 2010 you might want to "hack" the Classes.pas file and change the TReader.ReadString routine to always do the conversion using YOUR code page, not the system default.

Metz

Nice to know, but this problem is related only with Windows 7 and I already found a solution. Switching to Delphi 2010 will be very complicated. My application uses heavily JVCL 2 and other old components and has about 1000 forms. Delphi 2010 waits on the shelf for more spare time. – Trilateration

Answer 4

Check the Font.Charset property of the label. While I don't know how it got changed (was it pre-created for some wizard?) - it might have a different locale from the system one.

Blotto

Form has Font.Charset set to DEFAULT_CHARSET and labels have ParentFont set to true. – Trilateration