Locale-specific XFontSet expansion

Question

In my Motif application (source code is here, requires Motif 2.1+ (-lXm -lXt -lX11) and a C99-compliant compiler; on Cygwin should be additionally linked with -liconv), I'm trying to get an XFontSet instance converted from an encoding-agnostic XLFD (like -monotype-arial-medium-r-normal--*-90-*-*-p-0-*-*) to be correctly expanded into multiple individual XFontStruct entries (...-iso8859-1, ...-iso8859-5, ...-iso10646-1, etc.) matching the current locale.

I've found that in some locales (en_US.UTF-8, ru_RU.ISO-8859-5), XFontSet expansion works perfectly while in others (ru_RU.UTF-8, ru_RU) it does not (inappropriate individual fonts in a fontset), resulting in characters other than those from the Latin-1 plane rendered incorrectly.

Questions:

1. How can I rule out either Motif or X11 client libraries (libXt, libX11) as the main suspects?
2. Particularly, (how) can I display a text label using a font set (XFontSet) rather than a regular font (XFontStruct) without relying on Motif -- ideally with the Athena widget set?
3. Speaking of plain libX11 API, is my understanding correct I should experiment with XmbDrawString() and XwcDrawString() family of functions?

If you have additional comments reg. the code, which don't directly answer the above questions but can help track down the problem -- those are welcome.

---

The TL;DR version

I'm particularly interested in getting Cyrillic displayed properly regardless of the current locale, provided that:

• the locale is supported by X11, and
• Cyrillic is representable in the locale's charset.

This leaves me with the following list of locales:

• ru_RU;
• ru_RU.ISO-8859-5 (which should be the same as ru_RU, but it's not, see below);
• ru_RU.KOI8-R;
• ru_RU.CP1251;
• ru_RU.UTF-8;
• en_US.UTF-8 as well as any xx_YY.UTF-8.

ru_RU, with no loss of generality, can be easily replaced with uk_UA, be_BY or bg_BG.

In my test code, I'm creating XmLabel instances with labels produced by different XmStringCreate*()/XmStringGenerate() factory methods.

Motif 2.3.4 has generally shown to be Cyrillic-capable, with a few exceptions (or regressions with regards to Motif 2.1). I'm not sure whether the regressions stem from Motif itself (libXm) or X11 client libraries (libXt, libX11). X11 font path was the same in all tests, so a particular X server or X font server is not to be blamed.

Basically, I've found 2 issues when using Motif 2.3.4:

ru_RU locale support

Reference rendering obtained in ru_RU.ISO-8859-5 locale:

As you can see, the full Russian alphabet is displayed correctly, regardless of XmString factory used.

In each of the three cases, the application prints the contents of the XFontSet instance to stdout:

locale: ru_RU.ISO-8859-5
run-time charset: ISO-8859-5
XmStringCreateLocalized():
    XmFONT_IS_FONTSET (ru_RU.ISO-8859-5): -monotype-arial-medium-r-normal--*-90-*-*-p-0-*-*, -monotype-arial-regular-r-normal--*-90-*-*-p-0-*-*
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
XmStringCreate():
    XmFONT_IS_FONTSET (ru_RU.ISO-8859-5): -monotype-arial-medium-r-normal--*-90-*-*-p-0-*-*, -monotype-arial-regular-r-normal--*-90-*-*-p-0-*-*
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
XmStringGenerate(XmCHARSET_TEXT):
    XmFONT_IS_FONTSET (ru_RU.ISO-8859-5): -monotype-arial-medium-r-normal--*-90-*-*-p-0-*-*, -monotype-arial-regular-r-normal--*-90-*-*-p-0-*-*
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
        -monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5

---

Now, the same application launched in ru_RU locale. Motif 2.2:

For some reason, XFontSet only contains the KOI8-R font instead of the ISO-8859-5 font:

-monotype-arial-medium-r-normal--12-90-100-100-p-0-koi8-r

Motif 2.3:

In this case, nothing is rendered, while XFontSet contains more fonts than is actually necessary:

-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-1
-monotype-arial-medium-r-normal--12-90-100-100-p-0-koi8-r
-monotype-arial-medium-r-normal--12-90-100-100-p-0-microsoft-cp1251
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso10646-1

This is a regression compared to Motif 2.1.0 (versions 2.1.30 and later are affected).

ru_RU.UTF-8 locale support

Reference rendering obtained in en_US.UTF-8 locale:

In addition to the 3 XmString factories tested in 8-bit mode, XmStringGenerate(XmMULTIBYTE_TEXT) is additionally tested. XFontSet contains the following entries (actually, all possible XFontStruct entries for the Monotype Arial font):

-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-1
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-2
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-3
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-4
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
-monotype-arial-medium-r-normal--12-90-100-100-p-0-koi8-r
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-7
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-9
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-13
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-15
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso10646-1

---

Now, the same in ru_RU.UTF-8 locale:

Looks like an incorrect XFontStruct is used, despite XFontSet instance contains all the necessary XFontStruct entries:

-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-1
-monotype-arial-medium-r-normal--12-90-100-100-p-0-koi8-r
-monotype-arial-medium-r-normal--12-90-100-100-p-0-microsoft-cp1251
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso8859-5
-monotype-arial-medium-r-normal--12-90-100-100-p-0-iso10646-1

Update: it turns out in ru_RU.UTF-8 locale, the LANG environment variable is to be blamed (with the empty LANG or LANG=C or LANG=en_US.UTF-8, Cyrillic is displayed correctly), while LC_ALL can be safely set to ru_RU.UTF-8.

This is a regression compared to Motif 2.2 (versions 2.3.0 and later are affected).

Telepathist