read resourcebundle as UTF-8. getString() Method seems to change encoding to ISO-8859

Question

I have the honorable assignment to change the encoding of our complete workspace, projects and files to the UTF-8 encoding. We have several Resourcebundles which used to code special chars with unicode. We also wanted to get rid of that unicode stuff by switching to UTF-8 so I changed the encoding of the Resourcebundles (.properties) files too and replaced the Unicode characters.

We also have german resourcebundles and some chars like

Ä, Ö, Ü, ß. ä, ö, ü and also special characters like „ or “

are not shown properly in the browser. Example:

Resourcebundleentry:

executeShellCommand.label = Shellkommando ausführen

Result in browser:

The resourcebundles are read with the Java.util.ResourceBundle.getString(String key) Method:

    public String getLocalizedString(ResourceBundle bundle, String key) {
    try {
        System.out.println("getLocalizedString, key: " + key + ", resourcebundle: " + bundle.getString(key));
        return bundle.getString(key);
    } catch (MissingResourceException e) {
        return key;
    }
}

If i check the output of the above Sysout i get following: getLocalizedString, key: executeShellCommand.label, resourcebundle: Shellkommando ausführen

It seems that the getString(key) method changes the encoding of the chars while reading them from the bundles to the standard resourcbundleencoding(ISO-8859).

I tried to counter this issue:

    public String getLocalizedString(ResourceBundle bundle, String key) {
    try {
        System.out.println("getLocalizedString, key: " + key + ", resourcebundle: " + new String (bundle.getString(key).getBytes(), "UTF-8"));
        return new String (bundle.getString(key).getBytes(), "UTF-8");
    } catch (MissingResourceException e) {
        return key;
    } catch (UnsupportedEncodingException e) {
        return key;
    }
}

This helped to recover the most special characters but there are still a plenty of them which are not shown properly:

I also checked the content-type configuration of the WebApp and of every single request which gets the resource bundles everything is utf-8.

Does anyone have an idea how to prevent the getString()-Method from changing the encoding or is there a better way to solve this issue?

Panache

If your resourcebundles are stored in properties files: Java assumes by default that they are encoded in ISO-8859. – Madaih

I already figured it out by checking the outputs. but is there a way to force the getString method to read the chars as UTF-8 and not ISO-8859? – Panache

I think in your case the problem begins earlier where you instantiate the resource bundle. You have to read from a stream in ISO-encoding; Java converts that to UTF-8 in strings, so reading the string as UTF-8 is correct. (See docs.oracle.com/javase/7/docs/api/java/util/…) – Madaih

yes that is exactly what I want to do. but neither the PropertiesResourcebundle nor the Control class has any specification to change the reading encoding format. Is there any way to open the Bundle as UTF-8 encoded`?? – Panache

Load the bundle with a custom Control: https://mcmap.net/q/99507/-how-to-use-utf-8-in-resource-properties-with-resourcebundle – Lucky

Answer 1

Java ResourceBundles assume ISO-8859. I think you'll need to use Properties instead of ResourceBundle.

InputStream utf8in = getClass().getClassLoader().getResourceAsStream("/path/to/utf8.properties");
Reader reader = new InputStreamReader(utf8in, "UTF-8");
Properties props = new Properties();
props.load(reader);

Stafford

this works but needs to change the every single mehtod which reads from a resource bundles. and I have many of them. Is there no way with resourcebundles? I cannot imagine, that developers in japan or china cannot work with bundles due to tthis issue? – Panache

I see that PropertyResourceBundle has a constructor that takes a java.io.Reader. Im guessing you could make use of that. – Stafford

yes i finally solved it this way. the following code solved the problem: @Uwe Allner Thank you too. java.io.InputStream applStream = getClass().getClassLoader().getResourceAsStream(applPath); Reader applReader = new InputStreamReader(applStream, "UTF-8"); applBundle = new PropertyResourceBundle(applReader); – Panache

You are actually losing the whole point of a bundle by doing it this way. The great thing about bundles is that properties in the language_country file can override those in the country file which also override the base file. Take a look at my other answer. It's a proper hack but I think you'd keep the bundle behaviour. Another option is to chain a few PropertyResourceBundle instances together (one for base, base_language and base_language_country). See ResourceBundle.setParent(). – Stafford

you are right at that point. I have to ask for the local language before i select a Resource so there is no proper behaviour. This is not an elegant way to solve the problem. I check your hack. – Panache

Answer 2

How's this for a hack!!!

public class HackClassLoader extends ClassLoader {

    public HackClassLoader(ClassLoader parent) {
        super(parent);
    }

    @Override
    public InputStream getResourceAsStream(String name) {
        InputStream utf8in = getParent().getResourceAsStream(name);
        if (utf8in != null) {
            try {
                byte[] utf8Bytes = new byte[utf8in.available()];
                utf8in.read(utf8Bytes, 0, utf8Bytes.length);
                byte[] iso8859Bytes = new String(utf8Bytes, "UTF-8").getBytes("ISO-8859-1");
                return new ByteArrayInputStream(iso8859Bytes);
            } catch (IOException ex) {
                throw new RuntimeException("Could not load " + name, e);
            } finally {
                utf8in.close();
            }
        }
        return null;
    }
}

ClassLoader hackLoader = new HackClassLoader(getClass().getClassLoader());
ResourceBundle bundle = ResourceBundle.getBundle("foo", Locale.UK, hackLoader);

Stafford

dude the resourcebundles are not found if i use your hack. [31.07.14 19:02:19:279 CEST] 00000038 ServiceManage E at.co.sbs.kixfs.util.Trace traceExceptionCaught exception_caught: java.util.MissingResourceException: Can't find resource for bundle at.co.sbs.operator.backend.resources.ApplicationResources, key de – Panache

Where are your property files? Perhaps use getClass().getClassLoader() instead of Thread.currentThread().getContextClassloader(). – Stafford