Remove non-ASCII non-printable characters from a String

Asked 13/6, 2012 at 18:14 Answered 27/4, 2023 at 5:0

I get user input including non-ASCII characters and non-printable characters, such as

\xc2d
\xa0
\xe7
\xc3\ufffdd
\xc3\ufffdd
\xc2\xa0
\xc3\xa7
\xa0\xa0

for example:

email : [email protected]\xa0\xa0
street : 123 Main St.\xc2\xa0

desired output:

  email : [email protected]
  street : 123 Main St.

What is the best way to removing them using Java?
I tried the following, but doesn't seem to work

public static void main(String args[]) throws UnsupportedEncodingException {
        String s = "abc@gmail\\xe9.com";
        String email = "[email protected]\\xa0\\xa0";

        System.out.println(s.replaceAll("\\P{Print}", ""));
        System.out.println(email.replaceAll("\\P{Print}", ""));
    }

Output

abc@gmail\xe9.com
[email protected]\xa0\xa0

Counterfoil answered 13/6, 2012 at 18:14 Comment(3)

why do you want to remove them? – Variation 13/6, 2012 at 18:17

@jtahlborn, Mongo fails to serialize these values – Counterfoil 13/6, 2012 at 18:26

@Counterfoil [citation needed] \xc2d is a valid Unicode character. If MongoDB uses UTF-8 is should be able to serialize them. Perhaps you have an XY Problem here? How are you serializing your text? – Harleigh 26/11, 2018 at 11:11

Your requirements are not clear. All characters in a Java String are Unicode characters, so if you remove them, you'll be left with an empty string. I assume what you mean is that you want to remove any non-ASCII, non-printable characters.

String clean = str.replaceAll("\\P{Print}", "");

Here, \p{Print} represents a POSIX character class for printable ASCII characters, while \P{Print} is the complement of that class. With this expression, all characters that are not printable ASCII are replaced with the empty string. (The extra backslash is because \ starts an escape sequence in string literals.)

Apparently, all the input characters are actually ASCII characters that represent a printable encoding of non-printable or non-ASCII characters. Mongo shouldn't have any trouble with these strings, because they contain only plain printable ASCII characters.

This all sounds a little fishy to me. What I believe is happening is that the data really do contain non-printable and non-ASCII characters, and another component (like a logging framework) is replacing these with a printable representation. In your simple tests, you are failing to translate the printable representation back to the original string, so you mistakenly believe the first regular expression is not working.

That's my guess, but if I've misread the situation and you really do need to strip out literal \xHH escapes, you can do it with the following regular expression.

String clean = str.replaceAll("\\\\x\\p{XDigit}{2}", "");

The API documentation for the Pattern class does a good job of listing all of the syntax supported by Java's regex library. For more elaboration on what all of the syntax means, I have found the Regular-Expressions.info site very helpful.

Pneumoconiosis answered 13/6, 2012 at 18:39 Comment(12)

this doesn't work. may be I am doing something incorrect, but not working – Counterfoil 18/6, 2012 at 18:15

@Counterfoil Can you provide an SSCCE that shows what is not working? – Pneumoconiosis 18/6, 2012 at 18:19

public static void main(String args[]) throws UnsupportedEncodingException { String s = "abc@gmail\\xe9.com"; String email = "[email protected]\\xa0\\xa0"; System.out.println(s.replaceAll("\\P{Print}", "")); System.out.println(email.replaceAll("\\P{Print}", "")); } out put - abc@gmail\xe9.com [email protected]\xa0\xa0 – Counterfoil 18/6, 2012 at 18:21

@Counterfoil \\x doesn't mean anything special in Java source code. \\ in a String or char literal is an escape sequence that is replaced with \. If you want a Unicode escape, use \uXXXX, where XXXX is the Unicode point, in hexadecimal. – Pneumoconiosis 18/6, 2012 at 18:25

@Counterfoil E.g. String s = "abc@gmail\u00e9.com"; – Pneumoconiosis 18/6, 2012 at 18:27

ah I see, but the input I get is what I shared with you, does it mean it is not possible to strip it away? – Counterfoil 18/6, 2012 at 18:28

let us continue this discussion in chat – Counterfoil 18/6, 2012 at 18:29

No, you just need a different regular expression; your input is, apparently, all ASCII. Please see the update to my answer. – Pneumoconiosis 18/6, 2012 at 18:42

that seems to work, where I can learn more about creating such patterns like you created? please advice and thank you very much for your help, I appreciate it – Counterfoil 18/6, 2012 at 18:46

@Counterfoil I added a couple of links to great learning resources to the end of my answer. – Pneumoconiosis 18/6, 2012 at 18:52

how to remove this one � – Shelving 14/10, 2019 at 0:3

"All characters in a Java String are Unicode characters, so if you remove them, you'll be left with an empty string." xD – Disfeature 16/3, 2021 at 9:37

With Google Guava's CharMatcher, you can remove any non-printable characters and then retain all ASCII characters (dropping any accents) like this:

String printable = CharMatcher.INVISIBLE.removeFrom(input);
String clean = CharMatcher.ASCII.retainFrom(printable);

Not sure if that's what you really want, but it removes anything expressed as escape sequences in your question's sample data.

Herbart answered 13/6, 2012 at 18:47 Comment(1)

note, INVISIBLE removed whitespace which I find odd since it is indeed "printable" – Paradies 26/8, 2014 at 13:47

I know it's maybe late but for future reference:

String clean = str.replaceAll("\\P{Print}", "");

Removes all non printable characters, but that includes \n (line feed), \t(tab) and \r(carriage return), and sometimes you want to keep those characters.

For that problem use inverted logic:

String clean = str.replaceAll("[^\\n\\r\\t\\p{Print}]", "");

Sherlocke answered 15/7, 2015 at 7:33 Comment(3)

Upvoted for it's particular usefulness in mongo-land, to keep the shell from spewing ridiculous amounts of encoded non-ascii stuff (mongo really really prefers utf-8 if you want things to be easy) – Pomology 1/2, 2016 at 2:15

Got error: illegal escape character String clean = str.replaceAll("[^\n\r\t\p{Print}]", ""); . \p should be \P – Ovary 17/4, 2018 at 15:35

Really helped me a lot Thanks @Ivan – Margrettmarguerie 12/6, 2018 at 5:1

You can try this code:

public String cleanInvalidCharacters(String in) {
    StringBuilder out = new StringBuilder();
    char current;
    if (in == null || ("".equals(in))) {
        return "";
    }
    for (int i = 0; i < in.length(); i++) {
        current = in.charAt(i);
        if ((current == 0x9)
                || (current == 0xA)
                || (current == 0xD)
                || ((current >= 0x20) && (current <= 0xD7FF))
                || ((current >= 0xE000) && (current <= 0xFFFD))
                || ((current >= 0x10000) && (current <= 0x10FFFF))) {
            out.append(current);
        }

    }
    return out.toString().replaceAll("\\s", " ");
}

It works for me to remove invalid characters from String.

Hu answered 13/6, 2012 at 18:17 Comment(1)

That's a lot of magic numbers. How about extracting these clauses (especially the ranges) into aptly named local variables? – Herbart 13/6, 2012 at 18:48

You can use java.text.normalizer

Unrealizable answered 13/6, 2012 at 18:17 Comment(0)

Input => "This \u7279text \u7279is what I need" Output => "This text is what I need"

If you are trying to remove Unicode characters from a string like above this code will work

Pattern unicodeCharsPattern = Pattern.compile("\\\\u(\\p{XDigit}{4})");
Matcher unicodeMatcher = unicodeChars.matcher(data);
String cleanData = null;
if (unicodeMatcher.find()) {
    cleanData = unicodeMatcher.replaceAll("");
}

Vullo answered 10/5, 2017 at 15:4 Comment(0)

This simple function worked better for me:

function remove_non_ascii(str) {
  
    if ((str===null) || (str===''))
         return false;
   else
     str = str.toString();
    
    return str.replace(/[^\x20-\x7E]/g, '');
}

Idiotism answered 27/4, 2023 at 5:0 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags