How to remove non-printable/invisible characters in ruby?

Question

Sometimes I have evil non-printable characters in the middle of a string. These strings are user input, so I must make my program receive it well instead of try to change the source of the problem.

For example, they can have zero width no-break space in the middle of the string. For example, while parsing a .po file, one problematic part was the string "he is a man of god" in the middle of the file. While it everything seems correct, inspecting it with irb shows:

 "he is a man of god".codepoints
 => [104, 101, 32, 105, 115, 32, 97, 32, 65279, 109, 97, 110, 32, 111, 102, 32, 103, 111, 100] 

I believe that I know what a BOM is, and I even handle it nicely. However sometimes I have such characters on the middle of the file, so it is not a BOM.

My current approach is to remove all characters that I found evil in a really smelly fashion:

text = (text.codepoints - CODEPOINTS_BlACKLIST).pack("U*")

The most close I got was following this post which leaded me to :print: option on regexps. However it was no good for me:

"m".scan(/[[:print:]]/).join.codepoints
 => [65279, 109] 

so the question is: How can I remove all non-printable characters from a string in ruby?

Obliquity

It'd help a lot if you showed more source and sample strings with the characters you're trying to handle. The current sample doesn't help much when trying to determine the codeset or what other values you're encountering. – Elseelset

@theTinMan Thanks, I edited the question with a bit of more details. The charset is UTF-8, I believe, but I don't always have the info, I got many files without BOM. This one I suppose it is at least partially unicode by looking at the chinese translation. – Obliquity

Ruby has a method on String called dump which produces a new string with non-printing characters removed and special characters escaped. Docs for String#dump Ruby 2.3.0 but I can confirm it is in the docs as early as 1.8.7. – Dysphemism

Answer 1

try this:

>>"aaa\f\d\x00abcd".gsub(/[^[:print:]]/,'.')
=>"aaa.d.abcd"

Phylis

This unfortunately removes newlines, too. – Racing

That would still not remove some characters, like "\u200B" – Fungus

Answer 2

Codepoint 65279 is a zero-width no-break space. It is commonly used as a byte-order mark (BOM).

You can remove it from a string with:

my_new_string = my_old_string.gsub!("\xEF\xBB\xBF".force_encoding("UTF-8"), '')

A fast way to check if you have any invisible characters is to check the length of the string, if it's higher than what you can see in IRB, you do.

Pity

Answer 3

Ruby can help you convert from one multi-byte character set to another. Check into the search results, plus read up on Ruby String's encode method.

Also, Ruby's Iconv is your friend.

Finally, James Grey wrote a series of articles which cover this in good detail.

One of the things you can do using those tools is to tell them to transcode to a visually similar character, or ignore them completely.

Dealing with alternate character sets is one of the most... irritating things I've ever had to do, because files can contain anything, but be marked as text. You might not expect it and then your code dies or starts throwing errors, because people are so ingenious when coming up with ways to insert alternate characters into content.

Elseelset

Gave up.. I think that there isn't a better way to handle malformed files. However I'm accepting your answer because it is a good guideline for people that ends up here with well formed files. – Obliquity

None of these links are functional now :( – Ahoy

@Surya, Thanks, yes a couple were broken, but not all. The SO way is to help maintain the site. You're empowered to help by submitting edits to fix problems such as broken links. See "How do suggested edits work". – Elseelset

Thank you for bring this feature to my attention. – Ahoy