Delete non-UTF characters from a string in Ruby?

Question

How do I delete non-UTF8 characters from a ruby string? I have a string that has for example "xC2" in it. I want to remove that char from the string so that it becomes a valid UTF8.

This:

text = x = "foo\xC2bar"
text.gsub!(/\xC2/, '')

returns an error:

incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)

I was looking at text.unpack('U*') and string.pack as well, but did not get anywhere.

Twoway

You might find #11375842 useful – Continuant

Answer 1

You can use encode for that. text.encode('UTF-8', :invalid => :replace, :undef => :replace)

Or text.scrub

For more info look into Ruby-Docs, replaces it by default with a question mark box.

Brigittebriley

Every time you see that you got 10 points from this answer you must know how much head banging against a desk you've just saved someone. – Drown

Yup. Here, 10 more points for you. – Unsociable

Answer 2

You could do it like this

# encoding: utf-8

class String
  def validate_encoding
    chars.select(&:valid_encoding?).join 
  end
end

puts "testing\xC2 a non UTF-8 string".validate_encoding
#=>testing a non UTF-8 string

Repression

.select(&:valid_encoding?) instead of .collect{} is a lot shorter. – Levania

you'r right ephemient and it stays comprehensible, thanks, i adapt my answer – Repression

This actually works, unlike the most voted answer. – Staley

Answer 3

You text have ASCII-8BIT encoding, instead you should use this:

String.delete!("^\u{0000}-\u{007F}"); 

It will serve the same purpose.

Pierette

I ended up using mystring.delete() since delete! returns nil if the string was not modified, see apidock.com/ruby/v2_5_5/String/delete%21 – Exogenous

Removes characters with diacriticals and stuff like that, but works :) – Historied

Answer 4

You can use /n, as in

text.gsub!(/\xC2/n, '')

to force the Regexp to operate on bytes.

Are you sure this is what you want, though? Any Unicode character in the range [U+80, U+BF] will have a \xC2 in its UTF-8 encoded form.

Levania

This gives me incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) – Twoway

Answer 5

Try Iconv

1.9.3p194 :001 > require 'iconv'
# => true 
1.9.3p194 :002 > string = "testing\xC2 a non UTF-8 string"
# => "testing\xC2 a non UTF-8 string" 
1.9.3p194 :003 > ic = Iconv.new('UTF-8//IGNORE', 'UTF-8')
# => #<Iconv:0x000000026c9290> 
1.9.3p194 :004 > ic.iconv string
# => "testing a non UTF-8 string" 

Viveca

one note: Iconv is (will be) deprecated from Rails 3.2 in favour of String#encode – Toothpaste

Appears Iconv was deprecated in ruby 1.9 FWIW...guess it's just a gem now for 2.x – Historied

Answer 6

The best solution to this problem that I've found is this answer to the same question: https://mcmap.net/q/392251/-is-there-a-way-in-ruby-1-9-to-remove-invalid-byte-sequences-from-strings.

In short: "€foo\xA0".chars.select(&:valid_encoding?).join

Jawbone

Answer 7

Use String encode method with param 'replace' to return a string without invalid chars

'MyString'.encode('UTF-8', :invalid => :replace, :undef => :replace, :replace => '')

or using bang to change the string

'MyString'.encode!('UTF-8', :invalid => :replace, :undef => :replace, :replace => '')

Ruby Doc

Cabaret

Answer 8

data = '' if not (data.force_encoding("UTF-8").valid_encoding?)

Tug

This doesn't actually repair a string? – Historied