Thor & YAML outputting as binary?

Question

I'm using Thor and trying to output YAML to a file. In irb I get what I expect. Plain text in YAML format. But when part of a method in Thor, its output is different...

class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def set
    test = {"name" => "Xavier", "age" => 30}
    puts test
    # {"name"=>"Xavier", "age"=>30}
    puts test.to_yaml
    # !binary "bmFtZQ==": !binary |-
    #   WGF2aWVy
    # !binary "YWdl": 30
    File.open("data/config.yml", "w") {|f| f.write(test.to_yaml) }
  end
end

Any ideas?

Grafton

I just ran your example and it gave me perfectly fine output. I ran thor 0.14.6. – Chryselephantine

Thanks for taking the time to check. I'm at a loss of what to do at this point. I'm using Ruby 1.9.3p125 if that makes any difference what so ever. :) – Grafton

I installed 1.9.3 and ran it again, and indeed binary output. I noticed YAML got upgraded during install. It probably has something to do with that upgraded version. – Chryselephantine

Thanks Maran. I assume yaml is built into ruby, would it be easier to rvm install 1.9.2 rather than install 1.9.3 w/ the downgraded version of yaml? – Grafton

Yeah, as soon as I issue rvm 1.9.2 everything works again, yaml wise. If you don't need 1.9.3 I would switch back to 1.9.2 – Chryselephantine

I tried it using Ruby 1.9.3p0 and Thor 0.14.6 and it generates perfect yaml for me. Could it be the difference between p0 and p125? – Holy

Adding #encoding: UTF-8 to the top of the file seems to fix it. I don't know what's going on though - it looks like thor is changing the default encoding to ASCII-8BIT and then a change to yaml in Ruby 1.9.3 causes the output to be binary. – Septime

I think it's due to how thor reads in .thor files; it uses File.binread which sets the encoding to ASCII-8BIT. – Septime

Answer 1

All Ruby 1.9 strings have an encoding attached to them.

YAML encodes some non-UTF8 strings as binary, even when they look innocent, without any high-bit characters. You might think that your code is always using UTF8, but builtins can return non-UTF8 strings (ex File path routines).

To avoid binary encoding, make sure all your strings encodings are UTF-8 before calling to_yaml. Change the encoding with force_encoding("UTF-8") method.

For example, this is how I encode my options hash into yaml:

options = {
    :port => 26000,
    :rackup => File.expand_path(File.join(File.dirname(__FILE__), "../sveg.rb"))
}
utf8_options = {}
options.each_pair { |k,v| utf8_options[k] = ((v.is_a? String) ? v.force_encoding("UTF-8") : v)}
puts utf8_options.to_yaml

Here is an example of yaml encoding simple strings as binary

>> x = "test"
=> "test"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.to_yaml
=> "--- test\n...\n"
>> x.force_encoding "ASCII-8BIT"
=> "test"
>> x.to_yaml
=> "--- !binary |-\n  dGVzdA==\n"

Somewhat

Answer 2

After version 1.9.3p125, ruby build-in YAML engine will treat all BINARY encoding differently than before. All you need to do is to set correct non-BINARY encoding before your String.to_yaml.

in Ruby 1.9, All String object have attached a Encoding object and as following blog ( by James Edward Gray II ) mentioned, ruby have build in three type of encoding when String is generated: http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings.

One of encoding may solve your problem => Source code Encoding

This is the encoding of your source code, and can be specify by adding magic encoding string at the first line or second line ( if you have a sha-bang string at the first line of your source code ) the magic encoding code could be one of following:

• # encoding: utf-8
• # coding: utf-8
• # -- encoding : utf-8 --

so in your case, if you use ruby 1.9.3p125 or later, this should be solved by adding one of magic encoding in the beginning of your code.

# encoding: utf-8
require 'thor'
class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def bar
    test = {"name" => "Xavier", "age" => 30}
    puts test
    #{"name"=>"Xavier", "age"=>30}
    puts test["name"].encoding.name
    #UTF-8
    puts test.to_yaml
    #---
    #name: Xavier
    #age: 30
    puts test.to_yaml.encoding.name
    #UTF-8
  end
end

Travax

Answer 3

I have been struggling with this using 1.9.3p545 on Windows - just with a simple hash containing strings - and no Thor.

The gem ZAML solves the problem quite simply:

require 'ZAML'
yaml = ZAML.dump(some_hash)
File.write(path_to_yaml_file, yaml)

Overstrain