Node.js mikeal/request module - Garbled non-utf8 website (Shift_JIS)

Question

I am trying to access a non utf-8 website using request module. Response is garbled for this request.

var request = require('request');
request('http://www.alc.co.jp/', function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the web page.
  }
});

Even after setting the encoding option to Shift_JIS I am seeing garbled Japanese text.

Domingadomingo

github.com/ashtuchkin/iconv-lite I think you can use this to decode SHIFT_JIS buffer into UTF-8 string. – Moskva

found another for Node here. I'm going to try and make it into a Meteor package. github.com/polygonplanet/encoding.js – Roundworm

that looks good too @Moskva – Roundworm

Answer 1

You need to do the conversion yourself. The example code below uses node-iconv.

    var Iconv = require('iconv').Iconv;
    var request = require('request');
    request({
      uri: 'http://www.jalan.net/',
      encoding: null,
    }, function (error, response, body) {
      if (!error && response.statusCode == 200) {
        body = new Iconv('shift_jis', 'utf-8').convert(body).toString();
        console.log(body); // Print the web page.
      }
    });

1. The encoding: null parameter asks request not to convert the Buffer (a byte array) into String yet.
2. We pass this buffer to Iconv for converting into another Buffer of UTF-8 encoding.
3. Now this Buffer is good for being converted into a String.

(BTW, http://www.alc.co.jp has switched to UTF-8, so I substituted with another site.)

Unionize