Reading a UTF-8 text file (in Hebrew) shows gibrish in RStudio's console and fine in RGUI

Question

I am trying to understand if this is a bug in RStudio or am I missing something.

I am reading a csv file into R. When printing it into the console in RStudio I get gibrish (unless I look at a specific vector). While in Rgui this is fine.

The code I will run is this:

Sys.setlocale("LC_ALL", "Hebrew")
x <- read.csv("https://raw.githubusercontent.com/talgalili/temp2/gh-pages/Hebrew_UTF8.txt", encoding="UTF-8")  
x # shows gibrish
x[,2]
colnames(x)

Here is the output from RStudio (gibrish)

> x <- read.csv("https://raw.githubusercontent.com/talgalili/temp2/gh-pages/Hebrew_UTF8.txt", encoding="UTF-8")
> x
   âéì..áùðéí. îéâãø
1         23.0   æëø
2         24.0  ð÷áä
3         23.0  ð÷áä
4         24.0  ð÷áä
5         25.0   æëø
6         18.0   æëø
7         26.0   æëø
8         21.5  ð÷áä
9         24.0   æëø
10        26.0   æëø
11        24.0   æëø
12        19.0  ð÷áä
13        19.0  ð÷áä
14        24.5   æëø
15        21.0  ð÷áä
> x[,2]
 [1] זכר  נקבה נקבה נקבה זכר  זכר  זכר  נקבה זכר  זכר  זכר  נקבה נקבה זכר  נקבה
Levels: זכר נקבה
> colnames(x)
[1] "âéì..áùðéí." "îéâãø"      
> 

And here it is in Rgui (here it is fine):

>     x <- read.csv("https://raw.githubusercontent.com/talgalili/temp2/gh-pages/Hebrew_UTF8.txt", encoding="UTF-8")  
>     x # shows gibrish
   גיל..בשנים. מיגדר
1         23.0   זכר
2         24.0  נקבה
3         23.0  נקבה
4         24.0  נקבה
5         25.0   זכר
6         18.0   זכר
7         26.0   זכר
8         21.5  נקבה
9         24.0   זכר
10        26.0   זכר
11        24.0   זכר
12        19.0  נקבה
13        19.0  נקבה
14        24.5   זכר
15        21.0  נקבה
>     x[,2]
 [1] זכר  נקבה נקבה נקבה זכר  זכר  זכר  נקבה זכר  זכר  זכר  נקבה נקבה זכר  נקבה
Levels: זכר נקבה
>     colnames(x)
[1] "גיל..בשנים." "מיגדר"      
> 

In both sessions, my sessionInfo() is:

> sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=Hebrew_Israel.1255  LC_CTYPE=Hebrew_Israel.1255   
[3] LC_MONETARY=Hebrew_Israel.1255 LC_NUMERIC=C                  
[5] LC_TIME=Hebrew_Israel.1255    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] installr_0.17.0

I'm using the latest RStudio version 0.99.892

Thanks.

Strikebreaker

Were you able to solve this problem? I am having exact same problem with Japanese. – Purvis

Answer 1

This is a bug in R-studio and not the only one. I've seen you have received a general answer about problems R-studio currently having with non-English locale support on windows. As far as I know it is not the first time / version having similar problems. You may also meet some new problems that I think related to win 10 . Note that since I'm having the second type of problems as well, I am using English locale to print Hebrew.

So I have tried some debugging on your problem there and came with some work-around, and some new insights (I think..) on where is the problem. I think it can be further debugged to write a complete function that will fix it, but due to time (and hour) restrictions I've decide to stop here.

I've created this data:

x <- data.frame("x"= c("דור","dor"))

As mentioned already, using Hebrew locale I as well get gibrish

Sys.setlocale("LC_ALL", "Hebrew")
[1] "LC_COLLATE=Hebrew_Israel.1255;LC_CTYPE=Hebrew_Israel.1255;LC_MONETARY=Hebrew_Israel.1255;LC_NUMERIC=C;LC_TIME=Hebrew_Israel.1255"

"דור"
[1] "ãåø"

x
   x
1 ãåø
2 dor

Using English locale, I've get this output.

Sys.setlocale("LC_ALL", "English")
[1] "LC_COLLATE=English_United States.1252;LC_CTYPE=English_United States.1252;LC_MONETARY=English_United States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252"

 "דור"
[1] "דור"

x
                         x
1 <U+05D3><U+05D5><U+05E8>
2                      dor

Note that non data.frame output prints fine. It also occurs with data.table class, and prints fine with list and matrix.

Checking both print.data.frame and print.table methods reveals the main suspect: format.

Further investigation confirm these suspicions:

as.matrix(x)
     x    
[1,] "דור"
[2,] "dor"

format(as.matrix(x))
     x                         
[1,] "<U+05D3><U+05D5><U+05E8>"
[2,] "dor                     "

As such in your case I suggest following this workflow:

Sys.setlocale("LC_ALL", "Hebrew")
x <- read.csv("https://raw.githubusercontent.com/talgalili/temp2/gh-pages/Hebrew_UTF8.txt", encoding="UTF-8")  
as.matrix(x) 
      âéì..áùðéí. îéâãø 
 [1,] "23.0"      "זכר" 
 [2,] "24.0"      "נקבה"
 [3,] "23.0"      "נקבה"
 [4,] "24.0"      "נקבה"
 [5,] "25.0"      "זכר" 
 [6,] "18.0"      "זכר" 
 [7,] "26.0"      "זכר" 
 [8,] "21.5"      "נקבה"
 [9,] "24.0"      "זכר" 
[10,] "26.0"      "זכר" 
[11,] "24.0"      "זכר" 
[12,] "19.0"      "נקבה"
[13,] "19.0"      "נקבה"
[14,] "24.5"      "זכר" 
[15,] "21.0"      "נקבה"

Both locales: Hebrew and English worked on my machine, but col.names didn't work for neither.

To conclude, this is far from being a complete solution, but just a small and partial work-around the printing (or shall recall the formatting) problem. It also shed some more light on this Hebrew / non-English issue in R-studio, on which some better solutions may be written. One example for a solution for a similar problem of writing Hebrew in windows can be seen on this SO thread.

Pseudocarp

Thanks dof. Did you write this to the RStudio people? – Strikebreaker

No... it was way too late. – Pseudocarp