zsh_history stores non-ascii character with strange encoding

Question

When I open my .zsh_history file, I found that some non-ascii characters in my history have strange encoding or something like that.

Here is a minimal example:

First, the .zshrc is

# zshrc
HISTSIZE=10000
SAVEHIST=10000
HISTFILE=~/.zsh_history

I typed some commands with chinese character, exec zsh to save the history, and read .zsh_history file.

echo 字串測試 一二三
exec zsh
tail -3 .zsh_history

The result is replaced with some random weird characters.

echo 孃串測試 一亃七
exec zsh

the image

Also, when I use vim or any text editor to open this file, it opens this file use latin1 encoding and displays the following:

echo 孷串測試 ä¸äº¬ä¸©
exec zsh

the image

I cannot find an option to fix this, and I'm guessing it's an encoding problem. How can I force zsh to save the history in utf-8 encoding or there is another way to fix this?

Isotropic

Updated. The images are still there just in case that someone might not have proper font/encoding setting to see those characters. – Isotropic

I'm also experiencing the same issue (in Hebrew). Any luck figuring it out? – Counterfeit

Also suddenly Hebrew characters are all replaced with number codes. saying 'file .zsh_history' tells me .zsh_history: Non-ISO extended-ASCII text. Now I wonder how to set history to UTF-8? – Jebel

Answer 1

I found this https://www.zsh.org/mla/users/2011/msg00154.html and it seems that this behavior is intended.

The reply said that

This isn't a bug, the history file is saved in metafied format. If you want to print it outside zsh you can use this simple program.

... (a simple C program)

I guess the better way to browse the history in zsh shell is fc -l -i 1 | vim -. (maybe man zshbuiltins for more options)

Isotropic