How to find and remove the invisible characters in text file using emacs

C

3

14

I have a .txt file named COPYING which is edited on windows.
It contains Windows-style line breaks :

$ file COPYING 
COPYING: ASCII English text, with CRLF line terminators

I tried to convert it to Unix style using dos2unix. Below is the output :

$ dos2unix COPYING 
dos2unix: Skipping binary file COPYING

I was surprised to find that the dos2unix program reports it as a binary file. Then using some other editor (not Emacs) I found that the file contains a control character. I am interested in finding all the invisible characters in the file using Emacs.

By googling, I have found the following solution which uses tr :

tr -cd '\11\12\40-\176' < file_name

How can I do the same in an Emacs way? I tried the Hexl mode. The Hexl mode shows text and their corresponding ASCII values in a single buffer which is great. How do I find the characters which have ASCII values other than 11-12, 40-176 (i.e tab, space, and visible characters)? I tried to create a regular expression for that search, but it is quite complicated.

Churr answered 7/10, 2011 at 12:54 Comment(0)

S

4

Emacs won't hide any character by default. Press Ctrl+Meta+%, or Esc then Ctrl+% if the former is too hard on your fingers, or M-x replace-regexp RET if you prefer. Then, for the regular expression, enter

[^@-^H^K-^_^?]

However, where I wrote ^H, type Ctrl+Q then Ctrl+H, to enter a “control-H” character literally, and similarly for the others. You can press Ctrl+Q then Ctrl+Space for ^@, and usually Ctrl+Q then Backspace for ^?. Replace all occurrences of this regular expression by the empty string.

Since you have the file open in Emacs, you can change its line endings while you're at it. Press C-x RET f (Ctrl+X Return F) and enter us-ascii-unix as the new desired encoding for the file.

Sou answered 7/10, 2011 at 22:27 Comment(4)

Thanks for the answer. I wanted to understand what the regular expression does. I typed the command "describe-character-set" then selected the "ascii" to see the character set. Could you please explain why did you miss the C-j and where the C-? is documented. – Churr 8/10, 2011 at 15:34

@Churr C-i is tab, C-j is newline. – Superordinate 8/10, 2011 at 15:39

This link www2.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html says C-m is for RET key. I am not able to find the doc where C-? is documented. Could you please point it out. – Churr 8/10, 2011 at 15:59

@Churr I can't find this documented explicitly in the manual. There's brief mention of C-? in If <DEL> Fails to Delete, and character 127 is often called the DEL character and produced by the Backspace or Delete key. – Superordinate 9/10, 2011 at 17:40

I

20

To see invisible characters, you can try whitespace-mode. Spaces and tabs will be displayed with a symbol in a different face. If the coding system is automatically being detected as dos (showing (DOS) on the status bar), carriage returns at the end of a line will be hidden as well. Run revert-buffer-with-coding-system to switch it to Unix or binary (e.g. C-x RET r unix) and they'll always show up as ^M. The binary coding system will display any non-ASCII characters as control characters as well.

Irredentist answered 7/10, 2011 at 17:4 Comment(2)

All the characters does not show up as ^M. I have a very large file i can see some of the characters as ^L. I dont know how many such characters are out there in the file. I want to search for those characters. – Churr 7/10, 2011 at 20:40

There are many, many invisible characters that are not shown by whitespace-mode. Amongst them: zero-width space, zero-width non-joiner, zero-width joiner, byte-order mark... See how fingerprinting uses them, and the dangers of neglecting them. You might want to reflect that into your answer. – Yamen 8/6, 2020 at 14:56

S

4

Emacs won't hide any character by default. Press Ctrl+Meta+%, or Esc then Ctrl+% if the former is too hard on your fingers, or M-x replace-regexp RET if you prefer. Then, for the regular expression, enter

[^@-^H^K-^_^?]

However, where I wrote ^H, type Ctrl+Q then Ctrl+H, to enter a “control-H” character literally, and similarly for the others. You can press Ctrl+Q then Ctrl+Space for ^@, and usually Ctrl+Q then Backspace for ^?. Replace all occurrences of this regular expression by the empty string.

Since you have the file open in Emacs, you can change its line endings while you're at it. Press C-x RET f (Ctrl+X Return F) and enter us-ascii-unix as the new desired encoding for the file.

Sou answered 7/10, 2011 at 22:27 Comment(4)

Thanks for the answer. I wanted to understand what the regular expression does. I typed the command "describe-character-set" then selected the "ascii" to see the character set. Could you please explain why did you miss the C-j and where the C-? is documented. – Churr 8/10, 2011 at 15:34

@Churr C-i is tab, C-j is newline. – Superordinate 8/10, 2011 at 15:39

This link www2.lib.uchicago.edu/keith/tcl-course/emacs-tutorial.html says C-m is for RET key. I am not able to find the doc where C-? is documented. Could you please point it out. – Churr 8/10, 2011 at 15:59

@Churr I can't find this documented explicitly in the manual. There's brief mention of C-? in If <DEL> Fails to Delete, and character 127 is often called the DEL character and produced by the Backspace or Delete key. – Superordinate 9/10, 2011 at 17:40

M

1

Check out M-x set-buffer-file-coding-system. From the documentation:

(set-buffer-file-coding-system CODING-SYSTEM &optional FORCE NOMODIFY)

Set the file coding-system of the current buffer to CODING-SYSTEM. This means that when you save the buffer, it will be converted according to CODING-SYSTEM. For a list of possible values of CODING-SYSTEM, use M-x list-coding-systems.

So, going from DOS to UNIX, M-x set-buffer-file-coding-system unix.

Meniscus answered 7/10, 2011 at 14:13 Comment(0)

Recommended topics

Hot tags