How can I run dos2unix on an entire directory? [closed]

Question

I have to convert an entire directory using dos2unix. I am not able to figure out how to do this.

Grum

This is the subject of a meta question. – Going

See also: Unix & Linux: How to apply dos2unix recursively to all the contents of a folder? – Jalbert

Answer 1

find . -type f -print0 | xargs -0 dos2unix

Will recursively find all files inside current directory and call for these files dos2unix command

Moustache

i gave like this find . -type f -exec dos2unix {} /home/venuk/Desktop/NEO_Src and it gave the error find: missing argument to `-exec' – Grum

type f = Just 'F'files and not directories for example! – Kilmarnock

@vivekgaur, The folder path doesn't go at the end: it goes immediately after the find. In this example, it's the dot (i.e., the current directory). – Hypesthesia

Note the backslash that escapes the semi-colon ensures the dos2unix commands are separated by semi-colons so that they don't end up mashed together. If you want to run another command afterwards you'll need another semi-colon, so \;; echo Hello – Cai

Be careful with this command though, it doesn't check whether the file needs the conversion, so even binary files will be "converted", corrupting them. – Mediatorial

Would this go into subdirectories as well? – Idoux

@Pachonk Just tested that and it does. – Lowelllowenstein

@Mediatorial True, the find invocation does not do this check (although that would be simple enough to add), but modern dos2unix correctly skips binary files. – Phototherapy

@therealjumbo - If you want to use find with -print0 and xargs, then you should probably add your own answer. -print0 is not Posix, so the change could have a negative effect on some users. Considering NickC already provided that variant of the answer, there's little reason to change CyberDem0n's answer. – Evan

My only problem with this command is that it didn't do the dot files in the directory. Using this on a Chef cookbook, there are a few dot files such as .kitchen.yml, or .gitignore and those were not picked up. – Gyve

and to narrow it on a specific extension: find . -type f -name '*.py' -print0 | xargs -0 dos2unix – Reactivate

you need to pass the -i switch to dos2unix, or it will just print the converted files to stdout. – Labarum

Warning: If you run this command in a directory containing a .git index, your git index will be corrupted. – Gurgle

To avoid .git index corruption and preserve UTF-8 BOM, I've ended up doing find . -type f -print0 | xargs -0 dos2unix -ic0 | xargs -0 dos2unix -b. This command touches only those files (thanks to -ic0 filtering option), that contain windows line breaks, all other files are skipped. -b option is also helpful if your repository contains files with UTF-8 BOM, it preserves the BOM. – Pinson

I'd recommend git ls-files instead of find if you do anything with git repositories. – Insignificancy

Well, so far I have not seen anyone talk about the wildcard "*" I faced exactly this problem just moments ago and i solved it like this cd into the target directory and: dos2unix * All the files in the current directory will be converted to unix format. – Drawplate

@MichaelAmadi, one limitation is that the wildcard * will not access hidden files or folders which begin with a dot (.). – Jalbert

Great answer. I extended it with a multi-process version of this and other answers, plus some new knowledge, showing how to do this on a single dir, on a single dir within a git repo, and on multiple dirs within a git repo, here: How to recursively run dos2unix (or any other command) on your desired directory or path using multiple processes. – Jalbert

Answer 2

If it's a large directory you may want to consider running with multiple processors:

find . -type f -print0 | xargs -0 -n 1 -P 4 dos2unix 

This will pass 1 file at a time, and use 4 processors.

Roselinerosella

This method has the advantage, that it continues, even though dos2unix encounters any problems! Something like a "--force" method. Thank you for that! – Sufflate

Starting a new dos2unix process for each individual file will introduce massively unnecessary overhead. I'd bump that n up by an order of magnitude or two (depending on how many files we're talking about here) – Hypoxia

Great answer! @JonoCoetzee, here I use -n 50. – Jalbert

Answer 3

A common use case appears to be to standardize line endings for all files committed to a Git repository:

git ls-files -z | xargs -0 dos2unix

Keep in mind that certain files (e.g. *.sln, *.bat) are only used on Windows operating systems and should keep the CRLF ending:

git ls-files -z '*.sln' '*.bat' | xargs -0 unix2dos

If necessary, use .gitattributes

Aylesbury

I upvoted your answer, but one thing needs to be addressed: it fails on files with spaces in the path, since xargs defaults to having all whitespace be a delimiter. The xargs man page shows a --delimiter option, but suggests using the --null option when input might contain spaces. I have not tested this, but adding --null to yours produces the following: git ls-files | xargs --null dos2unix. – Extern

Thanks @Extern - Yes, you are correct, the command will fail if a filename contains spaces. Your comments helped me to find the answer (it also requires -z for ls-files). I've updated my answer. Thanks again! – Aylesbury

Answer 4

As I happened to be poorly satisfied by dos2unix, I rolled out my own simple utility. Apart of a few advantages in speed and predictability, the syntax is also a bit simpler :

endlines unix *

And if you want it to go down into subdirectories (skipping hidden dirs and non-text files) :

endlines unix -r .

endlines is available here https://github.com/mdolidon/endlines

Anticipate

This is perfect! The closest one-liner I could get to this is here: unix.stackexchange.com/a/365679/112190 – Gray

How does it improve on dos2unix? Genuinely curious. – Mustachio

1/ Mainly, there are tons of different dos2unix, with varying capabilities (some read UTF32 for example, while some don't ; endlines does not). There's only one endlines, which capabilities are well known. 2/ liberal on input, not all dos2unix are. 3/ efficient file tree exploration, designed to be fast and practical on tens of thousands of files. 4/ runs out of the box on OSX - which is less important now that Brew package exists. – Anticipate

Answer 5

It's probably best to skip hidden files and folders, such as .git. So instead of using find, if your bash version is recent enough or if you're using zsh, just do:

dos2unix **

Note that for Bash, this will require:

shopt -s globstar

....but this is a useful enough feature that you should honestly just put it in your .bashrc anyway.

If you don't want to skip hidden files and folders, but you still don't want to mess with find (and I wouldn't blame you), you can provide a second recursive-glob argument to match only hidden entries:

dos2unix ** **/.*

Note that in both cases, the glob will expand to include directories, so you will see the following warning (potentially many times over): Skipping <dir>, not a regular file.

Phototherapy

This didn't work for me. Does the globstar syntax work for dos2unix? I've used globstar elsewhere with success, but couldn't get this to work. I'm using Bash 4.3.11(1) – Beriosova

@NSduToit *NIX style shells don't allow executables to alter the behavior of argument-expansion, because (unlike in Windows) argument-expansion is performed before the executable ever receives the arguments. So the only thing I can think of is that you have some kind of dos2unix alias that's affecting how arguments are expanded. What is the output of type dos2unix on your system? – Phototherapy

Apologies - seems it works...sort of... Seems the problem I'm encountering has to do with dos2unix and not the actual globstar. It seems if I use dos2unix like that it just blindly ignores files that are hidden (start with '.', eg '.vimrc')......? But the globstar itself seems to work - the output of ls -a ** is as one would expect... – Beriosova

Script to test --- you'll see the dos2unix command never operates on the .vimrc file. Not the behavior I expected - any insights appreciated :) – Beriosova

@NSduToit There's some confusion here. My answer explicitly states that the point of using ** instead of find is to "skip hidden files and folders, such as .git". dos2unix never sees the hidden files, because ** does not expand to show them. If you want to automatically run dos2unix on hidden files and folders, you can use find or dos2unix ** **/.* The **/.* will expand only the hidden files and folders, including . (the root dir), .. (the parent dir), and any other hidden entries in the current folder. – Phototherapy

type dos2unix returns dos2unix is hashed (/usr/bin/dos2unix) – Beriosova

(Note that dos2unix simply prints Skipping <dir>, not a regular file. when run on a directory, so running on .. and . is safe.) Additionally, combining ls with a glob is not a good way to check how the glob is expanded; use echo instead: echo ** will print the arguments that dos2unix receives from dos2unix **. – Phototherapy

Thank you! Makes sense! Okay, I had a totally wrong idea on what ** implied, and using echo makes it clear. – Beriosova

Let us continue this discussion in chat. – Phototherapy

@feeela The entire point of the globstar ** is to make globs themselves exercise recursive descent. – Phototherapy

Answer 6

For any Solaris users (am using 5.10, may apply to newer versions too, as well as other unix systems):

dos2unix doesn't default to overwriting the file, it will just print the updated version to stdout, so you will have to specify the source and target, i.e. the same name twice:

find . -type f -exec dos2unix {} {} \;

Moffatt

Answer 7

I've googled this like a million times, so my solution is to just put this bash function in your environment.

.bashrc or .profile or whatever

dos2unixd() {
  find $1 -type f -print0 | xargs -0 dos2unix
}

Usage

$ dos2unixd ./somepath

This way you still have the original command dos2unix and it's easy to remember this one dos2unixd.

Bodycheck

Answer 8

I think the simplest way is:

dos2unix $(find . -type f)

Hoofbound

Answer 9

I have had the same problem and thanks to the posts here I have solved it. I knew that I have around a hundred files and I needed to run it for *.js files only. find . -type f -name '*.js' -print0 | xargs -0 dos2unix

Thank you all for your help.

Went

Answer 10

for FILE in /var/www/html/files/*
do
 /usr/bin/dos2unix FILE
done

Antitragus

Welcome to Stack Overflow. Although your solution is a valid solution, it would be great if you could add some explanation to it. You might also consider to reference other answers to justify your answer. Please have a look at How to Answer for more information. – Donation

Answer 11

If there is no sub-directory, you can also take

ls | xargs -I {} dos2unix "{}"

Serai

If there are no subdirectories, dos2unix * is simpler and will actually be more robust than this. (It's generally not recommended to pipe the output of ls, because it's a formatting tool and * is more reliable for programmatic usage.) – Phototherapy

the reason piping ls is not good – Ferdy