How can I get `find` to ignore .svn directories?

Question

I often use the find command to search through source code, delete files, whatever. Annoyingly, because Subversion stores duplicates of each file in its .svn/text-base/ directories my simple searches end up getting lots of duplicate results. For example, I want to recursively search for uint in multiple messages.h and messages.cpp files:

# find -name 'messages.*' -exec grep -Iw uint {} +
./messages.cpp:            Log::verbose << "Discarding out of date message: id " << uint(olderMessage.id)
./messages.cpp:    Log::verbose << "Added to send queue: " << *message << ": id " << uint(preparedMessage->id)
./messages.cpp:                Log::error << "Received message with invalid SHA-1 hash: id " << uint(incomingMessage.id)
./messages.cpp:            Log::verbose << "Received " << *message << ": id " << uint(incomingMessage.id)
./messages.cpp:            Log::verbose << "Sent message: id " << uint(preparedMessage->id)
./messages.cpp:        Log::verbose << "Discarding unsent message: id " << uint(preparedMessage->id)
./messages.cpp:        for (uint i = 0; i < 10 && !_stopThreads; ++i) {
./.svn/text-base/messages.cpp.svn-base:            Log::verbose << "Discarding out of date message: id " << uint(olderMessage.id)
./.svn/text-base/messages.cpp.svn-base:    Log::verbose << "Added to send queue: " << *message << ": id " << uint(preparedMessage->id)
./.svn/text-base/messages.cpp.svn-base:                Log::error << "Received message with invalid SHA-1 hash: id " << uint(incomingMessage.id)
./.svn/text-base/messages.cpp.svn-base:            Log::verbose << "Received " << *message << ": id " << uint(incomingMessage.id)
./.svn/text-base/messages.cpp.svn-base:            Log::verbose << "Sent message: id " << uint(preparedMessage->id)
./.svn/text-base/messages.cpp.svn-base:        Log::verbose << "Discarding unsent message: id " << uint(preparedMessage->id)
./.svn/text-base/messages.cpp.svn-base:        for (uint i = 0; i < 10 && !_stopThreads; ++i) {
./virus/messages.cpp:void VsMessageProcessor::_progress(const string &fileName, uint scanCount)
./virus/messages.cpp:ProgressMessage::ProgressMessage(const string &fileName, uint scanCount)
./virus/messages.h:    void _progress(const std::string &fileName, uint scanCount);
./virus/messages.h:    ProgressMessage(const std::string &fileName, uint scanCount);
./virus/messages.h:    uint        _scanCount;
./virus/.svn/text-base/messages.cpp.svn-base:void VsMessageProcessor::_progress(const string &fileName, uint scanCount)
./virus/.svn/text-base/messages.cpp.svn-base:ProgressMessage::ProgressMessage(const string &fileName, uint scanCount)
./virus/.svn/text-base/messages.h.svn-base:    void _progress(const std::string &fileName, uint scanCount);
./virus/.svn/text-base/messages.h.svn-base:    ProgressMessage(const std::string &fileName, uint scanCount);
./virus/.svn/text-base/messages.h.svn-base:    uint        _scanCount;

How can I tell find to ignore the .svn directories?

---

Update: If you upgrade your SVN client to version 1.7 this is no longer an issue.

A key feature of the changes introduced in Subversion 1.7 is the centralization of working copy metadata storage into a single location. Instead of a .svn directory in every directory in the working copy, Subversion 1.7 working copies have just one .svn directory—in the root of the working copy. This directory includes (among other things) an SQLite-backed database which contains all of the metadata Subversion needs for that working copy.

Aldwin

For performance, try to use find ... -print0 | xargs -0 egrep ... instead of find ... -exec grep ... (does not fork grep for each file, but for a bunch of files at a time). Using this form you can also prune .svn directories without using the -prune option of find, i.e. find ... -print0 | egrep -v '/\.svn' | xargs -0 egrep ... – Starknaked

@Vlad: As far as I know, using -exec with + doesn't fork grep for each file, while using it with ; does. Using -exec is actually more correct than using xargs. Please notice that commands like ls do something even if the argument list is empty, while commands like chmod give an error if there is insufficient arguments. To see what I mean, just try the following command in a directory that does not have any shell script: find /path/to/dir -name '*.sh' -print0 | xargs -0 chmod 755. Compare with this one: find /path/to/dir -name '*.sh' -exec chmod 755 '{}' '+'. – Icterus

@Vlad: Besides, grep-ing out .svn is not a good idea too. While find is specialized for handling file properties, grep does not. In your example, a file named '.svn.txt' will also be filtered by your egrep command. Although you can modify your regex to '^/\.svn$', it is still not a good practice to do so. The -prune predicate of find works perfectly for filtering a file (by filename, or creation timestamp, or whatever condition you supplied). It is just like even if you can kill a cockroach using a big sword doesn't mean it is the suggested way to do so :-). – Icterus

Switching to Git fixed this issue (among many others). In only makes a .git folder on the root of the working copy, not in every folder of it like SVN. Additionally, the .git folder does not contain plain files that would be confused with your actual files by same names. – Libelant

2Dan Moulding: svn 1.7 only creates a single top-level .svn directory – Paleobotany

See also serverfault.com/q/33308/29419 – Tertiary

Answer 1

For searching, can I suggest you look at ack ? It's a source-code aware find, and as such will automatically ignore many file types, including source code repository info such as the above.

Faraway

I like ack very much, but I have found it to be substantially slower than find -type f -name "*.[ch]" | xargs grep when dealing with a large codebase. – Lapidate

John, I'm the author of ack, and if you can give me details of the speed problems of ack vs. grep, I'd appreciate it. They've been completely comparable in all the cases I've found. Either let me know at github.com/petdance/ack/issues or email me at andy at petdance.com. Thansk. – Legaspi

Guys, thats a tip, but definitely not an answer to the question! :) – Blair

@dolzenko - I think that's a very good answer to the question and probably not worth a downvote – Faraway

Isn't ack billed as a better grep, not a source-aware find? Some examples of using it to replace find would make this a real answer. – Belshin

It’s an answer to the question he didn’t know he was asking. =) – Casebook

Example of ack as find: ack -g '\.clj$' (search for all Clojure source files). And it supports -print0 for piping to xargs; obviously a find replacement wouldn't be complete without piping to xargs. – Uribe

Answer 2

why not just

find . -not -iwholename '*.svn*'

The -not predicate negates everything that has .svn anywhere in the path.

So in your case it would be

find -not -iwholename '*.svn*' -name 'messages.*' -exec grep -Iw uint {} + \;

Engracia

Super big +1 for "-not" and "-iwholename". Ack is wonderful and I use it, but find/exec still has its uses. – Solarize

The only response that actually answered the original question. – Rutland

This is awesome. I am interested in finding only directories within an SVN WC so I can do some recursive cleanup. Adding -type d to the top line above gives me exactly what I need. – Microsome

I'm way out of my element and I'm sure I will get criticized for this comment, but apparently -not and -wholename are not POSIX-compliant. I used ! in place of -not and -path in place of -iwholename and got the same results. According to my man pages (Ubuntu 12.04) this syntax is POSIX-compliant. – Majorette

Note that this solution evaluates the whole directory tree, which is fine for avoiding occasional directories which are scattered all over a tree, but Kaleb Pederson's answer is more generally applicable if you want to avoid recursing into those directories in the first place. The difference can be huge, using a variant of this solution, a search I've just done took 16 minutes, using a variant of Kaleb Pederson's it took only 45 seconds! – Apocopate

@Engracia You said '*.svn*' at first but then '*.svn'. Which is right? Do both work? I think it should probably be '*.svn*'? – Shrub

@KeithM great catch actually. This answer has been sitting here for years and I don't think anyone had caught that until now. – Engracia

this is not ideal because find still traverses the directory. – Floe

Answer 3

As follows:

find . -path '*/.svn*' -prune -o -print

Or, alternatively based on a directory and not a path prefix:

find . -name .svn -a -type d -prune -o -print

Subdue

@Kaleb: Hi. I suggest find . -type d -name .svn -prune -o -print because it is a little bit faster. According to the POSIX standard, the expressions are evaluated one by one, in the order specified. If the first expression in -a is false, the second expression will not be evaluated (also called short-circuit and evaluation). – Icterus

@Kaleb: As comparing the file type (equivalent to testing whether a bit is set in an integer) is faster than comparing the filename (equivalent to a string comparison, which is O(n)), putting -type d before -name .svn is theoretically more efficient. However, it is usually insignificant except if you have a very very big directory tree. – Icterus

@Siu - Good point. Similarly, if you have any check that can be quickly performed (e.g. O(1)) and will avoid many additional checks, it's a good idea to place that check first. – Subdue

This doesn't work if I add any other conditions instead of the '-print'. For example, if I want to find all files (not directories), but exclude the '.git' directory and its contents, I try "find . -path './.git' -prune -o -type f", which does includes all files, and does exclude contents of the '.git' dir, but doesn't exclude the '.git' dir itself. – Harridan

@SiuChingPong-AsukaKenji- no, comparing only the filename is faster because -type requires a stat (2) call on every file. The filename, however, is part of the readdir (3) response. – Lightfingered

@JonathanHartley You're missing the -print as part of the last expression. Something like find . -name .git -prune -o \( -type f -name LICENSE -print \) works as expected. – Uniocular

If you want to ignore both .git and .svn and just list the other directories, find . -name .svn -prune -o -name .git -prune -o -type d -print. It might be a few milliseconds faster putting -type d before the two -name, but its not worth the extra typing. – Titration

This answer on another question explains why this is the best answer. It also shows how to ignore multiple directories. – Narrowminded

Answer 4

To ignore .svn, .git and other hidden directories (starting with a dot), try:

find . -type f -not -path '*/\.*'

---

However, if the purpose of using find is searching within the files, you may try to use these commands:

• git grep - specially designed command for searching patterns within the Git repository.
• ripgrep - which by default ignores hidden files and files specified in .gitignore.

Related: How do I find all files containing specific text on Linux?

Carpetbagger

Best answer imo. The others try to explain things that don't answer the simple question. – Vulgar

this will not work if attempting to exclude a dir for performance reasons – Floe

Answer 5

Here is what I would do in your case:

find . -path .svn -prune -o -name messages.* -exec grep -Iw uint {} +

---

Emacs' rgrep built-in command ignores .svn directory, and many more files you're probably not interested in when performing a find | grep. Here is what it uses by default:

find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS \
          -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr \
          -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) \
     -prune -o \
       \( -name .\#\* -o -name \*.o -o -name \*\~ -o -name \*.bin -o -name \*.lbin \
          -o -name \*.so -o -name \*.a -o -name \*.ln -o -name \*.blg \
          -o -name \*.bbl -o -name \*.elc -o -name \*.lof -o -name \*.glo \
          -o -name \*.idx -o -name \*.lot -o -name \*.fmt -o -name \*.tfm \
          -o -name \*.class -o -name \*.fas -o -name \*.lib -o -name \*.mem \
          -o -name \*.x86f -o -name \*.sparcf -o -name \*.fasl -o -name \*.ufsl \
          -o -name \*.fsl -o -name \*.dxl -o -name \*.pfsl -o -name \*.dfsl \
          -o -name \*.p64fsl -o -name \*.d64fsl -o -name \*.dx64fsl -o -name \*.lo \
          -o -name \*.la -o -name \*.gmo -o -name \*.mo -o -name \*.toc \
          -o -name \*.aux -o -name \*.cp -o -name \*.fn -o -name \*.ky \
          -o -name \*.pg -o -name \*.tp -o -name \*.vr -o -name \*.cps \
          -o -name \*.fns -o -name \*.kys -o -name \*.pgs -o -name \*.tps \
          -o -name \*.vrs -o -name \*.pyc -o -name \*.pyo \) \
     -prune -o \
     -type f \( -name pattern \) -print0 \
     | xargs -0 -e grep -i -nH -e regex

It ignores directories created by most version control systems, as well as generated files for many programming languages. You could create an alias that invokes this command and replace find and grep patterns for your specific problems.

Antimalarial

Answer 6

GNU find

find .  ! -regex ".*[/]\.svn[/]?.*"

Skyscape

I was loading the directory paths into an array for PHP to process. The other answers higher up (for whatever reason) didn't filter out the files in the find (despite the -type d) - this answer did. +1 – Kamerad

Answer 7

I use grep for this purpose. Put this in your ~/.bashrc

export GREP_OPTIONS="--binary-files=without-match --color=auto --devices=skip --exclude-dir=CVS --exclude-dir=.libs --exclude-dir=.deps --exclude-dir=.svn"

grep automatically uses these options on invocation

Cassidy

It's worth noting that 'grep' only gained the '--exclude-dir' option a year or two ago. Recent Linux distributions include it, but if I remember correctly I had to compile my own grep (or ask homebrew to do it) on OSX. – Harridan

I use a minor variant of this. My .bashrc creates a Bash function 'grp', which is defined as GREP_OPTIONS=xxx grep "$@". This means that the GREP_OPTIONS variable is only set for instances of grep that I run manually using 'grp'. This means I never get a situation where I run a tool, and internally it calls grep, but the tool gets confused because grep isn't behaving as it expected. Also, I have a second function 'grpy', which calls 'grp', but adds --include=*.py, to just search Python files. – Harridan

Actually, on reflection, doing it my way no longer needs to use GREP_OPTIONS at all. I now just have a shell function 'grp' which calls grep --exclude=tags --exclude_dir=.git ...etc... "$@". I like that this runs like 'ack', but I retain awareness of, and control over, what it's doing. – Harridan

Answer 8

Create a script called ~/bin/svnfind:

#!/bin/bash
#
# Attempts to behave identically to a plain `find' command while ignoring .svn/
# directories.

OPTIONS=()
PATHS=()
EXPR=()

while [[ $1 =~ ^-[HLP]+ ]]; do
    OPTIONS+=("$1")
    shift
done

while [[ $# -gt 0 ]] && ! [[ $1 =~ '^[-(),!]' ]]; do
    PATHS+=("$1")
    shift
done

# If user's expression contains no action then we'll add the normally-implied
# `-print'.
ACTION=-print

while [[ $# -gt 0 ]]; do
    case "$1" in
       -delete|-exec|-execdir|-fls|-fprint|-fprint0|-fprintf|-ok|-print|-okdir|-print0|-printf|-prune|-quit|-ls)
            ACTION=;;
    esac

    EXPR+=("$1")
    shift
done

if [[ ${#EXPR} -eq 0 ]]; then
    EXPR=(-true)
fi

exec -a "$(basename "$0")" find "${OPTIONS[@]}" "${PATHS[@]}" -name .svn -type d -prune -o '(' "${EXPR[@]}" ')' $ACTION

This script behaves identically to a plain find command but it prunes out .svn directories. Otherwise the behavior is identical.

Example:

# svnfind -name 'messages.*' -exec grep -Iw uint {} +
./messages.cpp:            Log::verbose << "Discarding out of date message: id " << uint(olderMessage.id)
./messages.cpp:    Log::verbose << "Added to send queue: " << *message << ": id " << uint(preparedMessage->id)
./messages.cpp:                Log::error << "Received message with invalid SHA-1 hash: id " << uint(incomingMessage.id)
./messages.cpp:            Log::verbose << "Received " << *message << ": id " << uint(incomingMessage.id)
./messages.cpp:            Log::verbose << "Sent message: id " << uint(preparedMessage->id)
./messages.cpp:        Log::verbose << "Discarding unsent message: id " << uint(preparedMessage->id)
./messages.cpp:        for (uint i = 0; i < 10 && !_stopThreads; ++i) {
./virus/messages.cpp:void VsMessageProcessor::_progress(const string &fileName, uint scanCount)
./virus/messages.cpp:ProgressMessage::ProgressMessage(const string &fileName, uint scanCount)
./virus/messages.h:    void _progress(const std::string &fileName, uint scanCount);
./virus/messages.h:    ProgressMessage(const std::string &fileName, uint scanCount);
./virus/messages.h:    uint        _scanCount;

Aldwin

This script does not work as i would expect. When running it with "svnfind -type f", it also prints svn-directories and the files in the svn-directories – Metamerism

@ifischer Can you add an echo to the find command and tell me what command is executed? svnfind -type f works great on my Red Hat machine. – Aldwin

Ok so it seems to be OS dependent. I'm running Debian Squeeze (its the same on Ubuntu). I do not understand what you mean by "add an echo"? – Metamerism

@ifischer Change the last line to echo find "${OPTIONS[@]}"... so it prints the find command instead of actually running it. – Aldwin

Ok changed the last line to echo find ${OPTIONS[@]} ${PATHS[@]} -name .svn -type d -prune -o ( ${EXPR[@]} ) $ACTION, This gives me the following output: find -type f -name .svn -type d -prune -o ( -true ) -print – Metamerism

@ifischer I've updated the second while loop. Can you try the script now? – Aldwin

Still the same, svnfind -type f gives me also files lying in .svn directories – Metamerism

@ifischer Well, if you can figure out why PATHS and/or OPTIONS are getting filled up incorrectly I'd appreciate it. They are supposed to be empty, and -type f is supposed to end up in EXPR. – Aldwin

Thanks for your help. but i'll write my own variant of a svn-aware find (without grep and with all find-functionality) which hopefully will be just a one-liner. Will post it here when it works. – Metamerism

Answer 9

find . | grep -v \.svn

Cumings

You have to escape . in the .svn regexp. – Starknaked

Use --fixed-strings with grep: | fgrep -v /.svn/ or ` | grep -F -v /.svn/` to exclude exactly the directory and not files with ".svn" as part of their name. – Castalia

Answer 10

Why dont you pipe your command with grep which is easily understandable:

your find command| grep -v '\.svn'

Damali

You have to escape . in the .svn regexp. – Starknaked

@Yclian without the shadow of a doubt; if you don't, directories called 'tsvn', '1svn', 'asvn' etc. will also be ignored since '.' is a regexp wildcard: 'match any character'. – Starknaked

Alright, I thought it would only happen for the case of -E and -G. I just tested, my bad. :( – Sherysherye

I like this answer because it is conceptually simpler than all of the others. I can't remember the ridiculous syntax for 'find' usage, but I can definitely remember how to use grep -v since it is used in very many situations. – Diaeresis

Answer 11

Just thought I'd add a simple alternative to Kaleb's and others' posts (which detailed the use of the find -prune option, ack, repofind commands etc.) which is particularly applicable to the usage you have described in the question (and any other similar usages):

1. For performance, you should always try to use find ... -exec grep ... + (thanks Kenji for pointing this out) or find ... | xargs egrep ... (portable) or find ... -print0 | xargs -0 egrep ... (GNU; works on filenames containing spaces) instead of find ... -exec grep ... \;.

   The find ... -exec ... + and find | xargs form does not fork egrep for each file, but rather for a bunch of files at a time, resulting in much faster execution.

2. When using the find | xargs form you can also use grep to easily and quickly prune .svn (or any directories or regular expression), i.e. find ... -print0 | grep -v '/\.svn' | xargs -0 egrep ... (useful when you need something quick and can't be bothered to remember how to set up find's -prune logic.)

   The find | grep | xargs approach is similar to GNU find's -regex option (see ghostdog74's post), but is more portable (will also work on platforms where GNU find is not available.)

Starknaked

@Vlad: Please notice that there are two forms for the -exec switch in find: one is ending with ; and the other is ending with +. The one ending with + replaces {} by a list of all matching files. Besides, your regex '/\.svn' matches file names like '.svn.txt' too. Please refer to my comments to the question for more information. – Icterus

@Vlad: Here is the POSIX standard for the find utility. Please see the -exec part :-). – Icterus

Answer 12

In a source code repository, I generally want to do things only to the text files.

The first line is all files, excluding CVS, SVN, and GIT repository files.

The second line excludes all binary files.

find . -not \( -name .svn -prune -o -name .git -prune -o -name CVS -prune \) -type f -print0 | \
xargs -0 file -n | grep -v binary | cut -d ":" -f1

Newmown

Answer 13

I use find with the -not -path options. I have not had good luck with prune.

find .  -name "*.groovy" -not -path "./target/*" -print

will find the groovy files not in the target directory path.

Wineshop

Answer 14

To resolve this problem, you can simply use this find condition:

find \( -name 'messages.*' ! -path "*/.svn/*" \) -exec grep -Iw uint {} +

You can add more restriction like this:

find \( -name 'messages.*' ! -path "*/.svn/*" ! -path "*/CVS/*" \) -exec grep -Iw uint {} +

You can find more information about this in man page section "Operators": http://unixhelp.ed.ac.uk/CGI/man-cgi?find

Lehmann

Answer 15

Note that if you do

find . -type f -name 'messages.*'

then -print is implied when the whole expression (-type f -name 'messages.*') is true, because there is no 'action' (like -exec).

While, to stop descending into certain directories, you should use anything that matches those directories and follow it by -prune (which is intended to stop descending into directories); like so:

find . -type d -name '.svn' -prune

This evaluates to True for the .svn directories, and we can use boolean short-circuit by following this by -o (OR), after which what follows after the -o is only checked when the first part is False, hence is not a .svn directory. In other words, the following:

find . -type d -name '.svn' -prune -o -name 'message.*' -exec grep -Iw uint {}

will only evalute what is right of the -o, namely -name 'message.*' -exec grep -Iw uint {}, for files NOT inside .svn directories.

Note that because .svn is likely always a directory (and not for example a file), and in this case certainly isn't matching the name 'message.*', you might as well leave out the -type d and do:

find . -name '.svn' -prune -o -name 'message.*' -exec grep -Iw uint {}

Finally, note that if you omit any action (-exec is an action), say like so:

find . -name '.svn' -prune -o -name 'message.*'

then the -print action is implied but will apply to the WHOLE expression, including the -name '.svn' -prune -o part and thus print all .svn directories as well as the 'message.*' files, which is probably not what you want. Therefore you always should use an 'action' in the right-hand side of the boolean expression when using -prune in this way. And when that action is printing you have to explicitly add it, like so:

find . -name '.svn' -prune -o -name 'message.*' -print

Interfile

Answer 16

Try findrepo which is a simple wrapper around find/grep and much faster than ack You would use it in this case like:

findrepo uint 'messages.*'

Ventage

Answer 17

wcfind is a find wrapper script that I use to automagically remove .svn directories.

Sharpshooter

Answer 18

This works for me in the Unix prompt

gfind . \( -not -wholename '*\.svn*' \) -type f -name 'messages.*' -exec grep -Iw uint {} +

The command above will list FILES that are not with .svn and do the grep you mentioned.

Lassa

is 'gfind' a typo? I don't have it on Ubuntu 14.04. – Harridan

Assuming you meant 'find', this doesn't quite work. It also filters out files like xxx.svnxxx. This is important - for example if you are using git instead of svn, you will often want to include files like .gitignore (which is not metadata, it is a regular file that is included in the repo) in the results from find. – Harridan

Answer 19

i usually pipe the output through grep one more time removing .svn, in my use it isn't much slower. typical example:

find -name 'messages.*' -exec grep -Iw uint {} + | grep -Ev '.svn|.git|.anythingElseIwannaIgnore'

OR

find . -type f -print0 | xargs -0 egrep messages. | grep -Ev '.svn|.git|.anythingElseIwannaIgnore'

Flaggy