os.walk without hidden folders

Question

I need to list all files with the containing directory path inside a folder. I tried to use os.walk, which obviously would be the perfect solution.

However, it also lists hidden folders and files. I'd like my application not to list any hidden folders or files. Is there any flag you can use to make it not yield any hidden files?

Cross-platform is not really important to me, it's ok if it only works for linux (.* pattern)

Mythical

What platform are you on? 'hidden' has different meanings on different operating systems. – Subarid

I'm on linux, I'll add that on the answer. – Mythical

Answer 1

No, there is no option to os.walk() that'll skip those. You'll need to do so yourself (which is easy enough):

for root, dirs, files in os.walk(path):
    files = [f for f in files if not f[0] == '.']
    dirs[:] = [d for d in dirs if not d[0] == '.']
    # use files and dirs

Note the dirs[:] = slice assignment; os.walk recursively traverses the subdirectories listed in dirs. By replacing the elements of dirs with those that satisfy a criteria (e.g., directories whose names don't begin with .), os.walk() will not visit directories that fail to meet the criteria.

This only works if you keep the topdown keyword argument to True, from the documentation of os.walk():

When topdown is True, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again.

Subarid

Thanks a lot, didn't know that you can modify the lists in place! – Mythical

I ran this but it did not print anything to the console. What is the typical method to display the files found this way? I added print root, dirs, files at the end but it came out very messy. – Parting

@user5359531: that depends entirely on your usecase; you could print '\n'.join([os.path.join(root, f) for f in dirs + files]), etc. – Subarid

Wander can files = [f for f in files if not f[0] == '.'] write like files[:] = [f for f in files if not f[0] == '.'], just like dirs[:] ? – Hebdomad

@linrongbin: you could, but there would be no advantage in that. files = [...] binds files to a new list, and files[:] = [...] replaces the elements in the list files is already bound to. No other code is using that list when os.walk() gives it to you. dirs on the other hand is used by os.walk() to find the next directories to go and produce files for, so if you did not use dirs[:] = [...] then directories that start with a . are still going to be visited. – Subarid

@MartijnPieters Cool, thanks, I'm not so familiar with python. – Hebdomad

@linrongbin: also see nedbatchelder.com/text/names.html to understand how Python variables work; it'll help to understand how the os.walk() implementation and the dir[:] = [...] interact. – Subarid

Answer 2

I realize it wasn't asked in the question, but I had a similar problem where I wanted to exclude both hidden files and files beginning with __, specifically __pycache__ directories. I landed on this question because I was trying to figure out why my list comprehension was not doing what I expected. I was not modifying the list in place with dirnames[:].

I created a list of prefixes I wanted to exclude and modified the dirnames in place like so:

    exclude_prefixes = ('__', '.')  # exclusion prefixes
    for dirpath, dirnames, filenames in os.walk(node):
        # exclude all dirs starting with exclude_prefixes
        dirnames[:] = [dirname
                       for dirname in dirnames
                       if not dirname.startswith(exclude_prefixes)]

Gunn

that is a great answer, works perfectly for excluding according to a list – Sutherlan

FYI, startswith can also take a tuple of strings, so you can get rid of the inner for loop and just use not dirname.startswith(exclude_prefixes) docs.python.org/2/library/stdtypes.html#str.startswith (python 2.5 and up) – Cattycornered

Answer 3

My use-case was similar to that of OP, except I wanted to return a count of the total number of sub-directories inside a certain folder. In my case I wanted to omit any sub-directories named .git (as well as any folders that may be nested inside these .git folders).

In Python 3.6.7, I found that the accepted answer's approach didn't work -- it counted all .git folder and their sub-folders. Here's what did work for me:

num_local_subdir = 0
for root, dirs, files in os.walk(local_folder_path):
    if '.git' in dirs:
        dirs.remove('.git')
    num_local_subdir += (len(dirs))

Barbarous

Answer 4

Another solution that can allow you to skip those hidden folders using any and map functions.

for root, dirs, files in os.walk(path):
    if any(map(lambda p: p[0] == '.', dirs)):
        continue

Shatterproof