os.walk without digging into directories below

Question

How do I limit os.walk to only return files in the directory I provide it?

def _dir_list(self, dir_name, whitelist):
    outputList = []
    for root, dirs, files in os.walk(dir_name):
        for f in files:
            if os.path.splitext(f)[1] in whitelist:
                outputList.append(os.path.join(root, f))
            else:
                self._email_to_("ignore")
    return outputList

Annemarie

Another case where the multitude of possible approaches and all the caveats that go with them suggests that this functionality should be added to the Python standard library. – Rowan

files_with_full_path = [f.path for f in os.scandir(dir) if f.is_file()]. In case you need only the filenames use f.name instead of f.path. This is the fastest solution and much faster than any walk or listdir, see https://mcmap.net/q/40694/-getting-a-list-of-all-subdirectories-in-the-current-directory. – Grier

Answer 1

Use the walklevel function.

import os

def walklevel(some_dir, level=1):
    some_dir = some_dir.rstrip(os.path.sep)
    assert os.path.isdir(some_dir)
    num_sep = some_dir.count(os.path.sep)
    for root, dirs, files in os.walk(some_dir):
        yield root, dirs, files
        num_sep_this = root.count(os.path.sep)
        if num_sep + level <= num_sep_this:
            del dirs[:]

It works just like os.walk, but you can pass it a level parameter that indicates how deep the recursion will go.

Protero

Does this function actually "walk" through the whole structure and then delete the entries below a certain point? Or is something more clever going on? I'm not even sure how to check this with code. --python beginner – Peroration

@mathtick: when some directory on or below the desired level is found, all of its subdirs are removed from the list of subdirs to search next. So they won't be "walked". – Protero

I just +1'd this because I was struggling with how to "delete" dirs. I had tried dirs = [] and dirs = None but those didn't work. map(dirs.remove, dirs) worked, but with some unwanted '[None]' messages printed. So, why del dirs[:] specifically? – Dorey

Note that this doesn't work when using topdown=False in os.walk. See the 4th paragraph in the docs: Modifying dirnames when topdown is False has no effect on the behavior of the walk, because in bottom-up mode the directories in dirnames are generated before dirpath itself is generated. – Gaiser

@ZacharyYoung dirs = [] and dirs = None won't work because they just create a new unrelated object and assign to the name dirs. The original list object needs to be modified in-place, not the name dirs. – Protero

How can I print the dirs of level 1 in this python script given a valid folder? Wait, I have to learn to use yield. – Asclepius

This is a great (really!) answer to another question. Therefore the amount of code is much higher than required. Other answers in this thread do the same thing for the specific request with much less code. – Karlik

Answer 2

Don't use os.walk.

Example:

import os

root = "C:\\"
for item in os.listdir(root):
    if os.path.isfile(os.path.join(root, item)):
        print item

Wellesley

@576i: this does not differentiate between files and directories – Digastric

@Alexandr os.path.isfile and os.path.isdir lets you differentiate. I don't get it, since os.path.isfile is in the sample code since '08 and your comment is from '16. This is clearly the better answer, as you're not intending to walk a directory, but to list it. – Uncritical

@DanielF, what I meant here is that you need to loop over all items, while walk gives you immediately the separate lists of dirs and files. – Digastric

Ah, ok. Actually Alex's answer seems to be better (using .next()) and it's much closer to your idea. – Uncritical

Python 3.5 has a os.scandir function which allows more sophisticated file-or-directory-object interaction. See my answer below – Barony

Answer 3

I think the solution is actually very simple.

use

break

to only do the first iteration of the for loop, there must be a more elegant way.

for root, dirs, files in os.walk(dir_name):
    for f in files:
        ...
        ...
    break
...

The first time you call os.walk, it returns tuples for the current directory, then on the next loop the contents of the next directory.

Take the original script and just add a break.

def _dir_list(self, dir_name, whitelist):
    outputList = []
    for root, dirs, files in os.walk(dir_name):
        for f in files:
            if os.path.splitext(f)[1] in whitelist:
                outputList.append(os.path.join(root, f))
            else:
                self._email_to_("ignore")
        break
    return outputList

Spagyric

This should have been the accepted answer. Simply adding a "break" after the "for f in files" loop stops the recursiveness. You might also want to make sure that topdown=True. – Friary

I just want to add this comment and say thank you for saving me time at work for giving such a good simplistic answer. – Derekderelict

same here. It's simple and imho straight forward. I'm just wondering if this behavior is in the function specification. – Placatory

Answer 4

The suggestion to use listdir is a good one. The direct answer to your question in Python 2 is root, dirs, files = os.walk(dir_name).next().

The equivalent Python 3 syntax is root, dirs, files = next(os.walk(dir_name))

Astilbe

Oh i was getting all sort of funny error from that one. ValueError: too many values to unpack – Annemarie

Nice! Feels like a hack, though. Like when you turn on an engine but only let it do one revolution and then pull the key to let it die. – Uncritical

Stumbled across this; root, dirs, files = os.walk(dir_name).next() gives me AttributeError: 'generator' object has no attribute 'next' – Lapointe

@Evan, probably because this is from 2008 and uses Python 2 syntax. In Python 3 you can write root, dirs, files = next(os.walk(dir_name)) and then the variables root, dirs, files will only correspond to the variables of the generator at the dir_name level. – Valorous

Answer 5

You could use os.listdir() which returns a list of names (for both files and directories) in a given directory. If you need to distinguish between files and directories, call os.stat() on each name.

Internode

Answer 6

If you have more complex requirements than just the top directory (eg ignore VCS dirs etc), you can also modify the list of directories to prevent os.walk recursing through them.

ie:

def _dir_list(self, dir_name, whitelist):
    outputList = []
    for root, dirs, files in os.walk(dir_name):
        dirs[:] = [d for d in dirs if is_good(d)]
        for f in files:
            do_stuff()

Note - be careful to mutate the list, rather than just rebind it. Obviously os.walk doesn't know about the external rebinding.

Ballast

Answer 7

for path, dirs, files in os.walk('.'):
    print path, dirs, files
    del dirs[:] # go only one level deep

Oracular

Answer 8

Felt like throwing my 2 pence in.

baselevel = len(rootdir.split(os.path.sep))
for subdirs, dirs, files in os.walk(rootdir):
    curlevel = len(subdirs.split(os.path.sep))
    if curlevel <= baselevel + 1:
        [do stuff]

Anny

Helpful, except "\\" assumes Windoze OS. Use os.path.sep – Puberty

Answer 9

Since Python 3.5 you can use os.scandir instead of os.listdir. Instead of strings you get an iterator of DirEntry objects in return. From the docs:

Using scandir() instead of listdir() can significantly increase the performance of code that also needs file type or file attribute information, because DirEntry objects expose this information if the operating system provides it when scanning a directory. All DirEntry methods may perform a system call, but is_dir() and is_file() usually only require a system call for symbolic links; DirEntry.stat() always requires a system call on Unix but only requires one for symbolic links on Windows.

You can access the name of the object via DirEntry.name which is then equivalent to the output of os.listdir

Barony

Not only "can" you use, you should use scandir(), as it's a lot faster than listdir(). See benchmarks here: https://mcmap.net/q/40694/-getting-a-list-of-all-subdirectories-in-the-current-directory. – Grier

Answer 10

The same idea with listdir, but shorter:

[f for f in os.listdir(root_dir) if os.path.isfile(os.path.join(root_dir, f))]

Estrous

Answer 11

root folder changes for every directory os.walk finds. I solver that checking if root == directory

def _dir_list(self, dir_name, whitelist):
    outputList = []
    for root, dirs, files in os.walk(dir_name):
        if root == dir_name: #This only meet parent folder
            for f in files:
                if os.path.splitext(f)[1] in whitelist:
                    outputList.append(os.path.join(root, f))
                else:
                    self._email_to_("ignore")
    return outputList

Peaceable

Amazingly simple and workable. Thank you Pedro – Barrett

Answer 12

You could also do the following:

for path, subdirs, files in os.walk(dir_name):
    for name in files:
        if path == ".": #this will filter the files in the current directory
             #code here

Irrefrangible

Won't this loop through all sub-dir's and files unnecessarily ? – Spagyric

Answer 13

In Python 3, I was able to do this:

import os
dir = "/path/to/files/"

#List all files immediately under this folder:
print ( next( os.walk(dir) )[2] )

#List all folders immediately under this folder:
print ( next( os.walk(dir) )[1] )

Floatplane

This also works for Python 2. How to get the second level? – Digastric

Answer 14

Why not simply use a range and os.walk combined with the zip? Is not the best solution, but would work too.

For example like this:

# your part before
for count, (root, dirs, files) in zip(range(0, 1), os.walk(dir_name)):
    # logic stuff
# your later part

Works for me on python 3.

Also: A break is simpler too btw. (Look at the answer from @Pieter)

Benign

Answer 15

import os

def listFiles(self, dir_name):
    names = []
    for root, directory, files in os.walk(dir_name):
        if root == dir_name:
            for name in files:
                names.append(name)
    return names

Rhodie

Hi Rich, welcome to Stack Overflow! Thank you for this code snippet, which might provide some limited short-term help. A proper explanation would greatly improve its long-term value by showing why this is a good solution to the problem, and would make it more useful to future readers with other, similar questions. Please edit your answer to add some explanation, including the assumptions you've made. – Sayed

Answer 16

This is how I solved it

if recursive:
    items = os.walk(target_directory)
else:
    items = [next(os.walk(target_directory))]

...

Thearchy

Answer 17

There is a catch when using listdir. The os.path.isdir(identifier) must be an absolute path. To pick subdirectories you do:

for dirname in os.listdir(rootdir):
  if os.path.isdir(os.path.join(rootdir, dirname)):
     print("I got a subdirectory: %s" % dirname)

The alternative is to change to the directory to do the testing without the os.path.join().

Dispirited

Answer 18

You can use this snippet

for root, dirs, files in os.walk(directory):
    if level > 0:
        # do some stuff
    else:
        break
    level-=1

Freddafreddi

Answer 19

create a list of excludes, use fnmatch to skip the directory structure and do the process

excludes= ['a\*\b', 'c\d\e']
for root, directories, files in os.walk('Start_Folder'):
    if not any(fnmatch.fnmatch(nf_root, pattern) for pattern in excludes):
        for root, directories, files in os.walk(nf_root):
            ....
            do the process
            ....

same as for 'includes':

if **any**(fnmatch.fnmatch(nf_root, pattern) for pattern in **includes**):

Doe

Answer 20

A slight change to Alex's answer, but using __next__():

print(next(os.walk('d:/'))[2]) or print(os.walk('d:/').__next__()[2])

with the [2] being the file in root, dirs, file mentioned in other answers

Canst

Answer 21

This is a nice python example

def walk_with_depth(root_path, depth):
        if depth < 0:
            for root, dirs, files in os.walk(root_path):
                yield [root, dirs[:], files]

            return

        elif depth == 0:
            return

        base_depth = root_path.rstrip(os.path.sep).count(os.path.sep)
        for root, dirs, files in os.walk(root_path):
            yield [root, dirs[:], files]

            cur_depth = root.count(os.path.sep)
            
            if base_depth + depth <= cur_depth:
                del dirs[:]

Donall