Python glob include hidden files and folders

Question

I try to loop over all files matching a certain extension, including those inside hidden folders. So far I haven't found a way to do this with iglob. This works for all folder except those starting with a dot:

import glob
for filename in glob.iglob('/path/**/*.ext', recursive=True):
    print(filename)

I have tried to add the dot as an optional character to no avail. I'd really like to use glob instead of residing to os.walk.

How to include all files/folders, even those starting with ., with glob?

Transpire

Answer 1

I had this same issue and wished glob.glob had an optional parameter to include dot files. I wanted to be able to include ALL dot files in ALL directories including directories that start with dot. Its just not possible to do this with glob.glob. However I found that Python has pathlib standard module which has a glob function which operates differently, it will include dot files. The function operates a little differently, in particular it does not return a list of strings, but instead path objects. However I used the following

files=[]
file_refs = pathlib.Path(".").glob(pattern)
for file in file_refs:
    files.append(str(file))

The other noticeable difference I found was a glob pattern ending with **. This returned nothing in the pathlib version but would return all the files in the glob.glob one. To get the same results I added a line to check if the pattern ended with ** and if so then append /* to it.

The following code is a replacement for your example that include the files in directories starting with dot

import pathlib
for fileref in pathlib.Path('/path/').glob('**/*.ext'):
    filename = str(fileref)
    print(filename)

Polyamide

This is indeed pretty much what I ended up with in the and, thanks – Transpire

I wonder if Path.rglob() has the same behaviour... – Diagnostic

Answer 2

From https://docs.python.org/3/library/glob.html

Note that unlike fnmatch.fnmatch(), glob treats filenames beginning with a dot (.) as special cases

If the directory contains files starting with . they won’t be matched by default. For example, consider a directory containing card.gif and .card.gif:

import glob  
glob.glob('*.gif') # ['card.gif']  
glob.glob('.c*') # ['.card.gif']

From what I see it requires two separate globs to get both hidden and not hidden ones, for example using https://mcmap.net/q/109443/-python-glob-multiple-filetypes.

Fadden

Do you think it would be possible with only one glob pass @reducingactivity? – Chest

Answer 3

From python 3.11 onward it is possible to do:

glob.iglob('/path/*', include_hidden=True)

Halfdan

This also works with glob.glob. – Yearly

Answer 4

Adding an answer for the bounty question; getting the result of hidden and non-hidden files in a single command.

As @reducidng activity mentioned, glob treats . files as a special use-case. To get both regular and hidden files in a single loop, we can use itertools.chain with glob.iglob iterators. for example,

→ ls -A
.chen     file.text so1.py

>>> import glob, itertools
>>> for i in itertools.chain(glob.iglob('**'), glob.iglob('.**')):
...     print(i)
...
file.text
so1.py
.chen

# If you want it as a variable, you can list() it.
>>> l = list(itertools.chain(glob.iglob('**'), glob.iglob('.**')))
>>> l
['file.text', 'so1.py', '.chen']
>>>

Note: it does not fully work (yet). Let's say you have .hello, .dot/hello.txt, .dot/.hello.txt, nodot/hello.txt, nodot/.hello.txt. Then neither:

itertools.chain(glob.iglob('**', recursive=True), glob.iglob('.**', recursive=True))

nor

itertools.chain(glob.iglob('**/*', recursive=True), glob.iglob('.**/*', recursive=True))

give all files.

Heterosexuality

@Chest recursive is a function parameter of iglob, so yes it works. – Heterosexuality

It does not fully work @ChenA.: let's say you have .hello, .dot/hello.txt, .dot/.hello.txt, nodot/hello.txt, nodot/.hello.txt. Then itertools.chain(glob.iglob('**', recursive=True), glob.iglob('.**', recursive=True)) does not take all of them. – Chest

I see. In this case you would need to iterate each directory. – Heterosexuality

I added this important note (for future readers) in your answer @ChenA. – Chest

Answer 5

To find hidden files matching a certain extension, you can try this

glob.glob('/path/**/.*.ext')

If you want to find all files in a folder

glob.glob('/path/*') + glob.glob('/path/.*')

Ptisan