Cross platform hidden file detection

Question

What is the best way to do cross-platform handling of hidden files? (preferably in Python, but other solutions still appreciated)

Simply checking for a leading '.' works for *nix/Mac, and file attributes work on Windows. However, this seems a little simplistic, and also doesn't account for alternative methods of hiding things (.hidden files, etc.). Is there a standard way to deal with this?

Brokendown

this abstract discussion is great, but I also want to see code! look at me having to google everything myself. – Erinerina

There's no concept of hidden files in linux, dotFiles are generally not-shown, but files cannot be hidden. It's a purely DOS/Windows concept AFAIK. – Baculiform

There is a concept of hidden files in OS X. Of course you can show hidden files in Finder, just as you can pass -a to ls, but they're still called hidden. And leading dot is not the only way for something to be hidden, either. See my answer if you need this. – Diabolic

Also, while the linux kernel obviously doesn't define a concept of "hidden files", freedesktop and other standards (or collaborations or whatever) define a lot of things the kernel doesn't. If your file manager, your apps' open/save dialogs, etc. all agree on what is hidden by default, that's what matters. Technically, you should really talk about "hidden files on GNU/linux/FSH/freedesktop/blah/blah" rather than "linux", but… the OP isn't the one who said "linux" in the first place anyway. – Diabolic

@abarnert: macOS has two kinds of hidden files. -a will include files starting with . in ls output. But files can be hidden from Finder using extra flags that do not hide them from ls output. ls doesn't normally even show these flags but adding -O will show them. – Plerre

Answer 1

We actually address this in a project we write. What we do is have a number of different "hidden file checkers" that are registered with a main checker. We pass each file through these to see if it should be hidden or not.

These checkers are not only for different OS's etc, but we plug into version control "ignored" files, and optional user overrides by glob or regular expression.

It mostly amounts to what you have done, but in a pluggable, flexible and extensible way.

See source code here: https://bitbucket.org/aafshar/pida-main/src/tip/pida/services/filemanager/filemanager.py

Erinn

Is this project open source? Can you post source to this code? You give a rough idea of how it works, but an example or pseduo code would be helpful. – Euonymus

Yes, it is open source. pida.co.uk, source code listed here pida.co.uk/trac/browser/pida/services/filemanager. Bear in mind that the architecture for plugging things into this is higher up in the core application, and any other part of code can "register" to use it. – Erinn

This URL is dead (no such domain), and en.wikipedia.org/wiki/PIDA links to the same URL. Since this is 5 years old, that's not surprising… but it's a bit disappointing for a question that people are still asking duplicates of to have an obsolete answer like this. – Diabolic

@Diabolic 2 years later, and I'm looking for an answer as well – Ascendant

Answer 2

Here's a script that runs on Python 2.5+ and should do what you're looking for:

import ctypes
import os

def is_hidden(filepath):
    name = os.path.basename(os.path.abspath(filepath))
    return name.startswith('.') or has_hidden_attribute(filepath)

def has_hidden_attribute(filepath):
    try:
        attrs = ctypes.windll.kernel32.GetFileAttributesW(unicode(filepath))
        assert attrs != -1
        result = bool(attrs & 2)
    except (AttributeError, AssertionError):
        result = False
    return result

I added something similar to has_hidden_attribute to jaraco.windows. If you have jaraco.windows >= 2.3:

from jaraco.windows import filesystem

def has_hidden_attribute(filepath):
    return filesystem.GetFileAttributes(filepath).hidden

As Ben has pointed out, on Python 3.5, you can use the stdlib:

import os, stat

def has_hidden_attribute(filepath):
    return bool(os.stat(filepath).st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)

Though you may still want to use jaraco.windows for the more Pythonic API.

Opisthognathous

I had to correct is_hidden because it wasn't doing a proper test on Unix. In fact, it's not even clear to me what a proper test is on Unix. Is '..' hidden? What about '../..'? Surely the special name is hidden, but should the function resolve it? For compatibility, I think so. I've changed is_hidden to test the basename for startswith('.'). – Opisthognathous

This still doesn't handle the HFS+ hidden attributes in OS X, or special rules like ~/Library being hidden in OS X 10.7+, or… – Diabolic

Also if a file doesn't exist is_hidden returns False instead of throwing. Perhaps an edit is in place. – Aubreir

@Aubreir os.path is<x> functions do not check for file existence. If these are chained, like if exists(f) and isfile(f) and islink(f), then each call would check for file existence. – Bourg

Note: on Python 3.5 and above, the Win32 file attributes are available directly via os.stat(path).st_file_attributes. See the docs. – Scriptwriter

Note that pre Python 3.5, Windows system files can also be checked for by using a mask of 4 instead of 2, i.e. is_system = bool(attrs & 4). HTH. – Gruber

Answer 3

Jason R. Coombs's answer is sufficient for Windows. And most POSIX GUI file managers/open dialogs/etc. probably follow the same "dot-prefix-means-hidden" convention as ls. But not Mac OS X.

There are at least four ways a file or directory can be hidden in Finder, file open panels, etc.:

• Dot prefix.
• HFS+ invisible attribute.
• Finder Info hidden flag.
• Matches a special blacklist built into CoreFoundation (which is different on each OS version—e.g., ~/Library is hidden in 10.7+, but not in 10.6).

Trying to write your own code to handle all of that is not going to be easy. And you'll have to keep it up-to-date, as I'm willing to bet the blacklist will change with most OS versions, Finder Info will eventually go from deprecated to completely unsupported, extended attributes may be supported more broadly than HFS+, …

But if you can require pyobjc (which is already included with recent Apple-supplied Python, and can be installed via pip otherwise), you can just call Apple's code:

import Foundation

def is_hidden(path):
    url = Foundation.NSURL.fileURLWithPath_(path)
    return url.getResourceValue_forKey_error_(None, Foundation.NSURLIsHiddenKey, None)[0]

def listdir_skipping_hidden(path):
    url = Foundation.NSURL.fileURLWithPath_(path)
    fm = Foundation.NSFileManager.defaultManager()
    urls = fm.contentsOfDirectoryAtURL_includingPropertiesForKeys_options_error_(
        url, [], Foundation.NSDirectoryEnumerationSkipsHiddenFiles, None)[0]
    return [u.path() for u in urls]

This should work on any Python that pyobjc supports, on OS X 10.6+. If you want 10.5 or earlier, directory enumeration flags didn't exist yet, so the only option is something like filtering something like contentsOfDirectoryAtPath_error_ (or just os.listdir) on is_hidden.

If you have to get by without pyobjc, you can drop down to the CoreFoundation equivalents, and use ctypes. The key functions are CFURLCopyResourcePropertyForKey for is_hidden and CFURLEnumeratorCreateForDirectoryURL for listing a directory.

See http://pastebin.com/aCUwTumB for an implementation.

I've tested with:

• OS X 10.6, 32-bit python.org 3.3.0
• OS X 10.8, 32-bit Apple 2.7.2
• OS X 10.8, 64-bit Apple 2.7.2
• OS X 10.8, 64-bit python.org 3.3.0

It works as appropriate on each (e.g., it skips ~/Library on 10.8, but shows it on 10.6).

It should work on any OS X 10.6+ and any Python 2.6+. If you need OS X 10.5, you need to use the old APIs (or os.listdir) and filter on is_hidden. If you need Python 2.5, change the bytes checks to str checks (which of course breaks 3.x) and the with to an ugly try/finally or manual releasing.

If anyone plans on putting this code into a library, I would strongly suggest checking for pyobjc first (import Foundation and, if you don't get an ImportError you win), and only using the ctypes code if it's not available.

---

One last note:

Some people looking for this answer are trying to reinvent a wheel they don't need to.

Often, when people are doing something like this, they're building a GUI and want to, e.g., show a file browsers with an option to hide or show hidden files. Many of the popular cross-platform GUI frameworks (Qt, wx, etc.) have this support built in. (Also, many of them are open source, so you can read their code to see how they do it.)

That may not answer your question—e.g., they may just be passing a "filter hidden files" flag to the platform's native file-browser dialog, but you're trying to build a console-mode file-browser and can't do that. But if it does, just use it.

Diabolic

Excellent answer, though it's obviously Mac specific. If you would be willing to provide an implementation using ctypes only and the CoreFoundation equivalents, I'll incorporate it into my answer for a universal implementation. – Opisthognathous

@JasonR.Coombs: I think it's actually better to use pyobjc if available (as, again, it is for Apple-supplied Python)… but having a fallback to CF isn't a bad idea. Let me find the equivalents and write it up. – Diabolic

Now that I have a Mac, I have the opportunity to implement and test this functionality. Unfortunately, I'm finding that even the pyobjc implementation isn't working for me. In particular, [0] of the result of getResourceValue_forKey_error_ is always True. Looking at the result, it seems that [1] is the desired value. My lack of familiarity with Objective-C and pyobjc leaves me lost for an explanation. – Opisthognathous

Answer 4

Incorporating my previous answer as well as that from @abarnert, I've released jaraco.path 1.1 with cross-platform support for hidden file detection. With that package installed, to detect the hidden state of any file, simply invoke is_hidden:

from jaraco import path
path.is_hidden(file)

Opisthognathous

That's very kind, unfortunately this library (currently) depends on Foundation. – Sherrie

I'll happily update it with a ctypes-based implementation or add a requirement on pyobjc if you can draft a technique. – Opisthognathous

Answer 5

The following code is from a python script where I needed to determine if a given file is hidden:

import os, sys
from stat import *

def isHidden(path):
    st = os.lstat(path)
    return st.st_flags & UF_HIDDEN

Slocum

stat.UF_HIDDEN is only applicable to MacOs 10.5+. This is not cross-platform. – Candlepower

Answer 6

"Is there a standard way to deal with this?" Yes. Use a standard (i.e., POSIX-compliant) OS.

Since Windows is non-standard -- well -- there's no applicable standard. Wouldn't it be great if there was? I feel your pain.

Anything you try to do that's cross-platform like that will have Win32 oddities.

Your solution is -- for the present state of affairs -- excellent. At some point in the future, Microsoft may elect to write a POSIX-compliant OS. Until then, you're coping well with the situation.

Hobble

@S.Lott: I normally like your comments, but this is not helpful. There are a number of other modules in the standard library that cope with the non-standard ways of Windows. I'm sure the OP was just looking for something like that. – Euonymus

@technomalogical: True, there are modules that minimize the differences. The point is that non-standard means non-standard. It's a bummer. There's no fairy-dust solution that makes non-standard operating systems appear standard. The OP had a good solution. – Hobble

POSIX is a standard for Unix-like operating systems. Complaining that an operating system that isn't Unix-like doesn't conform to it is like complaining Python doesn't conform to the ECMAScript standard. Meanwhile, this doesn't answer the question, and fails to notice that macOS is the more problematic OS in this regard. It comes across as so slanted as to be cringy. I recommend deleting. – Candlepower