Turn a string into a valid filename?

Question

I have a string that I want to use as a filename, so I want to remove all characters that wouldn't be allowed in filenames, using Python.

I'd rather be strict than otherwise, so let's say I want to retain only letters, digits, and a small set of other characters like "_-.() ". What's the most elegant solution?

The filename needs to be valid on multiple operating systems (Windows, Linux and Mac OS) - it's an MP3 file in my library with the song title as the filename, and is shared and backed up between 3 machines.

Earl

Shouldn't this be built into the os.path module? – Samarasamarang

Perhaps, although her use case would require a single path that's safe across all platforms, not just the current one, which is something os.path isn't designed to handle. – Shawannashawl

To expand on the above comment: the current design of os.path actually loads a different library depending on the os (see the second note in the documentation). So if a quoting function was implemented in os.path it could only quote the string for POSIX-safety when running on a POSIX system or for windows-safety when running on windows. The resulting filename would not necessarily be valid across both windows and POSIX, which is what the question asks for. – Spheroidal

It's easy enough to use the path functions for a different OS. For example, on unix, use import ntpath; ntpath.abspath("a.txt") to get the absolute path of a file on a (hypothetical) Windows file system. Or use posixpath for posix systems (linux, Mac OS) – Pardew

Answer 1

You can look at the Django framework (but take their licence into account!) for how they create a "slug" from arbitrary text. A slug is URL- and filename- friendly.

The Django text utils define a function, slugify(), that's probably the gold standard for this kind of thing. Essentially, their code is the following.

import unicodedata
import re

def slugify(value, allow_unicode=False):
    """
    Taken from https://github.com/django/django/blob/master/django/utils/text.py
    Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated
    dashes to single dashes. Remove characters that aren't alphanumerics,
    underscores, or hyphens. Convert to lowercase. Also strip leading and
    trailing whitespace, dashes, and underscores.
    """
    value = str(value)
    if allow_unicode:
        value = unicodedata.normalize('NFKC', value)
    else:
        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')
    value = re.sub(r'[^\w\s-]', '', value.lower())
    return re.sub(r'[-\s]+', '-', value).strip('-_')

And the older version:

def slugify(value):
    """
    Normalizes string, converts to lowercase, removes non-alpha characters,
    and converts spaces to hyphens.
    """
    import unicodedata
    value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')
    value = unicode(re.sub('[^\w\s-]', '', value).strip().lower())
    value = unicode(re.sub('[-\s]+', '-', value))
    # ...
    return value

There's more, but I left it out, since it doesn't address slugification, but escaping.

Bieber

The last line should be: value = unicode(re.sub('[-\s]+', '-', value)) – Cobham

Thanks - I could be missing something, but I'm getting: "normalize() argument 2 must be unicode, not str" – Chon

"normalize() argument 2". Means the value. If the value must be Unicode, then, you have to be sure that it's actually Unicode. Or. You might want to leave out unicode normalization if your actual value is actually an ASCII string. – Bieber

In case anyone hasn't noticed the positive side of this approach is that it doesn't just remove non-alpha characters, but attempts to find good substitutes first (via the NFKD normalization), so é becomes e, a superscript 1 becomes a normal 1, etc. Thanks – Kokoruda

For a non-web application. I've rewritten the function to retain German (and other non-ascii) letters as is: re_slugify = re.compile('[^\w\s-]', re.UNICODE) def slugify(value): value = unicodedata.normalize('NFKD', value) value = unicode(re_slugify.sub('', value).strip()) value = re.sub('[-\s]+', '-', value) return value – Hyponitrite

The slugify function has been moved to django/utils/text.py, and that file also contains a get_valid_filename function. – Digitize

I found it helpful to replace spaces with "-" as well – Borscht

then add return value at the end – Thermoelectrometer

The slugify function (python 3 version) is available at github.com/django/django/blob/master/django/utils/text.py – Thermoelectrometer

This is useful thanks. For my application I want to replace some non-alphanumeric characters with '-' rather than delete them. For example I want 'ab/cd' to be converted to 'ab-cd'. To achieve this I added the characters to your last two regex's, E.g.: r'[^-\w\s\/]' and r'[-\s\/]+'. – Ehling

Answer 2

You can use list comprehension together with the string methods.

>>> s
'foo-bar#baz?qux@127/\\9]'
>>> "".join(x for x in s if x.isalnum())
'foobarbazqux1279'

Octans

Although it doesn't include the few extra chars he wanted "_-.() ". Still my favourite solution though ;) – Sabin

Note that you can omit the square brackets. In this case a generator expression is passed to join, which saves the step of creating an otherwise unused list. – Purposeless

+1 Loved this. Slight modification I've done: "".join([x if x.isalnum() else "_" for x in s]) -- would yield a result where invalid items are _, like they're blanked. Maybe tha thelps someone else. – Bobettebobina

This solution is great! I made a slight modification though: filename = "".join(i for i in s if i not in "\/:*?<>|") – Ulda

Unfortunately it doesn't even allow spaces and dots, but I like the idea. – Newcomer

@tiktak: to (also) allow spaces, dots and underscores you can go for "".join( x for x in s if (x.isalnum() or x in "._- ")) – Gilbart

@AlexKrycek except that you need to double the backslash. – Laurent

Answer 3

What is the reason to use the strings as file names? If human readability is not a factor I would go with base64 module which can produce file system safe strings. It won't be readable but you won't have to deal with collisions and it is reversible.

import base64
file_name_string = base64.urlsafe_b64encode(your_string)

Update: Changed based on Matthew comment.

Whipstitch

Warning! base64 encoding by default includes the "/" character as valid output which isn't valid in filenames on a lot of systems. Instead use base64.urlsafe_b64encode(your_string) – Ciapha

This should absolutely be regarded as the ideal answer for webservers with any internal user-named content. Even if the administrator needs to go find something, you can easily write a script to transform all queries to the same form. – Cowes

Actually human readability is almost always a factor, even if only for debugging purposes. – Sphere

In Python 3 your_string needs to be a byte array or the result of encode('ascii') for this to work. – Clout

def url2filename(url): url = url.encode('UTF-8') return base64.urlsafe_b64encode(url).decode('UTF-8') def filename2url(f): return base64.urlsafe_b64decode(f).decode('UTF-8') – Jackboot

As @Nouemon says, you need to convert your string to a byte array, for example as shown in this answer: #7585935 – Cackle

Answer 4

This whitelist approach (ie, allowing only the chars present in valid_chars) will work if there aren't limits on the formatting of the files or combination of valid chars that are illegal (like ".."), for example, what you say would allow a filename named " . txt" which I think is not valid on Windows. As this is the most simple approach I'd try to remove whitespace from the valid_chars and prepend a known valid string in case of error, any other approach will have to know about what is allowed where to cope with Windows file naming limitations and thus be a lot more complex.

>>> import string
>>> valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
>>> valid_chars
'-_.() abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
>>> filename = "This Is a (valid) - filename%$&$ .txt"
>>> ''.join(c for c in filename if c in valid_chars)
'This Is a (valid) - filename .txt'

Furlong

valid_chars = frozenset(valid_chars) wouldn't hurt. It is 1.5 times faster if applied to allchars. – Refinement

Warning: This maps two different strings to the same string >>> import string >>> valid_chars = "-.() %s%s" % (string.ascii_letters, string.digits) >>> valid_chars '-.() abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789' >>> filename = "a.com/hello/world" >>> ''.join(c for c in filename if c in valid_chars) 'a.comhelloworld' >>> filename = "a.com/helloworld" >>> ''.join(c for c in filename if c in valid_chars) 'a.comhelloworld' >>> – Electrojet

Not to mention that naming a file "CON" on Windows will get you into trouble... – Profession

A slight rearrangement makes specifying a substitute character straightforward. First the original functionality: ''.join(c if c in valid_chars else '' for c in filename) or with a substituted character or string for every invalid character: ''.join(c if c in valid_chars else '.' for c in filename) – Grad

Minor point,".txt" is a valid filename on Windows, though I suspect there are lot more ".gitignore" files out there. – Dowdy

Answer 5

There is a nice project on Github called python-slugify:

Install:

pip install python-slugify

Then use:

>>> from slugify import slugify
>>> txt = "This\ is/ a%#$ test ---"
>>> slugify(txt)
'this-is-a-test'

Dremadremann

I like this library but it's not that good as I thought. Initial testing ok but it converts also dots. So test.txt gets test-txt which is too much. – Odilia

Slugify looks to be actively maintained, and the current (as of 2021-11-03) version has many options, some of which could be used to control . and - substitution. See also the comments to @therealmarv's answer below. – Alga

Answer 6

Just like S.Lott answered, you can look at the Django Framework for how they convert a string to a valid filename.

The most recent and updated version is found in utils/text.py, and defines get_valid_filename, which is as follows:

def get_valid_filename(name):
    s = str(name).strip().replace(" ", "_")
    s = re.sub(r"(?u)[^-\w.]", "", s)
    if s in {"", ".", ".."}:
        raise SuspiciousFileOperation("Could not derive file name from '%s'" % name)
    return s

( See https://github.com/django/django/blob/master/django/utils/text.py )

Pardew

for the lazy already on django: django.utils.text import get_valid_filename – Selwyn

In case you are unfamiliar with regex, re.sub(r'(?u)[^-\w.]', '', s) removes all characters which are not letters, not numbers (0-9), not the underscore ('_'), not the dash ('-'), and not the period ('.'). "Letters" here includes all unicode letters, such as 漢語. – Pardew

You may want to also check for length: Filenames are limited to 255 characters (or, you know, 32; depending on the FS) – Updraft

return re.sub(r'(?u)[^-\w.]', '_', s) for better readability – Metropolis

Answer 7

Just to further complicate things, you are not guaranteed to get a valid filename just by removing invalid characters. Since allowed characters differ on different filenames, a conservative approach could end up turning a valid name into an invalid one. You may want to add special handling for the cases where:

• The string is all invalid characters (leaving you with an empty string)

• You end up with a string with a special meaning, eg "." or ".."

• On windows, certain device names are reserved. For instance, you can't create a file named "nul", "nul.txt" (or nul.anything in fact) The reserved names are:

  CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9

You can probably work around these issues by prepending some string to the filenames that can never result in one of these cases, and stripping invalid characters.

Yowl

Answer 8

This is the solution I ultimately used:

import unicodedata

validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)

def removeDisallowedFilenameChars(filename):
    cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
    return ''.join(c for c in cleanedFilename if c in validFilenameChars)

The unicodedata.normalize call replaces accented characters with the unaccented equivalent, which is better than simply stripping them out. After that all disallowed characters are removed.

My solution doesn't prepend a known string to avoid possible disallowed filenames, because I know they can't occur given my particular filename format. A more general solution would need to do so.

Earl

you should be able to use uuid.uuid4() for your unique prefix – Carioca

camel case .. ahh – Inhesion

Could this be edited / updated to work with Python 3.6 ? – Purport

Answer 9

In one line:

valid_file_name = re.sub('[^\w_.)( -]', '', any_string)

you can also put '_' character to make it more readable (in case of replacing slashs, for example)

Fulfil

Answer 10

Keep in mind, there are actually no restrictions on filenames on Unix systems other than

• It may not contain \0
• It may not contain /

Everything else is fair game.

$ touch "
> even multiline
> haha
> ^[[31m red ^[[0m
> evil"
$ ls -la 
-rw-r--r--       0 Nov 17 23:39 ?even multiline?haha??[31m red ?[0m?evil
$ ls -lab
-rw-r--r--       0 Nov 17 23:39 \neven\ multiline\nhaha\n\033[31m\ red\ \033[0m\nevil
$ perl -e 'for my $i ( glob(q{./*even*}) ){ print $i; } '
./
even multiline
haha
 red 
evil

Yes, i just stored ANSI Colour Codes in a file name and had them take effect.

For entertainment, put a BEL character in a directory name and watch the fun that ensues when you CD into it ;)

Hammering

The OP states that "The filename needs to be valid on multiple operating systems" – Pardew

@Pardew that clarification was added 10 hours after my answer was posted :) Check the OP's edit log. – Hammering

\0 seems fine on Ubuntu – Mustee

Answer 11

I realise there are many answers but they mostly rely on regular expressions or external modules, so I'd like to throw in my own answer. A pure python function, no external module needed, no regular expression used. My approach is not to clean invalid chars, but to only allow valid ones.

def normalizefilename(fn):
    validchars = "-_.() "
    out = ""
    for c in fn:
      if str.isalpha(c) or str.isdigit(c) or (c in validchars):
        out += c
      else:
        out += "_"
    return out    

if you like, you can add your own valid chars to the validchars variable at the beginning, such as your national letters that don't exist in English alphabet. This is something you may or may not want: some file systems that don't run on UTF-8 might still have problems with non-ASCII chars.

This function is to test for a single file name validity, so it will replace path separators with _ considering them invalid chars. If you want to add that, it is trivial to modify the if to include os path separator.

Bedivere

Answer 12

You could use the re.sub() method to replace anything not "filelike". But in effect, every character could be valid; so there are no prebuilt functions (I believe), to get it done.

import re

str = "File!name?.txt"
f = open(os.path.join("/tmp", re.sub('[^-a-zA-Z0-9_.() ]+', '', str))

Would result in a filehandle to /tmp/filename.txt.

Wassyngton

You need the dash to go first in the group matcher so it doesn't appear as a range. re.sub('[^-a-zA-Z0-9_.() ]+', '', str) – Kirst

Had issues with blank spaces at the start and end of a base filename so changed the regex to [^-a-zA-Z0-9_.() ]+|^\s+|\s+$. This is in case anyone else comes across this problem. – Athalie

Answer 13

If you don't mind installing a package, this should be useful: https://pypi.org/project/pathvalidate/

From https://pypi.org/project/pathvalidate/#sanitize-a-filename:

from pathvalidate import sanitize_filename

fname = "fi:l*e/p\"a?t>h|.t<xt"
print(f"{fname} -> {sanitize_filename(fname)}\n")
fname = "\0_a*b:c<d>e%f/(g)h+i_0.txt"
print(f"{fname} -> {sanitize_filename(fname)}\n")

Output

fi:l*e/p"a?t>h|.t<xt -> filepath.txt
_a*b:c<d>e%f/(g)h+i_0.txt -> _abcde%f(g)h+i_0.txt

Lighter

This is a very clean solution – Velmaveloce

Answer 14

>>> import string
>>> safechars = bytearray(('_-.()' + string.digits + string.ascii_letters).encode())
>>> allchars = bytearray(range(0x100))
>>> deletechars = bytearray(set(allchars) - set(safechars))
>>> filename = u'#ab\xa0c.$%.txt'
>>> safe_filename = filename.encode('ascii', 'ignore').translate(None, deletechars).decode()
>>> safe_filename
'abc..txt'

It doesn't handle empty strings, special filenames ('nul', 'con', etc).

Refinement

+1 for translation tables, it is by far the most efficient method. For the special filenames/empties, a simple pre-condition check will suffice and for extraneous periods that's a simple correction as well. – Priestly

While translate is slightly more efficient than a regexp, that time will most likely be dwarfed if you actually try to open the file, which no doubt you are intending to do. Thus I prefer more a more readable regexp solution than the mess above – Tachistoscope

I'm also worried about the blacklist. Granted, it's a blacklist that's based off a whitelist, but still. It seems less... safe. How do you know that "allchars" is actually complete? – Evvie

@isaaclw: '.translate()' accepts 256-char string as a translation table (byte-to-byte translation). '.maketrans()' creates such string. All values are covered; it is a pure whitelist approach – Refinement

What about the filename '.' (a single dot). That would not work on Unixes as the present directory is using that name. – Michael

@FinnÅrupNielsen: yes, special filenames should be handled. It is worse on Windows. It depends on the task how exactly it should be done. – Refinement

@Tachistoscope you are right that you should use the most straightforward simple code first and reserve optimized code for [rare] special cases where it is necessary and if StackOverflow allowed just one answer per question then a regex answer that might be more readable then the .translate-based answer should have been preferred but SO allows more than one answer so for those who might need to handle a special case here it is. The "slight" performance improvement can be 4x – Refinement

Answer 15

Why not just wrap the "osopen" with a try/except and let the underlying OS sort out whether the file is valid?

This seems like much less work and is valid no matter which OS you use.

Answer

Does it valid the name though? I mean, if the OS is not happy, then you still need to do something, right? – Meeting

In some cases, the OS/Language may silently munge your filename into an alternative form, but when you do a directory listing, you'll get a different name out. And this can lead to a "when I write the file its there, but when I look for the file its called something else" problem. ( I'm talking about behaviour I've heard about on VAX ... ) – Hammering

Moreover, "The filename needs to be valid on multiple operating systems", which you can't detect with an osopen running on one machine. – Lollard

Answer 16

Another issue that the other comments haven't addressed yet is the empty string, which is obviously not a valid filename. You can also end up with an empty string from stripping too many characters.

What with the Windows reserved filenames and issues with dots, the safest answer to the question “how do I normalise a valid filename from arbitrary user input?” is “don't even bother try”: if you can find any other way to avoid it (eg. using integer primary keys from a database as filenames), do that.

If you must, and you really need to allow spaces and ‘.’ for file extensions as part of the name, try something like:

import re
badchars= re.compile(r'[^A-Za-z0-9_. ]+|^\.|\.$|^ | $|^$')
badnames= re.compile(r'(aux|com[1-9]|con|lpt[1-9]|prn)(\.|$)')

def makeName(s):
    name= badchars.sub('_', s)
    if badnames.match(name):
        name= '_'+name
    return name

Even this can't be guaranteed right especially on unexpected OSs — for example RISC OS hates spaces and uses ‘.’ as a directory separator.

Gaughan

Answer 17

Though you have to be careful. It is not clearly said in your intro, if you are looking only at latine language. Some words can become meaningless or another meaning if you sanitize them with ascii characters only.

imagine you have "forêt poésie" (forest poetry), your sanitization might give "fort-posie" (strong + something meaningless)

Worse if you have to deal with chinese characters.

"下北沢" your system might end up doing "---" which is doomed to fail after a while and not very helpful. So if you deal with only files I would encourage to either call them a generic chain that you control or to keep the characters as it is. For URIs, about the same.

Froggy

Answer 18

I liked the python-slugify approach here but it was stripping dots also away which was not desired. So I optimized it for uploading a clean filename to s3 this way:

pip install python-slugify

Example code:

s = 'Very / Unsafe / file\nname hähä \n\r .txt'
clean_basename = slugify(os.path.splitext(s)[0])
clean_extension = slugify(os.path.splitext(s)[1][1:])
if clean_extension:
    clean_filename = '{}.{}'.format(clean_basename, clean_extension)
elif clean_basename:
    clean_filename = clean_basename
else:
    clean_filename = 'none' # only unclean characters

Output:

>>> clean_filename
'very-unsafe-file-name-haha.txt'

This is so failsafe, it works with filenames without extension and it even works for only unsafe characters file names (result is none here).

Odilia

I like this, don't reinvent the wheel, don't import the whole Django framework if you don't need it, don't directly paste the code if you are not going to maintain it in the future, and generated string tries to matches similar letters to safe ones, so new string is easier to read. – Nook

To use underscore instead of dash: name=slugify(s, separator='_') – Nook

Answer 19

Answer modified for python 3.6

import string
import unicodedata

validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
def removeDisallowedFilenameChars(filename):
    cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
    return ''.join(chr(c) for c in cleanedFilename if chr(c) in validFilenameChars)

Notify

Could you explain your answer in details? – Hardden

Its the same answer accepted by Sophie Gage. But it has been modified to work on python 3.6 – Notify

Answer 20

Yet another answer for Windows specific paths, using simple replacement and no funky modules:

import re

def check_for_illegal_char(input_str):
    # remove illegal characters for Windows file names/paths 
    # (illegal filenames are a superset (41) of the illegal path names (36))
    # this is according to windows blacklist obtained with Powershell
    # from: https://mcmap.net/q/63621/-what-characters-are-forbidden-in-windows-and-linux-directory-names/44750843#44750843
    #
    # PS> $enc = [system.Text.Encoding]::UTF8
    # PS> $FileNameInvalidChars = [System.IO.Path]::GetInvalidFileNameChars()
    # PS> $FileNameInvalidChars | foreach { $enc.GetBytes($_) } | Out-File -FilePath InvalidFileCharCodes.txt

    illegal = '\u0022\u003c\u003e\u007c\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008' + \
              '\u0009\u000a\u000b\u000c\u000d\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015' + \
              '\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f\u003a\u002a\u003f\u005c\u002f' 

    output_str, _ = re.subn('['+illegal+']','_', input_str)
    output_str = output_str.replace('\\','_')   # backslash cannot be handled by regex
    output_str = output_str.replace('..','_')   # double dots are illegal too, or at least a bad idea 
    output_str = output_str[:-1] if output_str[-1] == '.' else output_str # can't have end of line '.'

    if output_str != input_str:
        print(f"The name '{input_str}' had invalid characters, "
              f"name was modified to '{output_str}'")

    return output_str

When tested with check_for_illegal_char('fas\u0003\u0004good\\..asd.'), I get:

The name 'fas♥♦good\..asd.' had invalid characters, name was modified to 'fas__good__asd'

Dupont

That worked great for me, I just commented the replacement of the backslash as I used this with os.makesidrs (I needed it to create directories...) But great answer, thanks. – Salisbarry

For some reason unknown to a Python newbie I had to replace the '\u0022' with '\x22' etc. to get working. Thanks. – Rancorous

Answer 21

Not exactly what OP was asking for but this is what I use because I need unique and reversible conversions:

# p3 code
def safePath (url):
    return ''.join(map(lambda ch: chr(ch) if ch in safePath.chars else '%%%02x' % ch, url.encode('utf-8')))
safePath.chars = set(map(lambda x: ord(x), '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+-_ .'))

Result is "somewhat" readable, at least from a sysadmin point of view.

Baxy

A wrapper for this with no spaces in files names: def safe_filename(filename): return safePath(filename.strip().replace(' ','_')) – Torrie

1) Does not allow unicode chars that are actually valid filename chars. 2) Can you provide inverse function? Love a mathematically-minded solution :) – Trincomalee

Answer 22

When confronted with the same problem I used python-slugify.

Usage was also suggested by Shoham but, as therealmarv pointed out, by default python-slugify also converts dots.

This behaviour can be overruled by including dots into the regex_pattern argument.

> filename = "This is a väryì' Strange File-Nömé.jpeg"
> pattern = re.compile(r'[^-a-zA-Z0-9.]+')
> slugify(filename,regex_pattern=pattern) 
'this-is-a-varyi-strange-file-nome.jpeg'

Note that the regex pattern was copied from the

ALLOWED_CHARS_PATTERN_WITH_UPPERCASE

global variable within the slugify.py file of the python-slugify package and extended with "."

Keep in mind that special characters like .() must be escaped with \.

If you want to preserve uppercase letters use the lowercase=False argument.

> filename = "This is a väryì' Strange File-Nömé.jpeg"
> pattern = re.compile(r'[^-a-zA-Z0-9.]+')
> slugify(filename,regex_pattern=pattern, lowercase=False) 
'This-is-a-varyi-Strange-File-Nome.jpeg'

This worked using Python 3.8.4 and python-slugify 4.0.1

Ziegler

Answer 23

Most of these solutions don't work.

'/hello/world' -> 'helloworld'

'/helloworld'/ -> 'helloworld'

This isn't what you want generally, say you are saving the html for each link, you're going to overwrite the html for a different webpage.

I pickle a dict such as:

{'helloworld': 
    (
    {'/hello/world': 'helloworld', '/helloworld/': 'helloworld1'},
    2)
    }

2 represents the number that should be appended to the next filename.

I look up the filename each time from the dict. If it's not there, I create a new one, appending the max number if needed.

Electrojet

note, if using helloworld1, you also need to check helloworld1 isn't in use and so on.. – Electrojet

Answer 24

Still haven't found a good library to generate a valid filename. Note that in languages like German, Norwegian or French special characters in filenames are very common and totally OK. So I ended up with my own library:

# util/files.py

CHAR_MAX_LEN = 31
CHAR_REPLACE = '_'

ILLEGAL_CHARS = [
    '#',  # pound
    '%',  # percent
    '&',  # ampersand
    '{',  # left curly bracket
    '}',  # right curly bracket
    '\\',  # back slash
    '<',  # left angle bracket
    '>',  # right angle bracket
    '*',  # asterisk
    '?',  # question mark
    '/',  # forward slash
    ' ',  # blank spaces
    '$',  # dollar sign
    '!',  # exclamation point
    "'",  # single quotes
    '"',  # double quotes
    ':',  # colon
    '@',  # at sign
    '+',  # plus sign
    '`',  # backtick
    '|',  # pipe
    '=',  # equal sign
]


def generate_filename(
        name, char_replace=CHAR_REPLACE, length=CHAR_MAX_LEN, 
        illegal=ILLEGAL_CHARS, replace_dot=False):
    ''' return clean filename '''
    # init
    _elem = name.split('.')
    extension = _elem[-1].strip()
    _length = length - len(extension) - 1
    label = '.'.join(_elem[:-1]).strip()[:_length]
    filename = ''
    
    # replace '.' ?
    if replace_dot:
        label = label.replace('.', char_replace)
    
    # clean
    for char in label + '.' + extension:
        if char in illegal:
            char = char_replace
        filename += char      
    
    return filename

generate_filename('nucgae zutaäer..0.1.docx', replace_dot=False)

nucgae_zutaäer..0.1.docx

generate_filename('nucgae zutaäer..0.1.docx', replace_dot=True)

nucgae_zutaäer__0_1.docx

Eugenaeugene

Answer 25

UPDATE

All links broken beyond repair in this 6 year old answer.

Also, I also wouldn't do it this way anymore, just base64 encode or drop unsafe chars. Python 3 example:

import re
t = re.compile("[a-zA-Z0-9.,_-]")
unsafe = "abc∂éåß®∆˚˙©¬ñ√ƒµ©∆∫ø"
safe = [ch for ch in unsafe if t.match(ch)]
# => 'abc'

With base64 you can encode and decode, so you can retrieve the original filename again.

But depending on the use case you might be better off generating a random filename and storing the metadata in separate file or DB.

from random import choice
from string import ascii_lowercase, ascii_uppercase, digits
allowed_chr = ascii_lowercase + ascii_uppercase + digits

safe = ''.join([choice(allowed_chr) for _ in range(16)])
# => 'CYQ4JDKE9JfcRzAZ'

ORIGINAL LINKROTTEN ANSWER:

The bobcat project contains a python module that does just this.

It's not completely robust, see this post and this reply.

So, as noted: base64 encoding is probably a better idea if readability doesn't matter.

• Docs https://svn.origo.ethz.ch/bobcat/src-doc/safefilename-module.html
• Source https://svn.origo.ethz.ch/bobcat/trunk/src/bobcatlib/safefilename.py

Jaejaeger

All links dead. Man, do something. – Without

Answer 26

I'm sure this isn't a great answer, since it modifies the string it's looping over, but it seems to work alright:

import string
for chr in your_string:
 if chr == ' ':
   your_string = your_string.replace(' ', '_')
 elif chr not in string.ascii_letters or chr not in string.digits:
    your_string = your_string.replace(chr, '')

Inculcate

I've found this "".join( x for x in s if (x.isalnum() or x in "._- ")) on this post comments – Hegarty

Answer 27

Here, this should cover all the bases. It handles all types of issues for you, including (but not limited too) character substitution.

Works in Windows, *nix, and almost every other file system. Allows printable characters only.

def txt2filename(txt, chr_set='normal'):
    """Converts txt to a valid Windows/*nix filename with printable characters only.

    args:
        txt: The str to convert.
        chr_set: 'normal', 'universal', or 'inclusive'.
            'universal':    ' -.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
            'normal':       Every printable character exept those disallowed on Windows/*nix.
            'extended':     All 'normal' characters plus the extended character ASCII codes 128-255
    """

    FILLER = '-'

    # Step 1: Remove excluded characters.
    if chr_set == 'universal':
        # Lookups in a set are O(n) vs O(n * x) for a str.
        printables = set(' -.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz')
    else:
        if chr_set == 'normal':
            max_chr = 127
        elif chr_set == 'extended':
            max_chr = 256
        else:
            raise ValueError(f'The chr_set argument may be normal, extended or universal; not {chr_set=}')
        EXCLUDED_CHRS = set(r'<>:"/\|?*')               # Illegal characters in Windows filenames.
        EXCLUDED_CHRS.update(chr(127))                  # DEL (non-printable).
        printables = set(chr(x)
                         for x in range(32, max_chr)
                         if chr(x) not in EXCLUDED_CHRS)
    result = ''.join(x if x in printables else FILLER   # Allow printable characters only.
                     for x in txt)

    # Step 2: Device names, '.', and '..' are invalid filenames in Windows.
    DEVICE_NAMES = 'CON,PRN,AUX,NUL,COM1,COM2,COM3,COM4,' \
                   'COM5,COM6,COM7,COM8,COM9,LPT1,LPT2,' \
                   'LPT3,LPT4,LPT5,LPT6,LPT7,LPT8,LPT9,' \
                   'CONIN$,CONOUT$,..,.'.split()        # This list is an O(n) operation.
    if result in DEVICE_NAMES:
        result = f'-{result}-'

    # Step 3: Maximum length of filename is 255 bytes in Windows and Linux (other *nix flavors may allow longer names).
    result = result[:255]

    # Step 4: Windows does not allow filenames to end with '.' or ' ' or begin with ' '.
    result = re.sub(r'^[. ]', FILLER, result)
    result = re.sub(r' $', FILLER, result)

    return result

This solution needs no external libraries. It substitutes non-printable filenames too because they are not always simple to deal with.

Banky

Answer 28

minimal working example with pytest

Transforming every character, which is not A-Z, a-z, 0-9 or - into _.

demo

slugify("a b c")
Out[9]: 'a_b_c'
slugify("https://www.algorithmus-schmiede.de/kontakt/")
Out[10]: 'https___www_algorithmus-schmiede_de_kontakt_'
slugify("a-b c")
Out[11]: 'a-b_c'

full code with pytest

import re

def slugify(str_: str):
    slug = re.sub(r'[^A-z0-9-]', '_', str_)
    return slug

import pytest

@pytest.mark.parametrize(
    "inp, outp_exp",
    [
        pytest.param("a b c", "a_b_c", id="whitespace -> underscore"),
        pytest.param("https://www.algorithmus-schmiede.de/kontakt/",
                     "https___www_algorithmus-schmiede_de_kontakt_", id="url"),
        pytest.param("a-b c", "a-b_c", id="minus conserved"),
    ]
)
def test_slugify(inp, outp_exp):
    assert slugify(inp) == outp_exp

Caballero