python listing dirs in a different order based upon platform

Question

I am writing and testing code on XPsp3 w/ python 2.7. I am running the code on 2003 server w/ python 2.7. My dir structure will look something like this

d:\ssptemp
d:\ssptemp\ssp9-1
d:\ssptemp\ssp9-2
d:\ssptemp\ssp9-3
d:\ssptemp\ssp9-4
d:\ssptemp\ssp10-1    
d:\ssptemp\ssp10-2
d:\ssptemp\ssp10-3
d:\ssptemp\ssp10-4

Inside each directory there is one or more files that will have "IWPCPatch" as part of the filename.

Inside one of these files (one in each dir), there will be the line 'IWPCPatchFinal_a.wsf'

What I do is

1) os.walk across all dirs under d:\ssptemp

2) find all files with 'IWPCPatch' in the filename

3) check the contents of the file for 'IWPCPatchFinal_a.wsf'

4) If contents is true I add the path of that file to a list.

My problem is that on my XP machine it works fine. If I print out the results of the list I get several items in the order I listed above.

When I move it to the server 2003 machine I get the same contents in a different order. It comes ssp10-X, then ssp9-X. And this is causing me issues with a different area in the program.

I can see from my output that it begins the os.walk in the wrong order, but I don't know why that is occuring.

import os
import fileinput

print "--createChain--"

listOfFiles = []
for path, dirs, files in os.walk('d:\ssptemp'):

    print "parsing dir(s)"
    for file in files:
        newFile = os.path.join(path,file)
        if newFile.find('IWPCPatch') >= 0:
            for line in fileinput.FileInput(newFile):
                if "IWPCPatchFinal_a.wsf" in line:
                    listOfFiles.append(newFile)                            
                    print "Added", newFile

for item in listOfFiles:
    print "list item", item

Webster

Answer 1

The order of directories within os.walk is not necessarily alphabetical (I think it's actually dependent upon how they're stored within the dirent on the filesystem). It will likely be stable on the same exact directory (on the same filesystem) if you don't change the directory contents (ie, repeated calls will return the same order), but the order is not necessarily alphabetical.

If you want to have an ordered list of filenames you will have to build the list and then sort it yourself.

Hulking

What he said. Don't rely on the operating system to do the right thing (especially in the case of Windows, which varies wildly between each release). Sort the data on your own before you iterate it. – Polled

Thats what I thought you were going to say :( – Webster

If I do a list.sort() on XP it puts them in the same order as 2003. What I can do is pull of the numbers in the dir name and create the list as [path,X,Y] (i.e. [<path>,10,1]). Then I could do some type of sort on the X,Y column. But how do you do a 2 column sort? – Webster

@Webster ask in a separate question with a little more detail, it's tough to do this sort of thing in comments (it's tough by design, because it ought to be a question!) – Hulking

This should point you in the right direction: #4623946 – Polled

Note that os.walk() calls os.listdir() for which the python documentation explicitly states that the entries are returned in arbitrary order: "Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order. It does not include the special entries '.' and '..' even if they are present in the directory." – Complexity

Answer 2

for path, dirs, files in os.walk('d:\ssptemp'):

    # sort dirs and files
    dirs.sort()
    files.sort()

    print "parsing dir(s)"
    # ...

Appressed

this answer is exactly what i need. and simple. – Leung