How might I wrap the FindXFile-style APIs to the STL-style Iterator Pattern in C++?

Question

I'm working on wrapping up the ugly innards of the FindFirstFile/FindNextFile loop (though my question applies to other similar APIs, such as RegEnumKeyEx or RegEnumValue, etc.) inside iterators that work in a manner similar to the Standard Template Library's istream_iterators.

I have two problems here. The first is with the termination condition of most "foreach" style loops. STL style iterators typically use operator!= inside the exit condition of the for, i.e.

std::vector<int> test;
for(std::vector<int>::iterator it = test.begin(); it != test.end(); it++) {
 //Do stuff
}

My problem is I'm unsure how to implement operator!= with such a directory enumeration, because I do not know when the enumeration is complete until I've actually finished with it. I have sort of a hack together solution in place now that enumerates the entire directory at once, where each iterator simply tracks a reference counted vector, but this seems like a kludge which can be done a better way.

The second problem I have is that there are multiple pieces of data returned by the FindXFile APIs. For that reason, there's no obvious way to overload operator* as required for iterator semantics. When I overload that item, do I return the file name? The size? The modified date? How might I convey the multiple pieces of data to which such an iterator must refer to later in an ideomatic way? I've tried ripping off the C# style MoveNext design but I'm concerned about not following the standard idioms here.

class SomeIterator {
public:
 bool next(); //Advances the iterator and returns true if successful, false if the iterator is at the end.
 std::wstring fileName() const;
 //other kinds of data....
};

EDIT: And the caller would look like:

SomeIterator x = ??; //Construct somehow
while(x.next()) {
    //Do stuff
}

Thanks!

Billy3

EDIT2: I have fixed some bugs and written some tests.

Implementation:

#pragma once
#include <queue>
#include <string>
#include <boost/noncopyable.hpp>
#include <boost/make_shared.hpp>
#include <boost/iterator/iterator_facade.hpp>
#include <Windows.h>
#include <Shlwapi.h>
#pragma comment(lib, "shlwapi.lib")
#include "../Exception.hpp"

namespace WindowsAPI { namespace FileSystem {

template <typename Filter_T = AllResults, typename Recurse_T = NonRecursiveEnumeration>
class DirectoryIterator;

//For unit testing
struct RealFindXFileFunctions
{
    static HANDLE FindFirst(LPCWSTR lpFileName, LPWIN32_FIND_DATAW lpFindFileData) {
        return FindFirstFile(lpFileName, lpFindFileData);
    };
    static BOOL FindNext(HANDLE hFindFile, LPWIN32_FIND_DATAW lpFindFileData) {
        return FindNextFile(hFindFile, lpFindFileData);
    };
    static BOOL Close(HANDLE hFindFile) {
        return FindClose(hFindFile);
    };
};

inline std::wstring::const_iterator GetLastSlash(std::wstring const&pathSpec) {
    return std::find(pathSpec.rbegin(), pathSpec.rend(), L'\\').base();
}

class Win32FindData {
    WIN32_FIND_DATA internalData;
    std::wstring rootPath;
public:
    Win32FindData(const std::wstring& root, const WIN32_FIND_DATA& data) :
        rootPath(root), internalData(data) {};
    DWORD GetAttributes() const {
        return internalData.dwFileAttributes;
    };
    bool IsDirectory() const {
        return (internalData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0;
    };
    bool IsFile() const {
        return !IsDirectory();
    };
    unsigned __int64 GetSize() const {
        ULARGE_INTEGER intValue;
        intValue.LowPart = internalData.nFileSizeLow;
        intValue.HighPart = internalData.nFileSizeHigh;
        return intValue.QuadPart;
    };
    std::wstring GetFolderPath() const {
        return rootPath;
    };
    std::wstring GetFileName() const {
        return internalData.cFileName;
    };
    std::wstring GetFullFileName() const {
        return rootPath + L"\\" + internalData.cFileName;
    };
    std::wstring GetShortFileName() const {
        return internalData.cAlternateFileName;
    };
    FILETIME GetCreationTime() const {
        return internalData.ftCreationTime;
    };
    FILETIME GetLastAccessTime() const {
        return internalData.ftLastAccessTime;
    };
    FILETIME GetLastWriteTime() const {
        return internalData.ftLastWriteTime;
    };
};

template <typename FindXFileFunctions_T>
class BasicNonRecursiveEnumeration : public boost::noncopyable
{
    WIN32_FIND_DATAW currentData;
    HANDLE hFind;
    std::wstring currentDirectory;
    void IncrementCurrentDirectory() {
        if (hFind == INVALID_HANDLE_VALUE) return;
        BOOL success =
            FindXFileFunctions_T::FindNext(hFind, &currentData);
        if (success)
            return;
        DWORD error = GetLastError();
        if (error == ERROR_NO_MORE_FILES) {
            FindXFileFunctions_T::Close(hFind);
            hFind = INVALID_HANDLE_VALUE;
        } else {
            WindowsApiException::Throw(error);
        }
    };
    bool IsValidDotDirectory()
    {
        return !Valid() &&
            (!wcscmp(currentData.cFileName, L".") || !wcscmp(currentData.cFileName, L".."));
    };
    void IncrementPastDotDirectories() {
        while (IsValidDotDirectory()) {
            IncrementCurrentDirectory();
        }
    };
    void PerformFindFirstFile(std::wstring const&pathSpec)
    {
        hFind = FindXFileFunctions_T::FindFirst(pathSpec.c_str(), &currentData);
        if (Valid()
            && GetLastError() != ERROR_PATH_NOT_FOUND
            && GetLastError() != ERROR_FILE_NOT_FOUND)
            WindowsApiException::ThrowFromLastError();
    };
public:
    BasicNonRecursiveEnumeration() : hFind(INVALID_HANDLE_VALUE) {};
    BasicNonRecursiveEnumeration(const std::wstring& pathSpec) :
        hFind(INVALID_HANDLE_VALUE) {
        std::wstring::const_iterator lastSlash = GetLastSlash(pathSpec);
        if (lastSlash != pathSpec.begin())
            currentDirectory.assign(pathSpec.begin(), lastSlash-1);
        PerformFindFirstFile(pathSpec);
        IncrementPastDotDirectories();
    };
    bool equal(const BasicNonRecursiveEnumeration<FindXFileFunctions_T>& other) const {
        if (this == &other)
            return true;
        return hFind == other.hFind;
    };
    Win32FindData dereference() {
        return Win32FindData(currentDirectory, currentData);
    };
    void increment() {
        IncrementCurrentDirectory();
    };
    bool Valid() {
        return hFind == INVALID_HANDLE_VALUE;
    };
    virtual ~BasicNonRecursiveEnumeration() {
        if (!Valid())
            FindXFileFunctions_T::Close(hFind);
    };
};

typedef BasicNonRecursiveEnumeration<RealFindXFileFunctions> NonRecursiveEnumeration;

template <typename FindXFileFunctions_T>
class BasicRecursiveEnumeration : public boost::noncopyable
{
    std::wstring fileSpec;
    std::deque<std::deque<Win32FindData> > enumeratedData;
    void EnumerateDirectory(const std::wstring& nextPathSpec) {
        std::deque<Win32FindData> newDeck;
        BasicNonRecursiveEnumeration<FindXFileFunctions_T> begin(nextPathSpec), end;
        for(; !begin.equal(end); begin.increment()) {
            newDeck.push_back(begin.dereference()); 
        }
        if (!newDeck.empty()) {
            enumeratedData.push_back(std::deque<Win32FindData>()); //Swaptimization
            enumeratedData.back().swap(newDeck);
        }
    };
    void PerformIncrement() {
        if (enumeratedData.empty()) return;
        if (enumeratedData.back().front().IsDirectory()) {
            std::wstring nextSpec(enumeratedData.back().front().GetFullFileName());
            nextSpec.append(L"\\*");
            enumeratedData.back().pop_front();
            EnumerateDirectory(nextSpec);
        } else {
            enumeratedData.back().pop_front();
        }
        while (Valid() && enumeratedData.back().empty())
            enumeratedData.pop_back();
    }
    bool CurrentPositionNoMatchFileSpec() const
    {
        return !enumeratedData.empty() && !PathMatchSpecW(enumeratedData.back().front().GetFileName().c_str(), fileSpec.c_str());
    }
public:
    BasicRecursiveEnumeration() {};
    BasicRecursiveEnumeration(const std::wstring& pathSpec) {
        std::wstring::const_iterator lastSlash = GetLastSlash(pathSpec);
        if (lastSlash == pathSpec.begin()) {
            fileSpec = pathSpec;
            EnumerateDirectory(L"*");
        } else {
            fileSpec.assign(lastSlash, pathSpec.end());
            std::wstring firstQuery(pathSpec.begin(), lastSlash);
            firstQuery.push_back(L'*');
            EnumerateDirectory(firstQuery);
            while (CurrentPositionNoMatchFileSpec())
                PerformIncrement();
        }
    };
    void increment() {
        do
        {
            PerformIncrement();
        } while (CurrentPositionNoMatchFileSpec());
    };
    bool equal(const BasicRecursiveEnumeration<FindXFileFunctions_T>& other) const {
        if (!Valid())
            return !other.Valid();
        if (!other.Valid())
            return false;
        return this == &other;
    };
    Win32FindData dereference() const {
        return enumeratedData.back().front();
    };
    bool Valid() const {
        return !enumeratedData.empty();
    };
};

typedef BasicRecursiveEnumeration<RealFindXFileFunctions> RecursiveEnumeration;

struct AllResults
{
    bool operator()(const Win32FindData&) {
        return true;
    };
}; 

struct FilesOnly
{
    bool operator()(const Win32FindData& arg) {
        return arg.IsFile();
    };
};

template <typename Filter_T, typename Recurse_T>
class DirectoryIterator : 
    public boost::iterator_facade<
        DirectoryIterator<Filter_T, Recurse_T>,
        Win32FindData,
        std::input_iterator_tag,
        Win32FindData
    >
{
    friend class boost::iterator_core_access;
    boost::shared_ptr<Recurse_T> impl;
    Filter_T filter;
    void increment() {
        do {
            impl->increment();
        } while (impl->Valid() && !filter(impl->dereference()));
    };
    bool equal(const DirectoryIterator& other) const {
        return impl->equal(*other.impl);
    };
    Win32FindData dereference() const {
        return impl->dereference();
    };
public:
    DirectoryIterator(Filter_T functor = Filter_T()) :
        impl(boost::make_shared<Recurse_T>()),
        filter(functor) {
    };
    explicit DirectoryIterator(const std::wstring& pathSpec, Filter_T functor = Filter_T()) :
        impl(boost::make_shared<Recurse_T>(pathSpec)),
        filter(functor) {
    };
};

}}

Tests:

#include <queue>
#include "../WideCharacterOutput.hpp"
#include <boost/test/unit_test.hpp>
#include "../../WindowsAPI++/FileSystem/Enumerator.hpp"
using namespace WindowsAPI::FileSystem;


struct SimpleFakeFindXFileFunctions
{
    static std::deque<WIN32_FIND_DATAW> fakeData;
    static std::wstring insertedFileName;

    static HANDLE FindFirst(LPCWSTR lpFileName, LPWIN32_FIND_DATAW lpFindFileData) {
        insertedFileName.assign(lpFileName);
        if (fakeData.empty()) {
            SetLastError(ERROR_PATH_NOT_FOUND);
            return INVALID_HANDLE_VALUE;
        }
        *lpFindFileData = fakeData.front();
        fakeData.pop_front();
        return reinterpret_cast<HANDLE>(42);
    };
    static BOOL FindNext(HANDLE hFindFile, LPWIN32_FIND_DATAW lpFindFileData) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        if (fakeData.empty()) {
            SetLastError(ERROR_NO_MORE_FILES);
            return 0;
        }
        *lpFindFileData = fakeData.front();
        fakeData.pop_front();
        return 1;
    };
    static BOOL Close(HANDLE hFindFile) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        return 1;
    };

};

std::deque<WIN32_FIND_DATAW> SimpleFakeFindXFileFunctions::fakeData;
std::wstring SimpleFakeFindXFileFunctions::insertedFileName;

struct ErroneousFindXFileFunctionFirst
{
    static HANDLE FindFirst(LPCWSTR, LPWIN32_FIND_DATAW) {
        SetLastError(ERROR_ACCESS_DENIED);
        return INVALID_HANDLE_VALUE;
    };
    static BOOL FindNext(HANDLE hFindFile, LPWIN32_FIND_DATAW) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        return 1;
    };
    static BOOL Close(HANDLE hFindFile) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        return 1;
    };
};

struct ErroneousFindXFileFunctionNext
{
    static HANDLE FindFirst(LPCWSTR, LPWIN32_FIND_DATAW) {
        return reinterpret_cast<HANDLE>(42);
    };
    static  BOOL FindNext(HANDLE hFindFile, LPWIN32_FIND_DATAW) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        SetLastError(ERROR_INVALID_PARAMETER);
        return 0;
    };
    static BOOL Close(HANDLE hFindFile) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        return 1;
    };
};

struct DirectoryIteratorTestsFixture
{
    typedef SimpleFakeFindXFileFunctions fakeFunctor;
    DirectoryIteratorTestsFixture() {
        WIN32_FIND_DATAW test;
        wcscpy_s(test.cFileName, L".");
        wcscpy_s(test.cAlternateFileName, L".");
        test.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
        GetSystemTimeAsFileTime(&test.ftCreationTime);
        test.ftLastWriteTime = test.ftCreationTime;
        test.ftLastAccessTime = test.ftCreationTime;
        test.nFileSizeHigh = 0;
        test.nFileSizeLow = 0;
        fakeFunctor::fakeData.push_back(test);

        wcscpy_s(test.cFileName, L"..");
        wcscpy_s(test.cAlternateFileName, L"..");
        fakeFunctor::fakeData.push_back(test);

        wcscpy_s(test.cFileName, L"File.txt");
        wcscpy_s(test.cAlternateFileName, L"FILE.TXT");
        test.nFileSizeLow = 1024;
        test.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
        fakeFunctor::fakeData.push_back(test);

        wcscpy_s(test.cFileName, L"System32");
        wcscpy_s(test.cAlternateFileName, L"SYSTEM32");
        test.nFileSizeLow = 0;
        test.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
        fakeFunctor::fakeData.push_back(test);
    };
    ~DirectoryIteratorTestsFixture() {
        fakeFunctor::fakeData.clear();
    };
};

BOOST_FIXTURE_TEST_SUITE( DirectoryIteratorTests, DirectoryIteratorTestsFixture )

template<typename fakeFunctor>
static void NonRecursiveIteratorAssertions()
{
    typedef DirectoryIterator<AllResults
        ,BasicNonRecursiveEnumeration<SimpleFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*");
    testType end;
    BOOST_CHECK_EQUAL(fakeFunctor::insertedFileName, L"C:\\Windows\\*");
    BOOST_CHECK(begin->GetFolderPath() == L"C:\\Windows");
    BOOST_CHECK(begin->GetFileName() == L"File.txt");
    BOOST_CHECK(begin->GetFullFileName() == L"C:\\Windows\\File.txt");
    BOOST_CHECK(begin->GetShortFileName() == L"FILE.TXT");
    BOOST_CHECK_EQUAL(begin->GetSize(), 1024);
    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK(begin != end);
    begin++;
    BOOST_CHECK(begin->GetFileName() == L"System32");
    BOOST_CHECK(begin->GetFullFileName() == L"C:\\Windows\\System32");
    BOOST_CHECK(begin->GetShortFileName() == L"SYSTEM32");
    BOOST_CHECK_EQUAL(begin->GetSize(), 0);
    BOOST_CHECK(begin->IsDirectory());
    begin++;
    BOOST_CHECK(begin == end);
}

BOOST_AUTO_TEST_CASE( BasicEnumeration )
{
    NonRecursiveIteratorAssertions<fakeFunctor>();
}

BOOST_AUTO_TEST_CASE( NoRootDirectories )
{
    fakeFunctor::fakeData.pop_front();
    fakeFunctor::fakeData.pop_front();
    NonRecursiveIteratorAssertions<fakeFunctor>();
}

static void EmptyIteratorAssertions() {
    typedef DirectoryIterator<AllResults
        ,BasicNonRecursiveEnumeration<SimpleFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*");
    testType end;
    BOOST_CHECK(begin == end);
}

BOOST_AUTO_TEST_CASE( Empty1 )
{
    fakeFunctor::fakeData.clear();
    EmptyIteratorAssertions();
}

BOOST_AUTO_TEST_CASE( Empty2 )
{
    fakeFunctor::fakeData.erase(fakeFunctor::fakeData.begin() + 2, fakeFunctor::fakeData.end());
    EmptyIteratorAssertions();
}

BOOST_AUTO_TEST_CASE( CorrectDestruction )
{
    typedef DirectoryIterator<AllResults
        ,BasicNonRecursiveEnumeration<SimpleFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*");
    testType end;
}

BOOST_AUTO_TEST_CASE( Exceptions )
{
    typedef DirectoryIterator<AllResults,BasicNonRecursiveEnumeration<ErroneousFindXFileFunctionFirst> >
        firstFailType;
    BOOST_CHECK_THROW(firstFailType(L"C:\\Windows\\*"), WindowsAPI::ErrorAccessDeniedException);
    typedef DirectoryIterator<AllResults,BasicNonRecursiveEnumeration<ErroneousFindXFileFunctionNext> >
        nextFailType;
    nextFailType constructedOkay(L"C:\\Windows\\*");
    BOOST_CHECK_THROW(constructedOkay++, WindowsAPI::ErrorInvalidParameterException);
}

BOOST_AUTO_TEST_SUITE_END()

struct RecursiveFakeFindXFileFunctions
{
    static std::deque<std::pair<std::deque<WIN32_FIND_DATA> , std::wstring> >  fakeData;
    static std::size_t openHandles;
    static HANDLE FindFirst(LPCWSTR lpFileName, LPWIN32_FIND_DATAW lpFindFileData) {
        BOOST_REQUIRE(!fakeData.empty());
        BOOST_REQUIRE_EQUAL(lpFileName, fakeData.front().second);
        openHandles++;
        BOOST_REQUIRE_EQUAL(openHandles, 1);
        if (fakeData.front().first.empty()) {
            openHandles--;
            SetLastError(ERROR_PATH_NOT_FOUND);
            return INVALID_HANDLE_VALUE;
        }
        *lpFindFileData = fakeData.front().first.front();
        fakeData.front().first.pop_front();
        return reinterpret_cast<HANDLE>(42);
    };
    static BOOL FindNext(HANDLE hFindFile, LPWIN32_FIND_DATAW lpFindFileData) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        if (fakeData.front().first.empty()) {
            SetLastError(ERROR_NO_MORE_FILES);
            return 0;
        }
        *lpFindFileData = fakeData.front().first.front();
        fakeData.front().first.pop_front();
        return 1;
    };
    static BOOL Close(HANDLE hFindFile) {
        BOOST_CHECK_EQUAL(reinterpret_cast<HANDLE>(42), hFindFile);
        openHandles--;
        BOOST_REQUIRE_EQUAL(openHandles, 0);
        fakeData.pop_front();
        return 1;
    };
};

std::deque<std::pair<std::deque<WIN32_FIND_DATA> , std::wstring> > RecursiveFakeFindXFileFunctions::fakeData;
std::size_t RecursiveFakeFindXFileFunctions::openHandles;

struct RecursiveDirectoryFixture
{
    RecursiveDirectoryFixture() {
        WIN32_FIND_DATAW tempData;
        ZeroMemory(&tempData, sizeof(tempData));
        std::deque<WIN32_FIND_DATAW> dequeData;

        wcscpy_s(tempData.cFileName, L".");
        wcscpy_s(tempData.cAlternateFileName, L".");
        tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
        GetSystemTimeAsFileTime(&tempData.ftCreationTime);
        tempData.ftLastWriteTime = tempData.ftCreationTime;
        tempData.ftLastAccessTime = tempData.ftCreationTime;
        dequeData.push_back(tempData);

        wcscpy_s(tempData.cFileName, L"..");
        wcscpy_s(tempData.cAlternateFileName, L"..");
        dequeData.push_back(tempData);

        wcscpy_s(tempData.cFileName, L"MySubDirectory");
        wcscpy_s(tempData.cAlternateFileName, L"MYSUBD~1");
        dequeData.push_back(tempData);

        wcscpy_s(tempData.cFileName, L"MyFile.txt");
        wcscpy_s(tempData.cAlternateFileName, L"MYFILE.TXT");
        tempData.nFileSizeLow = 500;
        tempData.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
        dequeData.push_back(tempData);

        RecursiveFakeFindXFileFunctions::fakeData.push_back
            (std::make_pair(dequeData, L"C:\\Windows\\*"));

        dequeData.clear();

        wcscpy_s(tempData.cFileName, L".");
        wcscpy_s(tempData.cAlternateFileName, L".");
        tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
        GetSystemTimeAsFileTime(&tempData.ftCreationTime);
        tempData.ftLastWriteTime = tempData.ftCreationTime;
        tempData.ftLastAccessTime = tempData.ftCreationTime;
        dequeData.push_back(tempData);

        wcscpy_s(tempData.cFileName, L"..");
        wcscpy_s(tempData.cAlternateFileName, L"..");
        dequeData.push_back(tempData);

        wcscpy_s(tempData.cFileName, L"MyFile2.txt");
        wcscpy_s(tempData.cAlternateFileName, L"NYFILE2.TXT");
        tempData.nFileSizeLow = 1024;
        tempData.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
        dequeData.push_back(tempData);

        RecursiveFakeFindXFileFunctions::fakeData.push_back
            (std::make_pair(dequeData, L"C:\\Windows\\MySubDirectory\\*"));
    };
    ~RecursiveDirectoryFixture() {
        RecursiveFakeFindXFileFunctions::fakeData.clear();
    };
};

BOOST_AUTO_TEST_SUITE( RecursiveDirectoryIteratorTests )

BOOST_AUTO_TEST_CASE( BasicEnumerationTxt )
{
    RecursiveDirectoryFixture DataFixture;
    typedef DirectoryIterator<AllResults
        ,BasicRecursiveEnumeration<RecursiveFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*.txt");
    testType end;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 1024);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows\\MySubDirectory");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile2.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MySubDirectory\\MyFile2.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 500);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MyFile.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin == end);
}

BOOST_AUTO_TEST_CASE( BasicEnumerationAll )
{
    RecursiveDirectoryFixture DataFixture;
    typedef DirectoryIterator<AllResults
        ,BasicRecursiveEnumeration<RecursiveFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*");
    testType end;

    BOOST_CHECK(begin->IsDirectory());
    BOOST_CHECK_EQUAL(begin->GetSize(), 0);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MySubDirectory");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MySubDirectory");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 1024);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows\\MySubDirectory");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile2.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MySubDirectory\\MyFile2.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 500);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MyFile.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin == end);
}

BOOST_AUTO_TEST_CASE( RecursionOrderMaintained )
{
    WIN32_FIND_DATAW tempData;
    ZeroMemory(&tempData, sizeof(tempData));
    std::deque<WIN32_FIND_DATAW> dequeData;

    wcscpy_s(tempData.cFileName, L".");
    wcscpy_s(tempData.cAlternateFileName, L".");
    tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
    GetSystemTimeAsFileTime(&tempData.ftCreationTime);
    tempData.ftLastWriteTime = tempData.ftCreationTime;
    tempData.ftLastAccessTime = tempData.ftCreationTime;
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"..");
    wcscpy_s(tempData.cAlternateFileName, L"..");
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"MySubDirectory");
    wcscpy_s(tempData.cAlternateFileName, L"MYSUBD~1");
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"MyFile.txt");
    wcscpy_s(tempData.cAlternateFileName, L"MYFILE.TXT");
    tempData.nFileSizeLow = 500;
    tempData.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"Zach");
    wcscpy_s(tempData.cAlternateFileName, L"ZACH");
    tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
    tempData.nFileSizeLow = 0;
    dequeData.push_back(tempData);

    RecursiveFakeFindXFileFunctions::fakeData.push_back
        (std::make_pair(dequeData, L"C:\\Windows\\*"));

    dequeData.clear();

    wcscpy_s(tempData.cFileName, L".");
    wcscpy_s(tempData.cAlternateFileName, L".");
    tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
    GetSystemTimeAsFileTime(&tempData.ftCreationTime);
    tempData.ftLastWriteTime = tempData.ftCreationTime;
    tempData.ftLastAccessTime = tempData.ftCreationTime;
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"..");
    wcscpy_s(tempData.cAlternateFileName, L"..");
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"MyFile2.txt");
    wcscpy_s(tempData.cAlternateFileName, L"NYFILE2.TXT");
    tempData.nFileSizeLow = 1024;
    tempData.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
    dequeData.push_back(tempData);

    RecursiveFakeFindXFileFunctions::fakeData.push_back
        (std::make_pair(dequeData, L"C:\\Windows\\MySubDirectory\\*"));

    dequeData.clear();
    wcscpy_s(tempData.cFileName, L".");
    wcscpy_s(tempData.cAlternateFileName, L".");
    tempData.dwFileAttributes = FILE_ATTRIBUTE_DIRECTORY;
    GetSystemTimeAsFileTime(&tempData.ftCreationTime);
    tempData.ftLastWriteTime = tempData.ftCreationTime;
    tempData.ftLastAccessTime = tempData.ftCreationTime;
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"..");
    wcscpy_s(tempData.cAlternateFileName, L"..");
    dequeData.push_back(tempData);

    wcscpy_s(tempData.cFileName, L"ZachFile.txt");
    wcscpy_s(tempData.cAlternateFileName, L"ZACHFILE.TXT");
    tempData.nFileSizeLow = 1024;
    tempData.dwFileAttributes = FILE_ATTRIBUTE_NORMAL;
    dequeData.push_back(tempData);

    RecursiveFakeFindXFileFunctions::fakeData.push_back
        (std::make_pair(dequeData, L"C:\\Windows\\Zach\\*"));

    typedef DirectoryIterator<AllResults
        ,BasicRecursiveEnumeration<RecursiveFakeFindXFileFunctions> > testType;
    testType begin(L"C:\\Windows\\*");
    testType end;

    BOOST_CHECK(begin->IsDirectory());
    BOOST_CHECK_EQUAL(begin->GetSize(), 0);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MySubDirectory");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MySubDirectory");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 1024);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows\\MySubDirectory");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile2.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MySubDirectory\\MyFile2.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 500);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"MyFile.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\MyFile.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsDirectory());
    BOOST_CHECK_EQUAL(begin->GetSize(), 0);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"Zach");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\Zach");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin->IsFile());
    BOOST_CHECK_EQUAL(begin->GetSize(), 1024);
    BOOST_CHECK_EQUAL(begin->GetFolderPath(), L"C:\\Windows\\Zach");
    BOOST_CHECK_EQUAL(begin->GetFileName(), L"ZachFile.txt");
    BOOST_CHECK_EQUAL(begin->GetFullFileName(), L"C:\\Windows\\Zach\\ZachFile.txt");
    BOOST_CHECK(begin != end);

    begin++;

    BOOST_CHECK(begin == end);
}

BOOST_AUTO_TEST_CASE( Exceptions )
{
    typedef DirectoryIterator<AllResults,BasicRecursiveEnumeration<ErroneousFindXFileFunctionFirst> >
        firstFailType;
    BOOST_CHECK_THROW(firstFailType(L"C:\\Windows\\*"), WindowsAPI::ErrorAccessDeniedException);
    typedef DirectoryIterator<AllResults,BasicRecursiveEnumeration<ErroneousFindXFileFunctionNext> >
        nextFailType;
    BOOST_CHECK_THROW(nextFailType(L"C:\\Windows\\*"), WindowsAPI::ErrorInvalidParameterException);
}

BOOST_AUTO_TEST_SUITE_END()

Gunmaker

doesn't boost already do this with the boost::filesystem library? – Hurrah

@Chris Kaminski: This iterator and boost's design goals are different. For this I wanted something that would be fast and incur little or no overhead over the straight FindXFile calls (This program spends a LOT of time on directory enumeration). Boost::Filesystem's goal is more for cross platform compatibility. For example, it includes it's own path parser and other associated things I don't want. Thus the reason for implementing this myself. – Gunmaker

Answer 1

Interestingly, I had the same idea some time ago. Here's what I wrote up:

#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <string>
#include <iterator>
#include <exception>

#ifndef DIR_ITERATOR_H_INC
#define DIR_ITERATOR_H_INC

class dir_iterator

#if (!defined(_MSC_VER)) || (_MSC_VER > 1200)
    : public std::iterator<std::input_iterator_tag, 
                            std::string, 
                            int, 
                            std::string *, 
                            std::string &> 
#endif
{ 
    mutable HANDLE it;
    std::string mask;
    std::string path;
    WIN32_FIND_DATA data;
    bool done;
    DWORD require;
    DWORD prohibit;
public:
    WIN32_FIND_DATA operator*() { 
        return data;
    }

dir_iterator(dir_iterator const &other) :
    it(other.it),
    mask(other.mask),
    path(other.path),
    data(other.data),
    done(other.done),
    require(other.require),
    prohibit(other.prohibit)
{
    // Transfer the handle instead of just copying it.
    other.it=INVALID_HANDLE_VALUE;
}

    dir_iterator(std::string const &s, 
        DWORD must = 0,
        DWORD cant = FILE_ATTRIBUTE_DIRECTORY)
        : mask(s),
        require(must),
        prohibit(cant & ~must),
        done(false),
        it(INVALID_HANDLE_VALUE) // To fix bug spotted by Billy ONeal.
    { 
        int pos;
        if (std::string::npos != (pos=mask.find_last_of("\\/"))) 
            path = std::string(mask, 0, pos+1);

        it = FindFirstFile(mask.c_str(), &data);
        if (it == INVALID_HANDLE_VALUE)
            throw std::invalid_argument("Directory Inaccessible");

        while (!(((data.dwFileAttributes & require) == require) &&
                ((data.dwFileAttributes & prohibit) ==  0)))
        {
            if (done = (FindNextFile(it, &data)==0))
                break;
        }
    }

    dir_iterator() : done(true) {}

    dir_iterator &operator++() {
        do { 
            if (done = (FindNextFile(it, &data)==0))
                break;
        } while (!(((data.dwFileAttributes & require) == require) &&
            (data.dwFileAttributes & prohibit) == 0));

        return *this;
    }

    bool operator!=(dir_iterator const &other) { 
        return done != other.done;
    }
    bool operator==(dir_iterator const &other) { 
        return done == other.done;
    }

    ~dir_iterator() { 
        // The rest of the bug fix -- only close handle if it's open.
        if (it!=INVALID_HANDLE_VALUE)
            FindClose(it);
    }
};

#endif

And a quick demonstration of it:

#include "dir_iterator.h"
#include <iostream>
#include <algorithm>

namespace std { 
    std::ostream &operator<<(std::ostream &os, WIN32_FIND_DATA const &d) { 
        return os << d.cFileName;
    }
}

int main() { 
    std::copy(dir_iterator("*"), dir_iterator(), 
        std::ostream_iterator<WIN32_FIND_DATA>(std::cout, "\n"));

    std::cout << "\nDirectories:\n";
    std::copy(dir_iterator("*", FILE_ATTRIBUTE_DIRECTORY), dir_iterator(), 
        std::ostream_iterator<WIN32_FIND_DATA>(std::cout, "\n"));

    return 0;
}

Ss

+1 -- though correct me if I'm wrong here -- doesn't that call FindClose() on garbage data when the default constructed iterator is destructed? – Gunmaker

@Billy:now that you mention it, yes, I think it does. I'll have to fix that. One of the benefits of posting code is getting a bug caught now and again -- thank you. – Ss

@Jerry Coffin: One more bug -- if the iterator is copied then FindClose() will be called twice on the same handle. Just don't call STL algorithms on this LOL!. Giving you the checkmark for the demonstration. – Gunmaker

@BillyONeal:at least in theory, you're right. So far, I've never seen a problem from it (or the other bug) though; I'm guessing that FindClose is smart enough to do nothing if what you pass it isn't a valid search handle. In theory the right answer would probably be an implementation object with a ref-counted wrapper. – Ss

@BillyONeal:Thinking about it a bit more, in this case ownership transfer semantics are probably fine -- I'll edit the code a bit to do that. – Ss

Yeah -- I tried the refcounted wrapper method but that broke because when any instance of a copied iterator had operator++ called, it was implicitly called on all instances of that iterator ( There's no way to duplicate a search handle :( ). – Gunmaker

@Jerry Coffin: I have edited my answer with what I eventually settled on in case you are interested. – Gunmaker

Answer 2

To solve the first problem, you can have end() return some sentinel value, then in your iterator's increment function, set the iterator equal to that sentinel value when it reaches the end of the container. This is effectively what the directory iterator in Boost.Filesystem does.

For the second problem, I'm not entirely familiar with the FindXFile APIs, but one option would be to return some struct that contains all the data you need or that has member functions to get each of the pieces of data you might want.

Grouch

Using a WIN32_FIND_DATA with .dwAttributes set to DWORD(-1) works quite well as a sentinel. Normally, FILE_ATTRIBUTE_NORMAL=0x0080 can't be combined with other flags like FILE_ATTRIBUTE_DIRECTORY==0x0010. – Macdermot

Answer 3

Matthew Wilson has written several articles on adapting directory enumeration to STL iterators. Unfortunately, the article that directly addresses the Windows APIs doesn't seem to be online anymore. However, I imagine the ideas that are discussed in some of his other articles are probably still very relevant, and he also provides an open source library (WinSTL - http://winstl.org/) with the Windows implementation.

• Wild-card Searches of UNIX Directories with Random-Access Iterators
• Reading UNIX Directories via STL-compliant Sequences

Also, I'm sure that the Boost::Filesystem source and documentation is a great source of ideas.

Areca

Answer 4

Unless you're doing it to learn, my answer would be: don't, it's already been done. The STLSoft libraries contain the winstl::basic_findfile_sequence class template that handles most/all use cases for searching a Windows directory, including wildcards, multi-part patterns (e.g. ".xls|.doc"), reparse points, and more. It's documented in great detail in Matthew Wilson's in-depth treatise on STL extension "Extended STL, volume 1: Collections and Iterators". The book's a major read, but it contains all the things you'll want to know (as well as many you may not) about writing STL extensions.

If you need recursive search, then consider the recls library (another of Wilson's), which also provides an STL interface, via the recls::search_sequence class. See here for a bunch of examples from a recent article about it.

Flam

That library doesn't support recursive searches. – Gunmaker

Ah, my bad. Have just updated my answer to include the recls library. – Flam

Answer 5

Try the recls library, discussed here. (It's also available for .NET, fwiw.)

Rasla

Answer 6

Boost.Filesystem has platform independent STL like iterators. Be aware though that the iterators can throw if a file is inaccessible.

Example (can be pimped using count_if and lambda):

namespace fs = boost::filesystem;

size_t nTotal = 0; 

try
{
   const fs::path pth("c:\\temp");

   //default construction yields past-the-end
   fs::recursive_directory_iterator itEnd;

   for (fs::recursive_directory_iterator it(pth); it != itEnd; ++it)
   {
      const fs::directory_entry& rEntry = *it;

      if (fs::is_regular_file(rEntry))
      {
         nTotal += fs::file_size(rEntry);
      }
   }
}
catch (const fs::filesystem_error& re)
{
   //when directoy or file is not found
}

Melda