After some investigation I have evidence that the bad performance had to do with files owned by a user from a different domain. Specifically, I arrived at the following conclusions:
- I'm working in a corporate environment with several domains and thousands of users.
- Due to organizational changes each user is, probably only during a transition phase, kept in two domains, his or her primary domain as well as a second domain. When changing object ownership through the Windows GUI each user appears twice and one must go to the extended user selection to identify the one assigned to a specific domain.
- cygwin with acl enabled displays the "other domain" file user as "<domain>+<username>". The primary domain self is just "<username>". Cygwin without acl displays just "<username>" in both cases. That can be fairly confusing because file permission and ownership as recognized by cygwin would indicate write permission, while the user factually does not have that.
- Files belonging to the "other domain" self are writable by my "this domain" self, so the domain assignment is largely transparent.
- A large source tree from our version control system (which was also mirrored in a git repo) had thousands of files owned by the "other domain self". That seemed to cause the slow file operations. Changing ownership to the "primary domain self" fixed the speed issue, both for git and for other file access.
I must assume that obtaining file permissions for users in other domains is slow, and for some reason not cached (it was always the same user).
The rest of the article below is what I originally posted. I let it stand.
For me (working in a large company with multiple, geographically distributed Windows domains) the culprit is that cygwin uses Windows acl per default. Consider this request for all known users in the Domain:
$ time (mkpasswd -D | wc -l)
45183
real 27m55,340s
user 0m1,637s
sys 0m0,123s
The fix (1)(2) was a simple matter of mounting the NTFS file systems with noacl, i.e. my /etc/fstab contains the line
none / cygdrive binary,posix=0,user,noacl 0 0
(at the same time eliminating the annoying cygdrive prefix).
I cannot help but imagine that cygwin/msys (same behavior there, except that the Windows git installation mounts noacl by default, probably for this reason) performs a domain server query for every file it touches and does not cache the results.
The change was introduced some time around 2015 with cygwin 2.4 or 2.5. From the release notes for 2.4:
To accommodate standard Windows ACLs, the POSIX permissions of the owner and all other users in the ACL are computed using the Windows AuthZ API. This may slow down the computation of POSIX permissions noticably in some circumstances [...] (emphasis by me).
The noacl option reduced the time to launch BeyondCompare (or echo a string, for that matter) from 25 seconds to 1. It is completely unintelligible why a simple git diff on the same file is very fast even with acl since I would naively assume that the required information and thus the required FS actions are identical.
I'll check out the cygserver now which may improve things by caching.
Update: cygserver does not improve the situation, unfortunately.
(1) The fix for git. mkpasswd is not affected.
(2) I have not understood and tested the impact on file permissions and ownership with respect to git (and ClearCase views which we also access through cygwin). My gut feeling is that one wants to stay true to Windows semantics as closely as possible (meaning that noacl may run into problems).
(3) The cygwin documentation discusses scenarios in which the query results are not cached. One consists of a sequence of cygwin processes which are not spawned from a common cygwin ancestor (like a bash) but from a windows program like cmd. I must assume that Windows provides a caching mechanism for native programs, or a Windows system would be unusable in this corporate environment. For some reason cygwin does not use it.
