FileStream very slow on application-cold start

Question

A very similar question has also been asked here on SO in case you are interested, but as we will see the accepted answer of that question is not always the case (and it's never the case for my application use-pattern).

The performance determining code consists of FileStream constructor (to open a file) and a SHA1 hash (the .Net framework implementation). The code is pretty much C# version of what was asked in the question I've linked to above.

Case 1: The Application is started either for the first time or Nth time, but with different target file set. The application is now told to compute the hash values on the files that were never accessed before.

• ~50ms
• 80% FileStream constructor
• 18% hash computation

Case 2: Application is now fully terminated, and started again, asked to compute hash on the same files:

• ~8ms
• 90% hash computation
• 8% FileStream constructor

Problem
My application is always in use Case 1. It will never be asked to re-compute a hash on a file that was already visited once.

So my rate-determining step is FileStream Constructor! Is there anything I can do to speed up this use case?

Thank you.

P.S. Stats were gathered using JetBrains profiler.

Siloxane

I'm seeing the exact same behavior. Using ReadAllBytes and computing the hash takes a short amount of time but can be terrible on memory (depending on the size of the file). So I tried to pass a FileStream to MD5's computehash($stream) and seeing completely unacceptable results. The result times are orders of magnitude in difference... – Thielen

@Alex K: From looking at the FileStream source, it looks like there are tons of preprocessor directives and, even in the most simple of paths, initialization appears to be pretty complex. You may want to specify which specific constructor you are having problems with and what your environment looks like and hopefully people (much) smarter than myself can help you. – Huguenot

Are you properly multithreaded? Yes, spindles are limited but when accessing a lot of small files one by one you may spend most of the time waiting for data, and you MAY speed that up a little (or more in SSD) by going multi threaded. Not "extreme" but 2-3 threads MAY help to reduce the downtime. – Mendoza

Need more information. How exactly are you invoking the constructor? What are your arguments, and what do they look like? Absolute path or relative path? Are you ngen'ing your program to reduce/remove JIT overhead? Are you running in a limited trust scenario, or does your app have full trust? Have you tried opening the file with FileOptions.SequentialScan? Have you tried enabling multicore jit via ProfileOptimization.StartProfile? – Resentful

@Resentful anything that takes 10 seconds is obviously not a function of those options but a bug, that I don't expect to get fixed. – Hallerson

@DanielDonnelly - You're responding to a question that has not been asked. You're not the poster, and the poster did not specify that they were having a problem with 10-second delays; theirs is a 50 ms delay. Please do not try to change the topic of another poster's question; ask your own instead. If you're having trouble asking your own due to low rep, I might suggest that you read the help center on how to ask productive questions. – Resentful

Answer 1

... but with different target file set.

Key phrase, your app will not be able to take advantage of the file system cache. Like it did in the second measurement. The directory info can't come from RAM because it wasn't read yet, the OS always has to fall back to the disk drive and that is slow.

Only better hardware can speed it up. 50 msec is about the standard amount of time needed for a spindle drive, 20 msec is about as low as such drives can go. Reader head seek time is the hard mechanical limit. That's easy to beat today, SSD is widely available and reasonably affordable. The only problem with it is that when you got used to it then you never move back :)

Digitalin

Answer 2

The file system and or disk controller will cache recently accessed files / sectors.

The rate-determining step is reading the file, not constructing a FileStream object, and it's completely normal that it will be significantly faster on the second run when data is in the cache.

Masterson

I don't believe this is the case. FileStream constructor does not read the entire file, the hash function calls in for that purpose. But it is the constructor that takes 80% of the time. – Siloxane

Answer 3

As stated earlier, the file system has its own caching mechanism which perturbates your measurement.

However, the FileStream constructor performs several tasks which, the first time are expensive and require accessing the file system (therefore something which might not be in the data cache). For explanatory reasons, you can take a look at the code, and see that the CompatibilitySwitches classes is used to detect sub feature usage. Together with this class, Reflection is heavily used both directly (to access the current assembly) and indirectly (for CAS protected sections, security link demands). The Reflection engine has its own cache, and requires accessing the file system when its own cache is empty.

It feels a little bit odd that the two measurements are so different. We currently have something similar on our machines equipped with an antivirus software configured with realtime protection. In this case, the antivirus software is in the middle and the cache is hit or missed the first time depending the implementation of such software.

The antivirus software might decide to aggressively check certain image files, like PNGs, due to known decode vulnerabilities. Such checks introduce additional slowdown and accounts the time in the outermost .NET class, i.e. the FileStream class.

Profiling using native symbols and/or with kernel debugging, should give you more insights.

Based on my experience, what you describe cannot be mitigated as there are multiple hidden layers out of our control. Depending on your usage, which is not perfectly clear to me right now, you might turn the application in a service, therefore you could serve all the subsequent requests faster. Alternative, you could batch multiple requests into one single call to achieve an amortized reduced cost.

Billyebilobate

Answer 4

Off track suggestion, but this is something that I have done a lot and got our analyses 30% - 70% faster:

Caching

---

Write another piece of code that will:

• iterate over all the files;
• compute the hash; and,
• store it in another index file.

Now, don't call a FileStream constructor to compute the hash when your application starts. Instead, open the (expectedly much) smaller index file and read the precomputed hash off it.

Further, if these files are log etc. files which are freshly created every time before your application starts, add code in the file creator to also update the index file with the hash of the newly created file.

This way your application can always read the hash from the index file only.

---

I concur with @HansPassant's suggestion of using SSDs to make your disk reads faster. This answer and his answer are complimentary. You can implement both to maximize the performance.

Kea

My issue is that it's taking 15 seconds to load a simple 8KB png. – Hallerson

@DanielDonnelly: 8KB sized data is by no means big enough to take 15 seconds to load. It may be that your hard-drive is being accessed by multiple programs. For that, try putting your data on another drive and see if the access time comes down to almost instantaneous. Also, is your drive on the network? – Kea

nope not on network. It's weird that every other program on my machine works except for one trying to load a small texture and that I coded in C# with Monogame. This caused me to stop the project and no longer use C# if I begin it again. The images that I need generated on the fly are LaTeX renderings of a small amount of text. Even the LaTeX renders quicker: 2 seconds, which is acceptable for my application under caching as you say, but 10-15 second wait is not useable. I've just started using monogame so my environment is 100% the latest version of everything. – Hallerson

@DanielDonnelly: Please edit the question and add complete code of yours in it... as an example case of the situation. This is extremely weird behavior. Could it be due any native library you are using? – Kea

They've blocked me from asking questions. Every time they unban me, I ask a perfect question, and then get banned again. See my history. So I cannot ask a question. Even though I'm 100% capable of creating a minimal working example and perfectly explaining it in English. – Hallerson

can I send you a minimal working example? I've tried 5 different snippets of code, but the commonality and the failure of each is in the FileStream ctor call. – Hallerson

@DanielDonnelly: While you may not be able to ask a new one, you may be able to edit this question and add your code. Is that possible? – Kea

@Kea - That's not a good idea. Daniel should not be modifying someone else's question to serve himself. If he can't ask a question due to low rep, after repeated interactions with SO, then that's a sign that he doesn't participate in good faith. – Resentful

Answer 5

You should try to use the native FILE_FLAG_SEQUENTIAL_SCAN, you will have to pinvoke CreateFile in order to get an handle and pass it to FileStream

Counterweight

Doesn't using the constructor that takes in a FileOptions do that to the stream already if you pass it FileOptions.SequentialScan? EDIT: yea, it does, looking through the reference source that enum parameter gets turned in to the dwFlagsAndAttributes of CreateFile, and the enum's value is 0x08000000 which is the same value as FILE_FLAG_SEQUENTIAL_SCAN – Cacao