Asp.Net Core - The configured user limit (128) on the number of inotify instances has been reached
Asked Answered
F

8

49

the asp.net core MVC report error after certain times of query to the mysql database (on Ubuntu 14.04/16.04) with the following message: "The configured user limit (128) on the number of inotify instances has been reached." It can be identified that the error was raised because of the controller opened too many files and exceeded the limits of iNotify setting (in the /proc/sys/fs/inotify/max_user_instances). But I was just baffled when the ASP.NET opened files on each http request and why it doesn't close the file properly? Any one encounted this issue too? Remarks: I was using Mysql.data.core and mysql.data.entityframeworkcore provider.

    private static string classiferstring = "sports,outdoor,startup,pets,child,adult,elderly";

    [AllowAnonymous]
    [HttpGet]
    public async Task<object> Classify([FromQuery] string classifyword)
    {
        string[] classifers = classiferstring.Split(',');
        if (!classifers.Contains(classifyword))
        {
            return new
            {
                status = 0,
                info = "WrongClassifier",
                Data = ""
            };
        }

        try
        {
            var predata = await (from d in _context.descriptor
                              join a in _context.combination on d.ID equals a.ID
                              select new ProductsVM
                              {
                                  CREATETIME = a.CREATETIME,
                                  ID = a.ID,
                                  COMPANY = a.COMPANY,
                                  NAME = a.NAME,
                                  PRICE = a.PRICE,
                                  TYPE = a.TYPE,
                                  HEADPHOTO = a.HEADPHOTO,
                                  REMARK = a.REMARK,

                                  Tags = d.Tags,
                                  Classifier = d.Classifier,
                                  OriginName = d.OriginName,
                                  Briefing = d.Briefing
                              }).ToListAsync();
            var data = (from x in predata
                          where x.Classifier.Contains(classifyword.ToLower())
                          select x).ToList();

            if(predata.Count<=0)
            {
                return new
                {
                    status = 2,
                    info = "NoResult",
                    Data = ""
                };
            }else
            {
                return new
                {
                    status = 1,
                    info = "Success",
                    Data = data
                };
            };
        }
        catch(Exception e)
        {
            return new
            {
                status = 0,
                info = "Error",
                Data = e.Message
            };
        }
    }

Please notice the exception was raised only in the try/catch code block instead of immediately after the action is invokded.

Many thanks if anyone has some clue to resolve this problem.

Fluecure answered 18/4, 2017 at 10:15 Comment(4)
You failed to mention which provider you are using. Oracle's MySQL Provider is pretty shitty and the latest version completely broken if you do more than 1 query (it closes connection after first query). Likely there is something horribly broken with the Oracle provider so that it doesn't correctly recycle connections. My advice: throw away Oracle's provider and use some otherHasa
Thanks for suggestion. Unfortunately I have to continue on MySQL provider for some reason. It's worse since I have to maintain connection simultaneously with a Java project to the database... Meanwhile I noticed that ASP.NET core might use lots of inotify instances for UseStaticFiles that under wwwroot. I use 'lsof -p pid |wc -l' command and found the dotnet process used hundred instances.Fluecure
It turns out it might not be a problem of mysql provider but a infrastructural bug of dotnet... The mysql provider was changed to Dapper and it works well untill the bug was reported again after several dotnet restarts... I will try to report this issue to Microsoft.Fluecure
I was talking about the Oracle's MySQL Provider for EntityFramework Core (since your question is tagged with efcore). Dapper doesn't use EF Core, just the underlying MySqlConnection and except for that, is agnostic about any database. i.e. pomello mysql driver doesn't have these issues with connection being closed after each query, which clearly indicates the bad quality of Oracles MySQL Provider for EF CoreHasa
B
5

@dmitry-pavlov is wrong, you can use the default builder as the last builder added wins - but doing so makes appsettings.json settings override all other config sources, env and command line. See https://github.com/dotnet/AspNetCore.Docs/pull/22391 where this will soon be officially documented. If you don't care about appsettings.json reloads, set it to false

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            config.AddJsonFile("appsettings.json",
                optional: true,
                reloadOnChange: false);
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });
}
Butterfly answered 21/5, 2021 at 23:54 Comment(0)
L
82

The best solution I found so far is to increase the fs.inotify.max_user_instances in /etc/sysctl.conf by running this command:

echo fs.inotify.max_user_instances=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Source: https://github.com/dotnet/aspnetcore/issues/8449#issuecomment-512275929

Labia answered 17/3, 2020 at 18:10 Comment(1)
RUN echo fs.inotify.max_user_instances=524288 >> /etc/sysctl.confAmbrosane
M
43

For containerized environments (where config won't change for a lifetime of a container) or on build servers, the way to mitigate the issue is to set environment variable

DOTNET_HOSTBUILDER__RELOADCONFIGONCHANGE=false

This allows to continue using Host.CreateDefaultBuilder and prevents the service from creating unnecessary inotify instanced.

Meridethmeridian answered 15/4, 2021 at 15:17 Comment(4)
Note DOTNET_HOSTBUILDER__RELOADCONFIGONCHANGE is for Host.CreateDefaultBuilder, not the legacy WebHost.CreateDefaultBuilder.Nihility
Thanks, I've updated my answer, not sure how I missed as generic host is what I use.Meridethmeridian
Thanks! I was really left scratching my head on this one, but your answer solved it for me. It would be great if you had a header in the beginning, highlighting that this is for containerized environments. It was really easy to miss that part when I first scrolled through.Shocker
@AndreasForslöw I'm glad that it helped and thanks for the heads up! I've updated the answer.Meridethmeridian
J
19

The reason why error happens is reloadOnChange cause the issue while accessing appSetting.json files.

the configured user limit (128) on the number of inotify instances has been reached

Set reloadOnChange to false:

.AddJsonFile($"appsettings.json", optional: true, reloadOnChange: false);

NOTE: if you also use the default WebHost.CreateDefaultBuilder, be aware that inside it reloadOnChange is also set to true.

So probably would be more safe to just configure your host from scratch - this is an example how to do that (a copy of Microsoft WebHost.CreateDefaultBuilder but without FileWatcher dependency and IISIntegration as you don't need IIS on Ubuntu).

Judaic answered 11/6, 2019 at 15:31 Comment(0)
N
5

I just had to close some visual studio code windows and especially files open in the visual studio code editor.

Nip answered 29/4, 2021 at 22:2 Comment(3)
This is the correct answer in case your problem is not reloading settings.jsonBialy
I had to also reboot the machine.Predikant
Using PopOS and Rider: restarting Rider seem to have released the locks.Fatherhood
B
5

@dmitry-pavlov is wrong, you can use the default builder as the last builder added wins - but doing so makes appsettings.json settings override all other config sources, env and command line. See https://github.com/dotnet/AspNetCore.Docs/pull/22391 where this will soon be officially documented. If you don't care about appsettings.json reloads, set it to false

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            config.AddJsonFile("appsettings.json",
                optional: true,
                reloadOnChange: false);
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        });
}
Butterfly answered 21/5, 2021 at 23:54 Comment(0)
L
5

The way I have solved it is by setting the environment variable DOTNET_USE_POLLING_FILE_WATCHER to true like so:

export DOTNET_USE_POLLING_FILE_WATCHER=1

This has been explained by Microsoft in here and here.

Locarno answered 24/2, 2024 at 12:46 Comment(0)
K
1

I recently saw this problem on dot.net 5.0. In my case it was caused by using the bundle tag helper found here: https://github.com/meziantou/Meziantou.AspNetCore.BundleTagHelpers . It seems that tag helpers which use asp-append-version have been known to cause this issue, see here https://github.com/dotnet/runtime/issues/27272#issuecomment-525007303 .

Kristinakristine answered 28/4, 2021 at 19:28 Comment(0)
S
1

I had this problem with applications running in Docker containers via SWARM. On the main machine, the system max_user_instances was set to 128 cat /proc/sys/fs/inotify/max_user_instances .

Two solutions were useful for me:

  1. echo 256 | sudo tee /proc/sys/fs/inotify/max_user_instances
  2. Setting DOTNET_USE_POLLING_FILE_WATCHER=1 in the environment.
Stodder answered 4/7, 2024 at 7:20 Comment(0)

© 2022 - 2025 — McMap. All rights reserved.