Filter Serilog logs to different sinks depending on context source?

Question

I have a .NET Core 2.0 application in which I successfully use Serilog for logging. Now, I would like to log some database performance statistics to a separate sink (they are not for debugging, which basically is the purpose of all other logging in the application, so I would like to keep them separate) and figured this could be accomplished by creating the DB statistics logger with Log.ForContext<MyClass>().

I do not know how I am supposed to configure Serilog using my appsettings.json to log my "debug logs" to one sink and my DB statistics log to another? I am hoping it is possible to do something like:

"Serilog": {
  "WriteTo": [
    {
      "Name": "RollingFile",
      "pathFormat": "logs/Log-{Date}.log",
      "Filter": {
        "ByExcluding": "FromSource(MyClass)"
      }
    },
    {
      "Name": "RollingFile",
      "pathFormat": "logs/DBStat-{Date}.log",
      "Filter": {
          "ByIncludingOnly": "FromSource(MyClass)"
      }
    }
  ]
}

The "Filter" parts of the configuration is pure guesswork on my part. Is this possible using my configuration filer or do I need to do this in code in my Startup.cs file?

EDIT: I have got it working using the C# API but would still like to figure it out using JSON config:

Log.Logger = new LoggerConfiguration()
            .WriteTo.Logger(lc => lc
                .Filter.ByExcluding(Matching.FromSource<MyClass>())
                .WriteTo.LiterateConsole())
            .WriteTo.Logger(lc => lc
                .Filter.ByExcluding(Matching.FromSource<MyClass>())
                .WriteTo.RollingFile("logs/DebugLog-{Date}.log"))
            .WriteTo.Logger(lc => lc
                .Filter.ByIncludingOnly(Matching.FromSource<MyClass>())
                .WriteTo.RollingFile("logs/DBStats-{Date}.log", outputTemplate: "{Message}{NewLine}"))
            .CreateLogger();

Screenplay

Short on time to write a proper answer; this is rather ugly in JSON, but possible. You'll need Serilog.Filters.Expressions, and a couple of WriteTo.Logger blocks as shown in github.com/skomis-mm/serilog-settings-configuration/blob/…. The filters need to be configured at the level of each sub-logger. HTH! – Amphitryon

@NicholasBlumhardt Thanks for the tip. The only way I can get the "Filter" to actually do something is to put it in the outmost level, i.e as a direction child of "Serilog". At that level it affects all sinks, which defeats the purpose of this exercise. If I put "Filter" either as a child of the Logger sinks or the subloggers (the "RollingFile"s) it has no effect what so ever. Any further tips? – Screenplay

Hmmm, unsure what would be happening; have you got it working using the C# API directly? Might be a good first step. – Amphitryon

@NicholasBlumhardt Got it working using the API. Check out updated question. – Screenplay

@NicholasBlumhardt, I have the same question about describing different sinks with different context based rules in appsettings.json. Example github.com/skomis-mm/serilog-settings-configuration/blob/… doesn’t apply different filters for different subloggers. Can you suggest some other example? Or is it better to raise an issue on Serilog filters GitHub ? – Crosspollination

Hi @MichaelFreidgeim - a ticket on the GitHub repo or a new post here should track down an answer. Cheers! – Amphitryon

Did you ever figure out the JSON config? – Quincuncial

Answer 1

I completed this work today, and thought that I'd provide a proper answer since it took me quite a few posts, issues and other pages to work through to get this sorted out.

It's useful to have all the logs, but I also wanted to log only my API code separately, and omit the Microsoft. namespace logs. The JSON config to do that looks like this:

  "Serilog": {
    "Using": [ "Serilog.Sinks.File" ],
    "MinimumLevel": "Debug",
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "/var/logs/system.log",
          ... //other unrelated file config
        }
      },
      {
        "Name": "Logger",
        "Args": {
          "configureLogger": {
            "WriteTo": [
              {
                "Name": "File",
                "Args": {
                  "path": "/var/logs/api.log",
                  ... //other unrelated file config
                }
              }
            ],
            "Filter": [
              {
                "Name": "ByExcluding",
                "Args": {
                  "expression": "StartsWith(SourceContext, 'Microsoft.')"
                }
              }
            ]
          }
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ]
    ... //Destructure and other config
  }

The top-level WriteTo is the first simple, global, sink. All log events write to this. If you add a Filter section on the same level as this, it will affect all configured WriteTo elements.

Then I configure another WriteTo as a Logger (not File), but the Args for this looks different and has a configureLogger element which serves the same purpose as Serilog on the top level, that is to say, it is the top level of the sub-logger. This means that you can easily split out the config for this into a separate file and add it additionally in the config builder (see bottom).

From here, this sub-logger works the same way: You can configure multiple WriteTos, and the Filter element on this level will affect only this sub-logger.

Simply add more "Name": "Logger" elements to the top level WriteTo section and setup filters for each one separately.

Note It is also important to note that, even though you are doing all this in config and not referencing a single bit of the Serilog.Expressions package in your code, you still have to add the NuGet reference to that package. It doesn't work without the package reference.

About splitting the config:

If I have to add more loggers, I would definitely split out the different loggers into separate files for clarity, e.g.

appsettings.json:

  "Serilog": {
    "Using": [ "Serilog.Sinks.File" ],
    "MinimumLevel": "Error",
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "/var/logs/system.log",
          ...
        }
      },
      {
        "Name": "Logger",
        "Args": {
          "configureLogger": {} // leave this empty
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ],
    ...

apilogger.json:

{
  "Serilog:WriteTo:1:Args:configureLogger": {   //notice this key
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "/var/logs/api_separateFile.log",
          ...
        }
      }
    ],
    "Filter": [
      {
        "Name": "ByExcluding",
        "Args": {
          "expression": "StartsWith(SourceContext, 'Microsoft.')"
        }
      }
    ]
  }
}

And then adjust my IWebHost builder to include the additional config:

    WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            config.AddJsonFile("apilogger.json", optional: false, reloadOnChange: false);
        })
        .UseStartup<Startup>();

This way it is easier to understand, read and maintain.

Erminois

This didn't quite solve my issues but has been the most complete, nearly what I was looking for, response I could find with my google-fu. Thank you! – Planksheer

Must the second one be named Logger? I couldnt get anything to work when i named it Hangfire – Steamship

Yes. Hangfire is a specific sink, much like File or something like AppInsights. Logger is like a reserved word in the config for setting up the sub-logger, within which you specify further specific sinks. Perhaps think of Logger here as being the equivalent of ILoggerFactory in the code. – Erminois

Isn't this solution working differently than the original question? In the question is expressed that first 2 sinks would be excluding something that only the 3rd sink would log. And the answer seems to indicate that the 1st sink is going to swallow everything that comes, while the 2nd one will do the same but excluding everthing that starts with "Microsoft." isn't it?? Or was this also intended on the original question? – Noddy

It's not possible using appsettings.json, as far as I know, to get it to to filter on the top level logger and not filter on the sub level logger. – Erminois

JFYI: Nowadays the Serilog.Filters.Expressions is marked as deprecated and they suggest to use Serilog.Expressions. – Gotten

Answer 2

Writing only entityframework log to a file is done below but we need to install Serilog.Filters.Expressions nuget package

"Serilog": {
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning",
        "Microsoft.EntityFrameworkCore": "Information"
      }
    },
    "WriteTo": [
      {
        "Name": "Logger",
        "Args": {
          "configureLogger": {
            "Filter": [
              {
                "Name": "ByIncludingOnly",
                "Args": {
                  "expression": "StartsWith(SourceContext, 'Microsoft.EntityFrameworkCore')"
                }
              }
            ],
            "WriteTo": [
              {
                "Name": "File",
                "Args": {
                  "path": "C:\\LOGS\\TestService_EF_.json",
                  "outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] {Message:lj}{NewLine}{Exception}",
                  "rollingInterval": "Day",
                  "retainedFileCountLimit": 7
                }
              }
            ]
          }
        }
      }
    ]
  }

Gallice

Answer 3

I had to do a similar thing, but in code, not JSON. Using a logger as a sink, as described at the bottom of https://github.com/serilog/serilog/wiki/Configuration-Basics did the trick.

In my case I wanted to log everything to a file and to the console, except that messages from a specific source should go only to the file:

private static bool SourceContextEquals(LogEvent logEvent, Type sourceContext)
    => logEvent.Properties.GetValueOrDefault("SourceContext") is ScalarValue sv && sv.Value?.ToString() == sourceContext.FullName;

private static ILogger CreateLogger() =>
    new LoggerConfiguration()
        .WriteTo.File("mylog.log")
        .WriteTo.Logger(lc =>
            lc.Filter.ByExcluding(le => SourceContextEquals(le, typeof(TypeThatShouldLogOnlyToFile)))
           .WriteTo.Console()
        )
        .CreateLogger();

Christ

but if you do it via code then you have to change the code every time you want to make a logging change.. – Needs

Answer 4

Serilog has added the ability to set the option within C#, as well. For instance:

.WriteTo.Console(
    restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Information)

Mayberry