57P01: terminating connection due to administrator command when running multiple tests in sequence

Question

I have a Postgres DB running in a Docker environment which is set up using FluentDocker in a Xunit test.

Xunit is configured to run tests serial.

When running a single test, everything works fine.

However, when adding another test (literally just copying the first test) and running both tests in one call using dotnet test, the second test always fails with 57P01: terminating connection due to administrator command.

I tried to close the DB connection, stop the containers after each test run etc. but the error remains and it always happens at the same line of code.

This is the code of the test:

[Fact]
public async Task ShouldProjectUserRegistration()
{
  var file = Path.Combine(
    Directory.GetCurrentDirectory(),
    (TemplateString)"Resources/docker-compose.yml"
  );

  var service = new Builder()
    .UseContainer()
    .UseCompose()
    .FromFile(file)
    .RemoveOrphans()
    .ForceRecreate()
    .WaitForPort(
      "database",
      "5432/tcp",
      30000 /*30s*/
    )
    .Build();
  var container = service.Start();

  var PgTestConnectionString =
    "PORT = 5432; HOST = localhost; TIMEOUT = 15; POOLING = True; MINPOOLSIZE = 1; MAXPOOLSIZE = 100; COMMANDTIMEOUT = 20; DATABASE = 'marten'; PASSWORD = '123456'; USER ID = 'marten'";
  using var store = DocumentStore.For(
    options =>
    {
      options.Connection(PgTestConnectionString);
      options.AutoCreateSchemaObjects = AutoCreate.All;
      options.Projections.SelfAggregate<User>(ProjectionLifecycle.Inline);
    }
  );

  var id = Guid.NewGuid();
  var username = "[email protected]";
  var userRegistered = new UserRegistered(
    id,
    username
  );

  await using var session = store.OpenSession();
  session.Events.StartStream(
    id,
    userRegistered
  );

  await session.SaveChangesAsync();


  var user = session.Load<User>(id);
  await session.Connection?.CloseAsync();
  service.Stop();
  Assert.Equal(
    username,
    user?.Username
  );

This is the docker-compose.yml:

version: "3"

services:
  database:
    image: library/postgres:14
    environment:
      POSTGRES_USER: 'marten'
      POSTGRES_PASSWORD: '123456'
      POSTGRES_DB: 'marten'
    ports:
      - "5432:5432"

The exception:

Npgsql.PostgresException
57P01: terminating connection due to administrator command
   at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|213_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
   at Weasel.Core.CommandBuilderBase`6.ExecuteReaderAsync(TConnection conn, CancellationToken cancellation, TTransaction tx)
   at Weasel.Core.SchemaMigration.Determine(DbConnection conn, ISchemaObject[] schemaObjects)
   at Weasel.Core.Migrations.DatabaseBase`1.executeMigration(ISchemaObject[] schemaObjects, CancellationToken token)
   at Weasel.Core.Migrations.DatabaseBase`1.executeMigration(ISchemaObject[] schemaObjects, CancellationToken token)
   at Weasel.Core.Migrations.DatabaseBase`1.generateOrUpdateFeature(Type featureType, IFeatureSchema feature, CancellationToken token)
   at Weasel.Core.Migrations.DatabaseBase`1.ensureStorageExists(IList`1 types, Type featureType, CancellationToken token)
   at Weasel.Core.Migrations.DatabaseBase`1.ensureStorageExists(IList`1 types, Type featureType, CancellationToken token)
   at Weasel.Core.Migrations.DatabaseBase`1.ensureStorageExists(IList`1 types, Type featureType, CancellationToken token)
   at Marten.Events.EventGraph.ProcessEventsAsync(DocumentSessionBase session, CancellationToken token)
   at Marten.Internal.Sessions.DocumentSessionBase.SaveChangesAsync(CancellationToken token)
   at Marten.Internal.Sessions.DocumentSessionBase.SaveChangesAsync(CancellationToken token)
   at MartenFluentDockerNpsql57P01Repro.Tests.UserProjectionTests.ShouldProjectUserRegistrationSecond() in /Users/alexzeitler/src/MartenFluentDockerNpsql57P01Repro/MartenFluentDockerNpsql57P01Repro.Tests/UserProjectionTests.cs:line 120
   at MartenFluentDockerNpsql57P01Repro.Tests.UserProjectionTests.ShouldProjectUserRegistrationSecond() in /Users/alexzeitler/src/MartenFluentDockerNpsql57P01Repro/MartenFluentDockerNpsql57P01Repro.Tests/UserProjectionTests.cs:line 126
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

I created a repro which can be found on GitHub.

Giamo

Answer 1

You have options in your connection string that generate a connection pool: POOLING = True; MINPOOLSIZE = 1; MAXPOOLSIZE = 100;

After you generate this pool and run your test logic, you are only closing the current connection in the pool when you call CloseAsync.

Because the pool has a minimum size of 1, the pool is going to immediately add a new connection after you close the current connection.

It looks like shutting down your database is then triggering the error on the new connection that is generated because the connection. So to solve the problem and destroy all connection related resources, you need to use DisposeAsync.

await session.Connection?.DisposeAsync();

You should likely also set POOLING = False since your test appears to only require a single connection.

The session appears to be properly closing/disposing the connection via using since the connection is created/managed by Marten. The issue here is that this will NOT properly clean up the connection pool that was generated and will simply return the connection to the pool where it will remain idle. Thus, as long as PgTestConnectionString is the same in ShouldProjectUserRegistration and ShouldProjectUserRegistrationSecond, the pool will try to provide the same connection you used in ShouldProjectUserRegistration. The problem is that, while the connection sits idle in the pool, you terminate the database that it is connected to. Consequently, when you fetch this connection from the pool and try to perform an operation on it, you get the error message you described.

This situation can be prevented in a few different ways:

• Set POOLING = False; in both test methods as you do not need a connection pool since you are destroying the database in between tests.
• If you want to keep POOLING = True;, you can destroy the pool at the end of each Fact with NpgsqlConnection.ClearPool(session.Connection); or NpgsqlConnection.ClearAllPools();

Goeger

When disabling pooling, it works. Having pooling enabled and using await session.Connection?.DisposeAsync(); ends with the initial error. – Giamo

@AlexanderZeitler try adding a block to your using declaration csharp await using (var session = store.OpenSession()) { // everything before service.stop() } service.Stop(); and then remove the line manually disposing/closing session.Connection. From the source code it looks like the session should call Dispose() on the connection itself. Let me know if that solves your problem and I'll update my answer. – Goeger

still the same error. – Giamo

@AlexanderZeitler I ran some tests on your repro code, did some more research and updated my answer. – Goeger

Thanks - NpgsqlConnection.ClearAllPools(); did the trick when POOLING = True. – Giamo

Answer 2

For me, the problem was that I was opening the same hardcoded port for multiple test containers via .WithPortBinding(“5433”, “5432”) in my PostgreSqlBuilder.

Each container (actually each test class) it ran terminated the previous one to get the port binding, resulting in a terminating connection due to administrator command error

Ginger