Unable to connect to Azure Cosmos Db Account using Microsoft.EntityFrameworkCore.Cosmos - Response status code

Question

The CosmosDb provider is sending this message:

“Response status code does not indicate success: 503 Substatus: 0 Reason: (The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server.”

In my tests, it works (.net core 3.1):

Task.Run(async () =>
        {
            var endpoint = “test”;
            var masterKey = “test”;
            using (var client = new DocumentClient(new Uri(endpoint), masterKey))
            {
                //Insert new Document  
                Console.WriteLine("\r\n>>>>>>>>>>>>>>>> Creating Document <<<<<<<<<<<<<<<<<<<");
                dynamic candidato = new
                {
                    Id = 1,
                    Nome = "Test"
                };

                var document1 = await client.CreateDocumentAsync(
                    UriFactory.CreateDocumentCollectionUri("Test", "Test"),
                    candidato);

                Console.ReadKey();
            }

        }).Wait();

It does not:

            Task.Run(async () =>
            {
                using (var context = new StudentsDbContext())
                {
                    context.Add(new FamilyContainer(2, "Test"));
                    await context.SaveChangesAsync();
                }

            }).Wait();

public class FamilyContainer
{
    public int Id { get; set; }
    public string Nome { get; set; }

    public FamilyContainer(int id, string nome)
    {
        Id = id;
        Nome = nome;
    }

}

public class StudentsDbContext : DbContext
{
    public DbSet<FamilyContainer> FamilyContainer { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.UseCosmos(
           "test",
           "test",
           "FamilyDatabase",
           options =>
           { }
         );
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<FamilyContainer>(x =>
        {
            x.ToContainer("FamilyContainer");
        });
    }
}

Packages

Can anyone help me? Thanks

fail: Microsoft.EntityFrameworkCore.Update[10000] An exception occurred in the database while saving changes for context type '...'. Microsoft.EntityFrameworkCore.Storage.RetryLimitExceededException: Maximum number of retries (6) exceeded while executing database operations with 'CosmosExecutionStrategy'. See inner exception for the most recent failure. ---> Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 503 Substatus: 0 Reason: (Microsoft.Azure.Documents.ServiceUnavailableException: Service is currently unavailable.ActivityId: 07fbf539-0d44-4e5a-89d0-cd46838ee605, {"RequestStartTimeUtc":"2020-02-21T16:34:09.1834993Z","RequestEndTimeUtc":"2020-02-21T16:34:41.3484203Z","RequestLatency":"00:00:32.1649210","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[{"ResponseTime":"2020-02-21T16:34:11.5964152Z","ResourceType":2,"OperationType":0,"StoreResult":"StorePhysicalAddress: rntbd:.../, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: True, StatusCode: 410, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused [0x0000274D]... --- End of inner exception stack trace ---

Aubrey

Please add the entire Exception stack trace and information – Led

Thanks @MatiasQuaranta, now i put the exeption. – Aubrey

Answer 1

TransportException: A client transport error occurred: Failed to connect to the remote endpoint. (Time: 2020-02-21T16:34:11.5298608Z, activity ID: 07fbf539-0d44-4e5a-89d0-cd46838ee605, error code: ConnectFailed [0x0005], base error: socket error ConnectionRefused

This means that the Connection was refused.

• Either your Cosmos DB account has Firewall/VPN enabled and the application is not able to establish a connection due not not being in a whitelisted IP/Network : Try checking your account configuration.
• The environment you are executing the code is restricting connections (some corporate Firewall or network might be blocking port ranges): Try running the app in a different network, or use GatewayMode. If that works, then this is related to the network.
• The machine might be running low on sockets or high on CPU.

Led

Thanks Matias, the problem was the Company Firewall. The Infraestructure in my company liberated some IP range and worked fine. – Aubrey

@LuisAugustoBarbosa Glad to hear it, please mark the answers to help others that run into the same issue. – Led

Nice, But Gateway mode is little slow, check : learn.microsoft.com/en-us/azure/cosmos-db/… – Virgievirgil

@ShaijuT yes, Gateway is not meant for performance as it adds an extra network hop, but there are scenarios where it might be required. For example, companies that cannot open all the required ports for TCP. My mention of Gateway was just in case it fit the scenario, it was just one of the potential options. You can also use TCP and open all the documented ports. – Led

Answer 2

I was facing same issue.

What worked for me is changing ConnectionMode to ConnectionMode.Gateway while initializing CosmosClient like :

var options = new CosmosClientOptions() { ConnectionMode = ConnectionMode.Gateway };
var client = new CosmosClient(endpoint, key, options); 

For more details on refer :

https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.cosmosclientoptions?view=azure-dotnet

https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.connectionmode?view=azure-dotnet

Messick

in my case I was running the software behind proxy, ConnectionMode = ConnectionMode.Gateway did the trick 😊 – Norenenorfleet

Answer 3

It is because Entity framework has a default connection mode of Direct. It worked for me after overriding it to Gateway.

    {
        optionsBuilder.UseCosmos(
           "test",
           "test",
           "FamilyDatabase",
           options =>
           { options.ConnectionMode(ConnectionMode.Gateway); }
         );
    }

Gherlein

Answer 4

My RCA for this is: Cosmos Partitions where served by individual processes on CosmosDB, each partition serving process has it's own TCP port. When client connects to 443 (Using TCP Direct Mode), CosmosDB Proxy sends partition ports back to client so that client can talk to server-partitions in parallel. Partition ports are random (11000 upwards afaik). Normal company firewall would allow outbound 443 (connection to cosmos works) but blocks the outbound random ports. So at the end, access fails. Workarounds:

1. Open firewall
2. Use Gateway Mode. This uses https/443 only by forwarding internally instead of redirecting to other ports.

Ocelot

"RCA" = Root Cause Analysis I think. – Quiles

Answer 5

I just want to add this because it wasted a lot of my time. The following code would instantly die with an error message that led me to this S.O. post:

var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync(dataitem, pk);

I disbelieved the error message because everything else has worked, upsert, read, etc. After messing with it for a while, I noticed the documentation shows a template type for CreateItemAsync.

var container = _client.Client.GetContainer(_databaseName, containername);
var result = await container.CreateItemAsync<T>(dataitem, pk);

Changing the code to that fixed it (inside of a templated function).

I wanted to add: if I had been catching exceptions, I would have gotten to the meat of the problem much sooner. The library I am working with is not meant to catch exceptions, they are handled by the layer above it.

Flessel