Unable to access grpc application behind nginx-ingress and nlb

Question

I have a grpc application installed in AWS EKS. The grpc endpoint works successfully with port forwarding:

grpcurl -plaintext -protoset-out=reflection.protoset localhost:8080 list 
                                 
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection

I've installed nginx-ingress via the helm chart, and set the following configuration:

USER-SUPPLIED VALUES:
controller:
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <cert-arn>
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
    targetPorts:
      https: 443

This has successfully created an NLB with listeners, one for HTTP and one for TLS with the cert attached.

I've then created an ingress like so:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  generation: 7
  labels:
    name: <app>
  name: <svc-name>
  namespace: <ns>
spec:
  ingressClassName: nginx
  rules:
  - host: <app>.development.<domain>
    http:
      paths:
      - backend:
          service:
            name: <svc>
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - <app>.development.<domain>
    secretName: dev-tls
status:
  loadBalancer:
    ingress:
    - hostname: <loadbalancer>

You'll notice I'm also terminating TLS with cert-manager/lets-encrypt at the ingress as well, as I believe that's required.

External DNS is correctly creating a record in route53 and the certificate is being fetched from let's encrypt. However, no matter what I do, grpc curl is not working and throws "context deadline exceed"

grpcurl -protoset-out=reflection.protoset <app>.development.<domain>:443 list
Failed to dial target host "<app>.development.<domain>:443": context deadline exceeded

What am I missing here? Other things I've tried:

Setting the ALPN policy on the TLS listener to HTTP/2 only Using an ALB ingress controller (yes, really!) same issue

EDIT: Have also verified this isn't an application problem with a sample grpc app

EDIT 2: Using a service Type=LoadBalancer and bypassing nginx-ingress entirely works, so this definitely seems like an issue with nginx-ingress

Gallimaufry

Do you have non-grpc services in the cluster, are those available? can you provide logfiles from the nginx ingress controller pod? Are you sure your service is configured for port 80? – Jahn

I think the following are needed: 1. Make sure that the dns recocrd is set on the nlb, since it should be the only place wich has internet facing 2. Either your nlb or nginx-ingress should do ssl termination. I don't think that the nlb can do forwarding to a service which listen with tls (and if it supports it, it probably requires configuration) – Maurene

@Jahn yes - non GRPC services work fine – Gallimaufry

Did you notice the remainder of my questions? – Jahn

> Do you have non-grpc services in the cluster, are those available? Yes > Are you sure your service is configured for port 80 Yes I don't have the logs available right now, but the long and short of it is that nginx pod would have log message very similar to this issue github.com/kubernetes/ingress-nginx/issues/7872 – Gallimaufry

It's worth noting that while the errors are the same, TLS configuration does not seem to be the problem and use-http2 is not set to false – Gallimaufry

I know it has been a long time since you first posted this, but im pretty sure your problem was related to the ALPN policy in the LoadBalancer. By adding this: service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Optional - you allow http2 connections, because the default is None, so it would only accept http1 connections. – Chemical