HTTP2 PING frames over AWS ALB (gRPC keepalive ping)

Question

I'm using AWS Application Load Balancer (ALB) to expose the ASP.NET Core gRPC services. The services are running in Fargate containers and expose unsecured HTTP ports. ALB terminates the outer TLS connection and forwards the unencrypted traffic to a target group based on the route. The gRPC application has several client streaming endpoints and the client can pause the streaming for several minutes. I know that there are HTTP2 PING frames, which can be used in such cases, to keep alive the connection that has no data transmission for some amount of time.

The gRPC server is configured to send HTTP2 pings every 20 seconds for keeping the connection alive. I tested this approach and it works, the ping frames went from the server and were acknowledged by the client. But this approach fails when it comes to ALB. During the transmission pauses, I don't see any packages from the server behind the load balancer (I use Wireshark). Then after the timeout of 1 minute, the ALB resets the connection.

I tried to use client-sent HTTP2 pings as well. But the connection also resets in 1 minute and I have no evidence whether these ping packages actually reached the server behind the ALB. I have an assumption that AWS ALB doesn't allow such packets to pass over it, but I didn't find any documentation that proves it.

Greyback

ALB support grpc aws.amazon.com/blogs/aws/…, it's weird that they don't support PING. – Quartis

Answer 1

ALB forwards requests based on HTTP protocol semantics, and not raw HTTP/2 frames. Therefore something like ping frames will only apply for one of the hops.

If you want an end to end ping, you could define a gRPC API which is performing the ping. For server to client you would be required to use a server side streaming APIs. But it might actually be preferrable to let the clients start the pings, to reduce the worker the server has to perform.

Lithograph

When I use Wireshark to inspect the local gRPC bidirectional streaming environment I can see frames with TCP payload that are reaching my server. Those do not say explicitly ping anywhere in the frame. if my protobuf message is reaching the target server ( which is also a frame ) over persistence grpc connection. Why isn't this ping frame reaching? – Nevernever

Answer 2

The AWS support team responded to my ticket and the short answer is ALB does not support the HTTP2 ping frames. They suggested increasing the value of idle timeout on the load balancer, but this solution may be not applicable in some cases.

As Matthias247 already mentioned, the possible workaround is to define a gRPC API for the purpose of doing a ping.

Greyback

Answer 3

Since ALB does not support the HTTP2 ping frames. A straightforward way to solve it is to use a custom PING message.

I think you can get another new stream to send messages when the current stream is closed by ALB due to idle timeout (without messages within idle time)

When idle timeout of ALB, the RST_STREAM message with ErrCode=PROTOCOL_ERROR will be sent from ALB to client-side. The client could handle this error in sender and receiver and then get another new stream to send new messages to reuse the http2 connection.

Here are the sample codes with gRPC-go

    conn, errD := grpc.Dial(ServerAddress, 
        grpc.WithTransportCredentials(cred), 
        grpc.WithConnectParams(grpc.ConnectParams{MinConnectTimeout: time.Duration(63) * time.Second}),
        grpc.WithKeepaliveParams(keepalive.ClientParameters{
            Time:                time.Second * 20,
            Timeout:             time.Second * 3,
            PermitWithoutStream: true,
        }))

    if errD != nil {
        log.Fatalf("net.Connect err: %v", errD)
    }
    defer conn.Close()

    grpcClient := protocol.NewChatClient(conn)
    ctx := context.Background()
    stream, errS := grpcClient.Stream(ctx, grpc.WaitForReady(true))
    if errS != nil {
        log.Fatalf("get BidirectionalHello stream err: %v", errS)
    }

    for i := 0; i < 200; i++ {
        err := stream.Send(
            // some message
        })
        if err != nil {
            if err == io.EOF {
                // get anothe stream to send new message on sender
                stream, errS = grpcClient.Stream(ctx, grpc.WaitForReady(true))
                if errS != nil {
                    log.Fatalf("get  stream err: %v", errS)
                }
            } else if s, ok := status.FromError(err); ok {
                switch s.Code() {
                case codes.OK:
                    // noop
                case codes.Unavailable, codes.Canceled, codes.DeadlineExceeded:
                    return
                default:
                    return
                }
            }
        }

        go func() {
            for {
                res, errR := stream.Recv()
                if errR != nil {
                    if errR == io.EOF {
                        log.Printf("stream recv err %+v \n", errR)
                    }

                    // get anothe stream to send new message on receiver
                    stream, errS = grpcClient.Stream(ctx, grpc.WaitForReady(true))
                    if errS != nil {
                        log.Fatalf("in recv to get stream err: %v", errS)
                    }
                    return
                }

                log.Printf("recv resp %+v", res)
            }
        }()

        // over the idle timeout of alb (60 seconds)
        time.Sleep(time.Duration(61) * time.Second)
    }

To view the details of gRPC message, you could run it through GODEBUG=http2debug=2 go run main.go

Bessie

grpc.WithKeepaliveParams is enabling client side http2 ping frame! Since this is not supported by ALB, why is it achieving? Is it required just to detect the connection have been dropped? Why not use tcp keep alive instead? – Chingchinghai