gRPC future callback got exception:StatusRuntimeException: CANCELLED

Question

io.grpc.StatusRuntimeException: CANCELLED
    at io.grpc.Status.asRuntimeException(Status.java:539)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
    at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:74)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:508)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:425)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:540)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
    at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:64)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:544)
    at io.grpc.internal.AbstractClientStream2$TransportState.closeListener(AbstractClientStream2.java:307)
    at io.grpc.internal.AbstractClientStream2$TransportState.transportReportStatus(AbstractClientStream2.java:287)
    at io.grpc.netty.NettyClientHandler.cancelStream(NettyClientHandler.java:455)
    at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
    at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
    at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1022)
    at io.netty.channel.AbstractChannel.write(AbstractChannel.java:291)
    at io.grpc.netty.WriteQueue.flush(WriteQueue.java:124)
    at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:46)
    at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:58)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)

I got an exception as above when i call another grpc service in a grpc server handle process with future & callback.

My code like this:

handle(){

....

***FutureClient futureClient = ...
ListenableFuture<***> future = futureClient.call(...);
Futures.addCallback(future, new FutureCallback<>() {...});

...

}

The access was canceled when the process handle() completed.

How can I resolve the problem?

Osswald

Cancel happens when there is a client side error, or it was cancelled by application code (i.e. your code). Are you sure you aren't cancelling the future somewhere? – Wolff

Answer 1

Outgoing RPCs from a server are implicitly tied to the incoming request that triggered them, via io.grpc.Context. If you are doing an RPC that proceeds even after the incoming RPC completed you should use context.fork():

Context forked = Context.current().fork();
Context old = forked.attach();
try {
  // RPCs at this point can continue after the incoming RPC completes
  futureClient.call(...);
} finally {
  forked.detach(old);
}

Context.run(Runnable) is also a convenient alternative to using attach/detach directly.

Lid

This was exactly what I was looking for! I hope this would be clearly stated also in documentation. – Mise

@Eric Anderson Could you please elaborate on what "proceeds even after the incoming RPC completed" ? I too encountered the issue. Client makes a unary request, and server sends out a stream of responses in the same thread. So, I fail to see how the "incoming RPC is completed"? Does it have more to do with timeout or something? – Crossly

@UmaPriyadarsi, the case here is when a server creates an RPC to process an RPC (that is, the server is also a client). By default that "child" RPC cannot outlive the parent RPC. In your case it sounds like the server just responds; there's no second RPC involved. – Lid

@EricAnderson yeah I got my doubt resolved over here : github.com/grpc/grpc-java/issues/8876#event-5990360727 – Crossly

HI @EricAnderson what's the difference between your approach here and using ServerCallStreamObserver described here: #54588882 – Gaines

@macemers, "what do you want to happen if the client cancels the RPC?" If you want any outgoing RPCs to be cancelled and also to stop processing, then you'd follow that other answer. If you want outgoing RPCs to continue and to continue processing, you'd use this answer. – Lid