How to inspect clients that are connected to a GRPC server

Question

To provide better debugging information for my GRPC server/client setup, I am trying to find an API for grpc.server that allows me to inspect what clients are connected to the server.

The most promising question I have found is question, which gives a starting point on how to do this in Java GRPC. However, the Java API doesn't exist in the Python GRPC implementation.

So far, I keep track of unique peers using the context.peer() method in a grpc.ServicerContext. If a peer hasn't sent a request in a while (a timeout I set to 2 seconds) I assume the client has disconnected.

I've started looking at the python-grpc source code but I haven't made any headway.

If anyone knows of a similar API in python I could use, that would be appreciated! Even an internal API would be sufficient for these debugging needs.

Quitt

Does grpc.io/blog/a_short_introduction_to_channelz work for your requirement? – Maribelmaribelle

@ZhouyihaiDing this is my most promising lead, but when I was doing the research I could find almost nothing about the python API for channelz, and the source code on grpc.github.io/grpc/python/grpc_channelz.html was not very informative. It seems that if I install the pip package grpc-channelz, then I'll be able to pull channelz statistics from "C-Core", but there is no documentation or examples as to how to do this programmatically in Python. Although if someone does have this documentation, I would love to discuss it in an answer! – Quitt

Answer 1

I found some more documentation and examples of channelz, other people have been suggesting it and it seems like what you want.

Here's a pull request that gives an example of channelz being used. It only uses the GetServer channelz API, so you'll have to adapt it.

And here's a unit test that uses channelz, which tests APIs that are probably relevant GetTopChannels API.

Harmonist

Thank you! This has what I was looking for! Richard and Zhouyihai both suggested channelz, but these examples allowed me to figure out a solution. I'll edit this answer with some code. – Quitt

I haven't been able to edit in a code sample of a channelz solution since my edit was rejected, but I was able to access channelz statistics of a GRPC server in the same process by just copying and pasting the stub implementations of, say, GetServerSockets from grpc.github.io/grpc/python/_modules/grpc_channelz/v1/…. Specifically, accessing grpc._cython.cygrpc.channelz_get_server_sockets will return an protobuf representing server sockets connected to clients. – Quitt

Answer 2

There isn't a native API for this, but you have all of the pieces you need. Here's a modified version of the helloworld example from the repo.

class PeerSet(object):
    def __init__(self):
        self._peers_lock = threading.RLock()
        self._peers = {}

    def connect(self, peer):
        print("Peer {} connecting".format(peer))
        with self._peers_lock:
            if peer not in self._peers:
                self._peers[peer] = 1
            else:
                self._peers[peer] += 1

    def disconnect(self, peer):
        print("Peer {} disconnecting".format(peer))
        with self._peers_lock:
            if peer not in self._peers:
                raise RuntimeError("Tried to disconnect peer '{}' but it was never connected.".format(peer))
            self._peers[peer] -= 1
            if self._peers[peer] == 0:
                del self._peers[peer]

    def peers(self):
        with self._peers_lock:
            return self._peers.keys()


class Greeter(helloworld_pb2_grpc.GreeterServicer):

    def __init__(self):
        self._peer_set = PeerSet()

    def _record_peer(self, context):
        def _unregister_peer():
            self._peer_set.disconnect(context.peer())
        context.add_callback(_unregister_peer)
        self._peer_set.connect(context.peer())

    def SayHello(self, request, context):
        self._record_peer(context)
        for i in range(10):
            print("[thread {}] Peers: {}".format(threading.currentThread().ident, self._peer_set.peers()))
            time.sleep(1)
        return helloworld_pb2.HelloReply(message='Hello, %s!' % request.name)

This will get you output like the following:

[thread 139905506195200] Peers: [u'ipv6:[::1]:57940', u'ipv6:[::1]:57930', u'ipv6:[::1]:57926', u'ipv6:[::1]:57920', u'ipv6:[::1]:57934']

As Ding commented above, channelz may also be a good fit for you.

Fantasize

Thanks for the code sample Richard! I'm already using context.peer() as you have suggested, but this only keeps track of currently-executing RPCs, not active channels in both my code and your solution. Apologies for not clarifying my context.peer() solution. channelz would work, but I can find almost no documentation on its python APIs as per my reply to Ding (specifically, how to read channelz statistics from the "C-Core" in Python). If there are no other public Python channelz docs, I'll just hack in a long-lived "heartbeat" RPC to detect when a channel closes from the server side. – Quitt

Answer 3

There is one tool grpcdebug, it could inspect clients that are connected to a GRPC server.

grpcdebug is a command line interface focusing on simplifying the debugging process of gRPC applications. grpcdebug fetches the internal states of the gRPC library from the application via gRPC protocol and provide a human-friendly UX to browse them. Currently, it supports Channelz/Health Checking/CSDS (aka. admin services)

grpcdebug is an gRPC service admin CLI

Usage:
  grpcdebug <target address> [flags] <command>

Available Commands:
  channelz    Display gRPC states in human readable way.
  health      Check health status of the target service (default "").
  help        Help about any command
  xds         Fetch xDS related information.

Cleodell