Can I publish/consume to/from multiple namespaces and topic of Apache Pulsar using single pulsar client?

Question

I am trying to write a multi-tenant (multiple namespaces) system using Apache Pulsar. What I cannot seem to achieve is publishing to multiple namespaces and topics in those namespaces using single client/connection. And the same goes on consumer side. Essentially I want to use single application instance to be able to publish to multiple namespaces and topics and a consumer application to be able to do that on while consuming. I just want to understand if that is even possible.

I don't want to maintain M*N connections for M Namespace and N topics each namespace. I see there is regex support in pulsar but what does that mean on consumer side if I want to consume each topic separately (do I get mixed messages from all the topics matching regex ?)

Twinflower

Answer 1

You can publish to multiple topics in a single client using multiple producers in the client. However, there is a one-to-one relationship between a producer and a topic. From the documentation of the Pulsar binary protocol, Pulsar first establishes a relationship between the producer and a single topic (in a particular tenant/namespace):

message CommandProducer {
  "topic" : "persistent://my-property/my-cluster/my-namespace/my-topic",
  "producer_id" : 1,
  "request_id" : 1
}

And when it sends messages, it references the producer ID:

message CommandSend {
  "producer_id" : 1,
  "sequence_id" : 0,
  "num_messages" : 1
}

Because of this relationship between a producer and a topic, so to send to multiple topics, you need multiple producers.

This diagram illustrates setting up a producer and sending a message:

Here is some Java code that illustrates multiple producers in a client to send to multiple topics:

       // Create client object
        PulsarClient client = PulsarClient.builder()
                .serviceUrl("pulsar+ssl://useast2.aws.kafkaesque.io:6651")
                .authentication(
                    AuthenticationFactory.token("eyJhbGciOiJS...")
                )
                .build();

        // Create producer on a topic
        Producer<byte[]> producer1 = client.newProducer()
                .topic("persistent://tenant/local-useast2-aws/test-topic4")
                .create();

        // Create another producer on a different namespace/topic
        Producer<byte[]> producer2 = client.newProducer()
                .topic("persistent://tenant/namespace2/test-topic5")
                .create();

        // Send messages on the producers
        producer1.newMessage().value("ping from producer1".getBytes()).send();
        producer2.send("ping".getBytes());

        // Send close the producers
        producer1.close();
        producer2.close();

On the consumer side, like the producer you can use multiple consumers in a client to consume from multiple topics. Or you can use a regex to associate a single consumer with multiple topics. But, yes, you will get messages from all topics matching the regex on that one consumer. BTW, you can read a great analysis of using regex to subscribe to multiple topics by Jack Vanlightly here.

Portfire

Thank you Chris. This was very helpful – Twinflower

I have found my test sub blocked the first time a pub sent on a new topic. This answer helped me find that out. – Freshman