Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

Question

I just started using Kafka and hit the following rookie error:

'Value cannot be null.
Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.'

It happens when trying to send a class object, ActMessage object, rather then the a simple string that comes with the example. The line of code that raises the erros is:

using (var p = new ProducerBuilder<Null, ActMessage>(config ).Build()

I am using the .net client.

My understanding is that i need to use one of the default serializes in the first type parameter, one that come with Kafka client, as explained here, but can't find them on this .net package. I guess i could build one but that would be a waste of time.

Here a reproducible example:

public class ActMessage  {
    public int SomeId {get;set;}
    public string SomeContent {get;set;}
}

class Tester  {

void send(){

    var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

        using (var p = new ProducerBuilder<Null, ActMessage>(config).Build()) //throws error here
        {
            var dr = p.ProduceAsync("news", new Message<Null, ActMessage>
            {
                Value = new ActMessage { SomeId = 1, SomeContent="hi" },
            }
                ).Result;
        }
    }
}

Martian

Okay. You didn't set a serializer in the ProducerConfig like the error says. How would Kafka know what to do with your class to convert it to bytes? Similarly, you should at least add a toString implementation to your class – Bonilla

I would expect some default implementation, to json, xml, or even binary. But as i say, i am starting with kafka, and try to use it in the simpler way possible. – Martian

Given that C# doesn't have a default JSON or XML lib, then binary would be an option, assuming you know how to use serialization interfaces of C# – Bonilla

I am going for json, can be useful to read the messages exchanged between services with a naked eye. I am moving away from MSMQ and kafka is quite different. – Martian

Avro can also be inspected by the naked eye. Look at services like KSQL, for example. Benefit of which would be far greater throughput on your topic, and almost all external Kafka services kinda use the Confluent Schema Registry API to do really cool things – Bonilla

That is a strong argument. Thanks – Martian

Answer 1

I suggest checking out the working examples/ dir in that repo to see working code that you can copy into your own projects.

---

If you have your own class, you need to implement the ISerializer and IDeserializer interfaces.

Or you can use the built-in ones

---

However, an alternative is to use Avro

This requires writing an Avro schema file, then using avrogen to create your class, not manually write it. E.g.

dotnet tool install --global Apache.Avro.Tools

avrogen -s User.avsc .

Then you must always add some ValueSerializer in Kafka clients in order to send data

Bonilla

I was expecting that i could use some general purpose, built-in serializer, but that is ok anyway. – Martian

You can. Implement ToString, then use StringSerializer – Bonilla

In the examples above i see there is a string schemaRegistryUrl.It is provided as an argument to the Main. I was wondering how does that value look like. Thanks. – Halfhardy

@BercoviciAdrian Its just the registry HTTP endpoint. http://registry.address:8081 – Bonilla

I do not really get how i can go from a C# model to write it in KSQL (Kafka) as AVRO as vice-versa. – Halfhardy

@BercoviciAdrian In the linked example, after .SetValueSerializer(new AvroSerializer<User>(schemaRegistry, avroSerializerConfig)), where User is a generated class from an Avro schema using avrogen tool mentioned in the repo. Then bytes are sent to Kafka.. From KSQL (or any other client), they need the corresponding deserializer. – Bonilla

I am facing the same issue yet my class implements Serializeable and has toString method but the ProducerConfig class in Confluent.Kafka does not have a definition for ValueSerializer or Serializer at all. So how to bind Serializers to ProducerConfig? – Leund

@Guy As answered, you use SetValueSerializer. Do not implement native Serializable interface. You instead define your own Serializer subclass for your POCO – Bonilla

@Bonilla this is just what I did (beware everybody of Serializeable interface). Thanks! – Leund

@Guy Not really "beware". Just a misunderstanding that "native serialization" is not cross-language. E.g. If someone tried to consume your C# Serializable class in JVM client, and had also implements Serializable on those class types for a consumer, then it wouldn't work. Thus, why you use cross language formats like JSON/Avro/Protobuf – Bonilla