Why is my Protobuf message (in Python) ignoring zero values?

Question

I've been working on implementing protobufs for IPC for a project. For some reason, values that are set to 0 are not being set/serialized. For context, the .proto file contains the following message:

syntax = "proto3";

enum SetGet {
    SET = 0;
    GET = 1;
}

message State {
    SetGet setget = 1;
    double x = 2;
    double y = 3;
    double depth = 4;
    double yaw = 5;
    double pitch = 6;
    double roll = 7; 
}

I compile the file to a Python _pb2 file with protoc, and then I try running the following test script:

import filename_pb2 as pb

state = pb.State()
state.x = 0
state.y = 0
state.depth = 0
state.yaw = 0
state.pitch = 0
state.roll = 0
state.setget = pb.SET

print("State: {}".format(state))

state2 = pb.State()
state2.ParseFromString(state.SerializeToString())

print("State2: {}".format(state2))

When I run it, the following output is printed:

State: 
State2: 

It seems that nothing is being set, or that the zero values are somehow being ignored. However, when I change the values (x, y, depth, etc.) to something nonzero, say 0.1, I get the following, expected result:

State: x: 0.1
y: 0.1
depth: 0.1
yaw: 0.1
pitch: 0.1
roll: 0.1

State2: x: 0.1
y: 0.1
depth: 0.1
yaw: 0.1
pitch: 0.1
roll: 0.1

Even though the numbers are printed out, for some reason the enum still isn't. Why does this happen with protobufs? Is the double type 0 by default, so the protobuf serializer saves on space by ignoring them? Why, then, are they not being restored when State2 is parsed in? Is there some line in the documentation that I missed? Thanks in advance!

-- Tim

Ranunculaceous

If you're used to proto2, then yes: this is a key change in proto3 – Mononuclear

Answer 1

Yes, 0 is the default. This case is mentioned explicitly in the documentation:

Note that for scalar message fields, once a message is parsed there's no way of telling whether a field was explicitly set to the default value (for example whether a boolean was set to false) or just not set at all: you should bear this in mind when defining your message types. For example, don't have a boolean that switches on some behaviour when set to false if you don't want that behaviour to also happen by default. Also note that if a scalar message field is set to its default, the value will not be serialized on the wire.

Amblyopia

Good to know. Sorry I didn't catch that! – Ranunculaceous

Answer 2

This has been changed recently; proto3 now supports presence tracking, which is enabled by adding the optional keyword:

message State {
    optional SetGet setget = 1;
    optional double x = 2;
    optional double y = 3;
    optional double depth = 4;
    optional double yaw = 5;
    optional double pitch = 6;
    optional double roll = 7; 
}

from https://github.com/protocolbuffers/protobuf/blob/master/CHANGES.txt:

2021-02-05 version 3.15.0 (C++/Java/Python/PHP/Objective-C/C#/Ruby/JavaScript)

Protocol Compiler

• Optional fields for proto3 are enabled by default, and no longer require the --experimental_allow_proto3_optional flag.

which is the RTM of this:

2020-05-12 version 3.12.0 (C++/Java/Python/PHP/Objective-C/C#/Ruby/JavaScript)

Protocol Compiler

• [experimental] Singular, non-message typed fields in proto3 now support presence tracking. This is enabled by adding the "optional" field label and passing the --experimental_allow_proto3_optional flag to protoc.

Mononuclear

Why do we have to set the field as optional to allow the printing of 0 valued enums? Whether or not the setget field is optional, it doesn't change that the SetGet enum has a valid value for #0 – Misappropriate

Answer 3

Zero is the default for numerics in protobuf, and empty strings are defaults for strings. For efficiency, default values are not transmitted across the wire.

If you really want to be able to tell if it was set explicitly, don't use the default zero for a real operation:

enum SetGet {
    NONE = 0;
    SET = 1;
    GET = 2;
}

Just keep in mind this will result in extra traffic over the wire and, since you're only really worried out what's being printed, you can alternatively just understand that zeros are defaults, or write your own print routines that output everything.

Quotidian

What you say is certainly true for proto3 - but for proto2 and the raw binary format, it is more complicated. – Mononuclear