Can I reuse an existing protobuf binary when marshalling the messages containing it? (protobuf3)

Question content

protobuf is defined as follows:

syntax = "proto3"

message hugemessage {
    // omitted
}

message request {
    string name = 1;
    hugemessage payload = 2;
}

In one case, I received a hugemessage from someone and I wanted to pack it with additional fields and then transmit that message to someone else. So I have to hugemessage the binary unmarshal into a go structure, package it into a request and then marshal# it again ##. Due to the hgue size of hugemessage, the cost of unmarshal and marshal is prohibitive. So can I reuse the hugemes​sage binary without changing the protobuf definition?

func main() {
    // receive it from file or network, not important.
    bins, _ := os.ReadFile("hugeMessage.dump")
    var message HugeMessage
    _ = proto.Unmarshal(bins, &message) // slow
    request := Request{
        name: "xxxx",
        payload: message,
    }
    requestBinary, _ := proto.Marshal(&request) // slow
    // send it.
    os.WriteFile("request.dump", requestBinary, 0644)
}

Correct Answer

The short answer is: No, there is no easy or standard way to achieve this.

The most obvious strategy is to do it the way you currently do it - unmarshal

hugemessage, set it to request, and then marshal again. The Golang protobuf API surface doesn't really provide a way to do more - and for good reason.

That said, there are

ways to achieve what you want to do. But these aren't necessarily safe or reliable, so you have to weigh that cost against what you have now.

One way to avoid unmarshalling is to take advantage of the way messages are normally serialized;

message request {
    string name = 1;
    hugemessage payload = 2;
}

..Equivalent to

message request {
    string name = 1;
    bytes payload = 2;
}

.. where

payload contains the results of calling marshal(...) against some hugemes​​sage.

So, if we have the following definition:

syntax = "proto3";

message hugemessage {
  bytes field1 = 1;
  string field2 = 2;
  int64 field3 = 3;
}

message request {
  string name = 1;
  hugemessage payload = 2;
}

message rawrequest {
  string name = 1;
  bytes payload = 2;
}

The following code:

req1, err := proto.Marshal(&pb.Request{
    Name: "name",
    Payload: &pb.HugeMessage{
        Field1: []byte{1, 2, 3},
        Field2: "test",
        Field3: 948414,
    },
})
if err != nil {
    panic(err)
}

huge, err := proto.Marshal(&pb.HugeMessage{
    Field1: []byte{1, 2, 3},
    Field2: "test",
    Field3: 948414,
})
if err != nil {
    panic(err)
}

req2, err := proto.Marshal(&pb.RawRequest{
    Name:    "name",
    Payload: huge,
})
if err != nil {
    panic(err)
}

fmt.Printf("equal? %t\n", bytes.Equal(req1, req2))

Output

equal? ​​true

It's unclear whether this "quirk" is completely reliable, and there's no guarantee it will continue to function indefinitely. Obviously the

rawrequest type must exactly mirror the request type, which is not ideal.

Another option is to build the message in a more manual way, i.e. using the

protowire package - again, feel free, caution is recommended.

The above is the detailed content of Can I reuse an existing protobuf binary when marshalling the messages containing it? (protobuf3). For more information, please follow other related articles on the PHP Chinese website!