How to Implement Protocol Buffers Delimited I/O in C : Missing Functionality and Potential Optimizations

C Equivalents for Protocol Buffers Delimited I/O Functions in Java

In Protocol Buffers, adding length prefixes before messages is essential for reading and writing multiple messages from files. While Java has "Delimited" I/O functions (e.g., parseDelimitedFrom, mergeDelimitedFrom, writeDelimitedTo) for this purpose, C lacks equivalent functions.

Google's Recommendation

Google recommends attaching length prefixes to messages manually in C . This involves:

• Writing the message size as a Varint
• Serializing the message into a buffer

However, this approach lacks optimizations and may fail for inputs larger than 64MB.

Optimized C Implementations

Subsequently, optimized C implementations were developed and shared by a former Google protobuf library author. These implementations include:

• Writing the size as a Varint
• Using the GetDirectBufferForNBytesAndAdvance method for efficient serialization
• Enforcing a per-message size limit of 64MB

Implementation Details

The writeDelimitedTo function creates a new CodedOutputStream for each message and serializes it using the SerializeWithCachedSizes method. If the message fits in one buffer, the faster SerializeWithCachedSizesToArray method is used.

The readDelimitedFrom function creates a new CodedInputStream and imposes a per-message size limit using the PushLimit method. It then parses the message using the MergeFromCodedStream method and verifies that the entire message was consumed.

Availability

These optimized C implementations are not part of the official protobuf library. However, they can be found in various third-party repositories or implemented manually using the provided code snippets.

The above is the detailed content of How to Implement Protocol Buffers Delimited I/O in C : Missing Functionality and Potential Optimizations. For more information, please follow other related articles on the PHP Chinese website!