Protobuf got schemas right, mostly, but its 2001 wire format made two costly mistakes: it requires the schema just to decode values, and its byte-by-byte LEB128 varints are slow as hell.
OmniBuf gives you all the benefits of Protobuf (your
existing .proto files, compact binary payloads, schema
evolution, and raw byte concatenation) in a wire format that is over 6x to 8x faster, easier to debug without a schema, and simpler to implement in any language.
You shouldn’t have to abandon .proto or accept bloated zero-copy pointer graphs
just to get fast serialization and readable packets.
Protobuf’s
LEB128
varints check a continuation bit on every single byte, stalling CPUs with branch
mispredictions, and force encoders to walk the message tree twice just to calculate
nested lengths. OmniBuf encodes integer byte-length in the first byte (
TZCNT
) for single-load decoding and writes messages forward in a single pass.
Protobuf made a mistake by lumping strings, bytes, sub-messages, maps, and packed arrays
into a single opaque wire type (Wire Type 2). OmniBuf spends slightly more
tag space (~5% message size) so wire types match Protobuf types directly. Any CLI,
proxy, or log viewer can decode values without the original .proto file,
and you never have to wrestle with
FlatBuffers
’ bottom-up builders.
Zero-copy formats require complex vtables, relative pointer graphs, alignment padding,
and arena allocators. Even Protobuf requires special cases for groups, unpacked vs.
packed arrays, and synthetic MapEntry sub-messages. OmniBuf is a flat
stream of [Tag][Value] pairs with 4 integer widths (1, 2, 4, 9
bytes) and 16 explicit wire types.
OmniBuf works directly with your existing .proto schemas and generated
structs. You keep field numbers, safe forward and backward compatibility, near-Protobuf
wire size (40% smaller than FlatBuffers), and raw byte concatenation (
append(msgA, msgB...)) for zero-parse message merging.
Measured on an 11-field telemetry message with nested structs and arrays. Shorter bars = Faster.
| Format |
Uses .proto | Wire Size | 1-Pass Encode | Schema-Less Types | append(a, b) Merge
|
|---|---|---|---|---|---|
| OmniBuf | Yes | ~6% larger | Yes (6x faster) | Yes (+ opt Field #0) | Yes |
| Protobuf | Yes | Baseline (1x) | No (2-pass Size) | No (Opaque Type 2) | Yes |
| FlatBuffers |
No (.fbs)
| +76% larger | Bottom-up only | No | No |
| Cap'n Proto |
No (.capnp)
| +64% larger | Arena pointers | No | No |
| MessagePack / JSON | Optional | +92% / +135% larger | Yes (Slow) | Yes (Repeated Keys) | No |