Wire Format
Specification.

1. Design Goals & Overview

1.1 Design Goals

OmniBuf is designed around four core goals:

1. Stay Compatible with .proto File Semantics and Types:OmniBuf uses existing .proto definitions, field numbers (1..N), scalar types, nested messages, repeated lists, and maps without requiring a new IDL, custom memory layouts, or manual alignment rules. Existing Protocol Buffers schemas and generated struct types can target the OmniBuf wire format directly.
2. Faster Encoding and Decoding on Modern Hardware:Protocol Buffers (designed in 2001) relies on 7-bit LEB128 continuation loops (0x80 branch per byte), two-pass Size() calculations for nested sub-messages, and byte-by-byte varint parsing when skipping fields. OmniBuf aligns the wire format with modern 64-bit little-endian CPUs:
   • Single-instruction trailing-zero count (TZCNT / CTZ) varints load 1, 2, 4, or 8 bytes in a single unaligned little-endian memory access.
   • Single-pass forward encoding reserves and backpatches 2-byte length prefixes in place without a pre-pass.
   • Explicit 4-bit wire types and outer container byte-length prefixes allow decoders to skip entire fields or nested containers in a single branchless TZCNT header hop without parsing inside strings, sub-messages, or arrays.
3. Schema-Less Value Decoding (with Optional Inline Field Names):In Protocol Buffers, Wire Type 2 (LEN) is overloaded for strings, raw bytes, nested messages, and packed repeated scalars, making it impossible to decode a packet without the .proto schema. OmniBuf uses explicit 4-bit wire types (0..15) so any decoder can losslessly inspect and decode every field's exact type and structure (DecodeBlind), including unsigned vs. ZigZag signed integers, fixed-width float arrays, and typed maps, with zero schema knowledge. When human-readable keys are desired on the wire, an optional inline schema sub-message (Field #0, omnibuf.v1.SchemaMeta) embeds field-number-keyed field names and nested child schemas at byte 0 of the root message or any nested sub-message.
4. Advanced Wire Techniques with a Flat Stream Layout:Rather than requiring vtables, footer offset tables, relative pointer graphs, or arena allocators, OmniBuf achieves single-pass encoding, fast zero-copy unmarshaling, raw byte concatenation (append(msgA, msgB...)), and rapid header-hop field/array/map traversal using a single contiguous stream of [Tag][Value] fields with 4 integer widths and 16 explicit wire types.

1.2 Core Wire Primitives

To achieve these goals, OmniBuf replaces Protobuf's Tag-Length-Value (TLV) + LEB128 with four wire primitives:

1. Branchless Prefix-Varints (TZCNT): The trailing zero bit count (TZCNT) of the first byte determines the total byte width (1, 2, 4, or 9 bytes) in little-endian order, eliminating byte-by-byte continuation-bit loops.
2. Explicit 4-Bit Wire Types (0..15): Every field tag encodes (FieldNumber << 4) | WireType, unambiguously distinguishing unsigned integers (WireUint, WireUintArr), ZigZag-encoded signed integers (WireSint, WireSintArr), strings, bytes, sub-messages, string arrays, length-prefixed sub-message arrays, typed maps, fixed-width floats/ints (and packed arrays WireFixedArr), zero-byte booleans (true/false encoded directly in the tag), and a length-prefixed extension escape hatch (WireExtended = 15).
3. Single-Pass Overlong Length Backpatching: Variable-length containers (WireSubMsg, WireStringArr, WireUintArr, WireSintArr, WireSubMsgArr, WireMap) reserve a 2-byte Prefix-Varint length slot (TZCNT = 1) before writing their body in a single forward pass and backpatch the 14-bit length upon completion, eliminating two-pass Size() pre-computation.
4. Concat-Compatible Field Stream with Outer Container Lengths: Because every scalar width is encoded in b[0] via TZCNT and every container begins with its total ByteLen, a decoder can skip any field in a single O(1) arithmetic jump without parsing inside sub-messages or arrays, and concatenating two encoded OmniBuf messages (append(msgA, msgB...)) produces a valid merged OmniBuf message.

2. Message Packet Layout & Concatenation Semantics

Every OmniBuf root message (and every nested sub-message in WireSubMsg or WireSubMsgArr) is a flat sequence of encoded [Tag][Value] fields with no header table or trailing footer:

1. Field Ordering & Raw Message Concatenation (append(msgA, msgB...)):
   • Encoders SHOULD emit fields in ascending order of FieldNumber for deterministic output.
   • Decoders MUST accept fields in any order and support duplicate field numbers so concatenating two encoded messages (append(msgA, msgB...)) or appending an updated [Tag][Value] to the end of an existing packet follows Protocol Buffers merge semantics:
     • Singular Scalars: Last occurrence wins ("last write wins").
     • Repeated Fields: Elements from later occurrences are appended in order to earlier elements.
     • Nested Sub-Messages (WireSubMsg): Later occurrences merge recursively into the existing sub-message.
     • Maps (WireMap): Entries from later occurrences merge into the map, with later keys overriding earlier keys.
2. Zero-Byte Empty Message: An empty message with no present fields is 0 bytes.
3. No Trailing Footer or Padding: The message ends immediately after the final byte of the last field.

3. Prefix-Varint Encoding (TZCNT)

All variable-length unsigned integers (field tags, WireUint values, container byte lengths, and element counts) use Prefix-Varints.

3.1 Bit-Level Layout

The number of trailing zero bits (TZCNT) in Byte 0 (b[0]) tells the CPU the exact byte length of the integer (1, 2, 4, or 9 bytes) up front. The remaining high bits of the little-endian word store the unsigned value:

3.2 Prefix-Varint Binary & Hex Examples

3.3 Overlong 2-Byte Prefix-Varints (Single-Pass Length Backpatching)

Decoders MUST dispatch strictly on TZCNT(b[0]) without rejecting non-minimal ("overlong") encodings. Specifically, a length v < 128 encoded with the 2-byte formula (uint16(v << 2) | 2) is valid on the wire.

• Fast Path (lengths up to 16,383 bytes): When encoding a container in a single forward pass, the encoder reserves a 2-byte slot at lenPos, writes the container body forward, and patches the 2-byte length slot at the end: LE16(buf[lenPos:lenPos+2]) = (byteLen << 2) | 2.
• Large Container Expansion (> 16,383 bytes): If a container exceeds 16,383 bytes, the encoder shifts the container body right by 2 bytes (for a 4-byte varint up to 512 MiB) or 7 bytes (for a 9-byte varint) and writes the wider header.
• Example (byteLen = 42 as 2-byte varint): (42 << 2) | 2 = 170 = 0x00AA → hex aa 00. The decoder sees TZCNT(0xAA) == 1 (2 bytes) and computes 0x00AA >> 2 = 42.

4. Signed Integers (ZigZag Encoding, WireSint = 12)

Signed integers (int32, sint32, int64, sint64) map to unsigned uint64 values via ZigZag encoding before being written as a WireSint (12) Prefix-Varint. Giving signed integers their own 4-bit wire type (WireSint = 12, distinct from WireUint = 0) lets schema-less decoders (DecodeBlind) decode negative and positive integers directly from raw bytes.

4.1 ZigZag Formulas

// Encode signed int64 -> unsigned uint64:
u := uint64((n << 1) ^ (n >> 63))

// Decode unsigned uint64 -> signed int64:
n := int64(u >> 1) ^ -int64(u & 1)

4.2 ZigZag Binary & Hex Examples

5. Field Tags & Explicit 4-Bit Wire Types

Every field begins with a Tag that packs the schema field number and the 4-bit wire type into a single Prefix-Varint:

Tag = (FieldNumber << 4) | WireType

• FieldNumber is the schema field number (0 for the inline schema table, 1..N for message fields).
• WireType is a 4-bit unsigned integer (0..15).

5.1 Tag Width for Field Numbers 0..7 vs 8..1023

Because WireType occupies the low 4 bits of the tag value, the byte width of a tag depends only on how many bits FieldNumber needs:

6. Wire Type Specifications (0..15)

OmniBuf assigns each of the 16 4-bit wire types (0..15) an explicit payload layout so any decoder can skip or inspect fields without external schema metadata:

6.1 WireUint (0) & WireSint (12)

• WireUint (0): Unsigned integer (uint32, uint64, enum) written as a single Prefix-Varint right after the tag.
• WireSint (12): Signed integer (int32, int64, sint32, sint64) ZigZag-mapped and written as a single Prefix-Varint right after the tag.

// Example 1: Field #1 (asn = 64512, WireUint = 0)
21   04 e0 07 00
Tag  64512 as 4B Prefix-Varint (LE32: 0x0007E004 >> 3)

// Example 2: Field #5 (rssi_dbm = -62, WireSint = 12)
b9   f7
Tag  -62 -> ZigZag 123 -> 1B Prefix-Varint (0xF7 >> 1 = 123)

6.2 WireBoolFalse (1) & WireBoolTrue (2)

Booleans consume 0 payload bytes after the tag; the boolean value is carried directly in the 4-bit wire type (1 = false, 2 = true).

65   // Field #3 = true  ((3 << 4) | 2 = 50 -> 0x65)
63   // Field #3 = false ((3 << 4) | 1 = 49 -> 0x63)

6.3 WireString (3) & WireBytes (4)

The tag is followed by [ByteLen (Prefix-Varint)] and [ByteLen raw bytes]. Strings are UTF-8 encoded without a NUL terminator.

47   09     72 74 72 31
Tag  Len=4  "rtr1" (Field #2, 4 UTF-8 bytes)

6.4 WireSubMsg (5)

WireSubMsg encodes a nested message as [ByteLen (Prefix-Varint)][Inner Message Bytes]. Single-pass encoders reserve a 2-byte Prefix-Varint slot (TZCNT = 1) for ByteLen and patch it after writing the nested fields; decoders accept any valid Prefix-Varint width.

// Inner Location message (city = "FRA" at #3, rack = 42 at #4):
67 07 46 52 41   // #3: "FRA" (5 bytes)
81 55            // #4: 42    (2 bytes)

// Outer Field #9 wrapping the 7-byte inner message:
56 02  1e 00  67 07 46 52 41 81 55
Tag#9  Len=7  Inner Sub-Message (7 bytes)

6.5 WireStringArr (6)

WireStringArr encodes repeated string or repeated bytes with an outer ByteLen prefix followed by ElemCount and each length-prefixed element, so a reader can skip the entire list in one addition (pos += ByteLen):

Layout:  [Tag] [ByteLen] [ElemCount] [StrLen_0][Bytes_0] ... [StrLen_N-1][Bytes_N-1]

// Example: Field #7 (tags = ["a", "bc"])
ed   1a 00  05   03 61   05 62 63
Tag  Len=6  N=2  "a"     "bc"

6.6 WireUintArr (7) & WireSintArr (13)

These wire types encode a packed array of unsigned (WireUintArr) or ZigZag-signed (WireSintArr) varints with an outer ByteLen prefix and ElemCount:

Layout:  [Tag] [ByteLen] [ElemCount] [Elem_0] ... [Elem_N-1]

// Example: Field #8 (prefixes = [10, 300])
1e 02  12 00  05   15  b2 04
Tag#8  Len=4  N=2  10  300

6.7 WireSubMsgArr (8)

WireSubMsgArr encodes repeated message inside a single outer ByteLen container followed by length-prefixed sub-message elements:

Layout:  [Tag] [ByteLen] [ElemLen_0][Elem_0] ... [ElemLen_N-1][Elem_N-1]
                         |<-------------- ByteLen bytes -------------->|

• Each element i starts with its byte length ElemLen_i (Prefix-Varint) followed by ElemLen_i bytes of the element message.
• The sum of all len(ElemLen_i) + ElemLen_i across elements MUST equal ByteLen.

6.8 WireMap (9)

WireMap encodes map<K, V> directly inside one length-prefixed container without synthetic MapEntry wrapper sub-messages:

Layout:  [Tag] [ByteLen] [MapType (1B)] [EntryCount] [Key_0][Val_0] ...
                         |<-------------- ByteLen bytes ------------->|

• 1-Byte MapType Header: Encodes (KeyWireType << 4) | ValWireType:
  • KeyWireType: WireUint (0), WireString (3), WireFixed32 (10), WireFixed64 (11), or WireSint (12).
  • ValWireType: WireUint (0), WireString (3), WireBytes (4), WireSubMsg (5), WireFixed32 (10), WireFixed64 (11), or WireSint (12).
• Minimum Container Length: Because every WireMap includes the 1-byte MapType and at least 1 byte for EntryCount, ByteLen MUST be at least 2.
• Untagged Entries & Unordered Keys: Each entry writes the raw payload of KeyWireType followed by the raw payload of ValWireType. Keys may appear in any order; duplicate keys follow "last write wins".

6.9 WireFixed32 (10) & WireFixed64 (11)

• WireFixed32 (10): 4 bytes in little-endian order (float, fixed32, sfixed32).
• WireFixed64 (11): 8 bytes in little-endian order (double, fixed64, sfixed64).

// Example: Field #6 (cpu_load = 0.5 as float64)
d7   00 00 00 00 00 00 e0 3f
Tag  IEEE-754 0.5 (8 bytes little-endian)

6.10 WireFixedArr (14)

WireFixedArr encodes a packed array of fixed-width 32-bit or 64-bit little-endian elements with an outer ByteLen prefix and a 1-byte ElemType header:

Layout:  [Tag] [ByteLen] [ElemType (1B: 10 or 11)] [Little-Endian Elements...]
                         |<------------------ ByteLen bytes ----------------->|

• ElemType = 10 (WireFixed32): The header is followed by (ByteLen - 1) / 4 4-byte elements.
• ElemType = 11 (WireFixed64): The header is followed by (ByteLen - 1) / 8 8-byte elements.

6.11 WireExtended (15)

WireExtended reserves wire type 15 as a length-prefixed extension escape hatch capable of expressing up to 256 extended wire types while remaining skippable in O(1) time:

Layout:  [Tag] [ByteLen] [ExtType (1B: 0..255)] [ExtPayload (ByteLen - 1 bytes)]
                         |<------------------- ByteLen bytes ----------------->|

• Because WireExtended starts with ByteLen just like every other container, existing decoders can skip unknown extended fields in a single jump (pos += ByteLen).

7. Branchless TZCNT Header-Hop Field Lookup (LocateField)

In Protocol Buffers, skipping a preceding varint field requires a byte-by-byte loop. In OmniBuf, every scalar width is known from b[0], booleans are 0 value bytes, and every container starts with ByteLen, so LocateField jumps across fields in O(1) time per field:

// Jump from field to field in O(1) time without parsing inside containers:
for pos < len(packet) {
    tag, tn := DecodeVarint(packet[pos:])
    pos += tn
    fnum, wtype := uint16(tag>>4), uint8(tag&0x0F)

    if fnum == targetField {
        valSlice, wireType, found = packet[pos:], wtype, true
    }
    pos = SkipValue(packet, pos, wtype) // pos += scalarWidth or container ByteLen
}

8. Optional Inline Schema (Field #0, omnibuf.v1.SchemaMeta)

When self-describing mode is enabled, Field #0 (Tag = 0x0B, WireSubMsg = 5) is written at byte 0 of the message as a standard OmniBuf sub-message encoding omnibuf.v1.SchemaMeta.

• Zero-Cost Skipping for Schema-Aware Decoders: Because Field #0 is a normal WireSubMsg starting with ByteLen, schema-aware decoders read ByteLen and jump over the entire metadata block in one addition.
• Extensible via .proto: Because SchemaMeta is a normal Protobuf message (proto/omnibuf/v1/meta.proto), future optional metadata fields can be added without changing the wire format.

8.1 omnibuf.v1.SchemaMeta Definition (meta.proto)

syntax = "proto3";
package omnibuf.v1;

// Written at Field #0 (Tag = 0x0B, WireSubMsg = 5) when self-describing mode is enabled.
message SchemaMeta {
  string type_name = 1;               // Optional message name (e.g. "Location")
  map<uint32, string> fields = 2;     // FieldNumber -> FieldName (e.g. 3 -> "city")
  map<uint32, SchemaMeta> children = 3; // Parent FieldNumber -> child SchemaMeta (stored once)
}

• Keyed by FieldNumber (fields = 2): Maps each field number to its name (1 → "id"), so omitted optional fields never shift field names.
• Hoisted Child Schemas (children = 3): Maps a parent field number (for a nested sub-message, sub-message array, or map value) to its child SchemaMeta so repeated sub-messages only encode their field names once.

8.2 Single Message Wire Encoding with Field #0

Consider a Location message (city = "FRA" at #3, rack = 42 at #4) encoded with Field #0 (type_name = "Location", fields = {3: "city", 4: "rack"}):

8.3 repeated message (WireSubMsgArr) Without Field-Name Duplication

In JSON, an array of 10,000 objects repeats the key strings "city" and "rack" 10,000 times. In OmniBuf, the parent message's Field #0 places the child schema in children[fieldNum] once, and all 10,000 elements in the WireSubMsgArr carry only compact numeric tags and values:

// 1. Stored ONCE in Parent Field #0:
children[10] = {
  type_name: "Neighbor",
  fields:    { 1: "peer_asn", 2: "router", 3: "latency_us", 4: "established" }
}

// 2. Data Field #10 (WireSubMsgArr, N = 10,000 elements):
[ElemLen_0][Elem_0: compact tags + values, 0 repeated field names]
[ElemLen_1][Elem_1: compact tags + values, 0 repeated field names]
...

8.4 Raw Concatenation (append(msgA, msgB...)) with Field #0

Because Field #0 is a regular WireSubMsg (5) whose maps use WireMap (9), concatenating two self-describing packets (append(msgA, msgB...)) works without special framing: when DecodeBlind sees a second Field #0 later in the stream, it merges the new SchemaMeta into the active schema.

9. Complete End-to-End Annotated Wire Example

message Location {
  string city = 3; // "FRA"
  uint32 rack = 4; // 42
}

message Node {
  uint32   id       = 1; // 300
  bool     active   = 3; // true
  Location location = 9; // { city: "FRA", rack: 42 }
}

10. Wire Validity & Malformed Packet Rejection Rules

A conforming OmniBuf decoder MUST validate untrusted wire input and reject malformed or adversarial packets without panicking or reading out of bounds:

1. Empty Message: A 0-byte slice (len(packet) == 0) is a valid empty message with zero present fields.
2. Field #0 Wire Type & Structure: Any occurrence of Field #0 MUST have WireType == WireSubMsg (5) and encode a valid omnibuf.v1.SchemaMeta sub-message (where fields is a WireMap with MapType = 0x03 and children is a WireMap with MapType = 0x05).
3. Defined 4-Bit Wire Types: Wire types 0..14 are core wire types and WireExtended (15) is the length-prefixed extension escape hatch. Any tag whose payload violates its wire type's structural rules MUST be rejected.
4. Payload & Container Bounds:
   • No Prefix-Varint or fixed-width value (WireFixed32, WireFixed64) may read past len(packet).
   • Length-prefixed fields (WireString, WireBytes, WireSubMsg, WireStringArr, WireUintArr, WireSubMsgArr, WireMap, WireSintArr, WireFixedArr, WireExtended) MUST satisfy byteLen <= uint64(len(packet) - pos).
   • WireStringArr, WireUintArr, and WireSintArr elements MUST end on an exact element boundary at pos + byteLen.
   • WireFixedArr MUST satisfy byteLen >= 1, with ElemType == WireFixed32 (10) and (byteLen - 1) % 4 == 0, or ElemType == WireFixed64 (11) and (byteLen - 1) % 8 == 0.
   • WireExtended MUST satisfy byteLen >= 1 (at least 1 byte for ExtType); schema-less decoders MUST reject unrecognized ExtType values, while schema-aware decoders skip unknown fields with WireExtended in O(1) time.
   • WireSubMsgArr elements MUST each satisfy elemLen <= uint64(len(arrPayload) - pos) and the final element MUST end exactly at len(arrPayload).
   • WireMap MUST satisfy byteLen >= 2, contain a valid 1-byte MapType header, satisfy entryCount*2 <= uint64(len(mapPayload) - pos) before allocating entries (preventing OOM pre-allocation attacks), and the final entry MUST end exactly at len(mapPayload). Duplicate keys follow "last write wins" semantics.
5. Resource & Message Size Limits:
   • Maximum Nesting Depth (MaxNestingDepth = 32): By default, sub-message recursion deeper than 32 levels MUST be rejected. Implementations MAY provide an option to configure a custom recursion limit.
   • Message & Container Byte-Size Limits: At the wire level, Prefix-Varints can represent container lengths up to 229 - 1 bytes (~512 MiB) using a 4-byte header and up to 264 - 1 bytes using a 9-byte header. To guard against memory exhaustion on untrusted inputs, implementations and transport layers MAY enforce configurable maximum message size limits:
     • RPC and framing layers MAY enforce a default per-message limit of 4 MiB (4,194,304 bytes, matching gRPC's default receive limit), and general-purpose decoders MAY enforce a default total message limit of 64 MiB (67,108,864 bytes, matching Protocol Buffers' default limit) unless overridden by the caller.
     • Implementations MAY reject individual container or string lengths exceeding 536,870,911 bytes (the 4-byte Prefix-Varint maximum) unless explicitly configured for >512 MiB payloads.

11. Language Implementation Requirements & Conformance

OmniBuf implementations across programming languages SHOULD follow modern, idiomatic practices for their target ecosystem (such as zero-copy slice or span views where memory safety permits, buffer-append encoding APIs to support pooling, and standard language error handling). To provide a consistent developer experience across ecosystems, every complete OmniBuf implementation MUST support three ways to work with types and pass the full conformance test suite.

11.1 Required Usage Modes

Every complete OmniBuf language implementation MUST support encoding and decoding in three ways:

1. Protobuf Compiler Plugin (protoc-gen-omnibuf-*): A dedicated protoc / buf plugin that generates reflection-free, single-pass forward encoders, branchless TZCNT decoders, and lazy header-hopping views (Opaque accessors and raw field splicing) directly from .proto schemas.
2. Usable on Protobuf Types Without the Plugin: Drop-in serialization and deserialization of standard protoc-generated message types (for example, .pb.go structs in Go or standard Protobuf classes in C++, Java, Python, Rust, and TypeScript) using the language's Protobuf runtime descriptors or reflection, requiring zero build-pipeline changes.
3. Usable on Normal Classes, Objects, or Structs with Annotations (or Declarative Type Mapping): Direct serialization of plain language types (structs, classes, records, or data objects) without .proto files using native field annotations, attributes, decorators, or struct tags (such as Go's `omnibuf:"1,name=city"`, Rust derive attributes, or Java/C#/Python/TypeScript decorators) to declare field numbers, JSON/Field #0 names, and wire-type overrides (wire=sint, wire=fixed32, wire=fixed64). In languages that do not have a built-in annotation or attribute feature, the implementation MUST provide an idiomatic declarative type-mapping API or descriptor table to bind field numbers, JSON names, and wire types to plain objects or structs.

11.2 Conformance Test Suite

All implementations MUST pass the official OmniBuf Conformance Test Suite across all supported usage modes. Modeled after Protocol Buffers' conformance harness, a dedicated conformance test runner binary executes a small adapter binary provided by each language implementation, feeding test requests and verifying responses to ensure:

• Wire Round-Trip & Interoperability: Deterministic decoding and re-encoding across all 16 wire types (0..15), single-byte and 2-byte field tags, overlong 2-byte container length prefixes, optional Field #0 self-describing headers, raw concatenation merges, and schema-less inspection.
• Cross-Mode Equivalence: Identical wire semantics whether a message is processed via generated plugin code, standard Protobuf types without the plugin, or annotated/mapped native classes and structs.
• Malformed Input Rejection: Strict, panic-free rejection of all invalid or adversarial wire packets defined in Section 10.