OmniBuf
for Go.

Q: When should I use Opaque lazy views instead of Unmarshal?

Use Unmarshal when your handler reads most fields of a message. Use generated Opaque views in routers, proxies, and ingestion pipelines that only inspect a header field or update a single value before forwarding the packet. Getters hop directly to the target field with zero heap allocations, and setters splice updated fields onto the raw byte slice without decoding the rest of the message.

Q: How do I enable Field #0 inline field names or decode without a schema?

Pass omnibuf.MarshalOptions{SelfDescribing: true} when marshaling any struct or .pb.go message to prepend Field #0 with the root and nested field-name tables. To inspect a packet without a target struct or schema file, call omnibuf.DecodeBlind(pkt). It returns a map keyed by field numbers, or by field names automatically when Field #0 is present.

Q: What does the zeroCopy flag do, and how do I pool buffers?

The second argument to UnmarshalOmniBuf(pkt, zeroCopy) controls string allocation. Passing false copies strings onto the heap so the packet buffer can be reused right away; passing true points string headers directly into the packet buffer for zero-allocation reads. On the encode path, pass a pooled slice via MarshalOmniBufAppend(buf[:0]) to serialize messages without allocating a new buffer.

Q: How does protoc-gen-omnibuf compare to vtprotobuf?

vtprotobuf speeds up Go Protobuf by generating unrolled varint loops and a separate size-calculation pass before writing backwards into the buffer, because Protobuf requires knowing every sub-message length up front. OmniBuf reserves a 2-byte length slot and fills it in after writing, and reads integers in one step instead of a byte-by-byte loop. As a result, protoc-gen-omnibuf runs over 50% faster on Marshal and over 2x faster on Read-Modify-Write while generating less than half the code.

Reference runtime and code generator for Go: serialize plain Go structs, drop in on existing .pb.go types, or generate single-pass codecs and lazy Opaque views — see the Go benchmarks below.

GitHub Repository →pkg.go.dev Reference →

Install Go Runtime

$ go get github.com/sudorandom/omnibuf

Three ways to use OmniBuf in Go.

Use plain Go structs with struct tags, pass existing .pb.go types with no build changes, or generate single-pass methods and lazy Opaque views with the compiler plugin.

Plain Go structs

Add omnibuf:"..." struct tags to exported fields and pass a struct pointer to omnibuf.Marshal and omnibuf.Unmarshal. On first use, OmniBuf caches each field's memory offset and wire type so subsequent calls read and write fields directly without per-call reflection:

1..65535 sets the wire field number (untagged fields are ignored).
name=... sets the field name written to Field #0 (defaults to the Go field name).
wire=... overrides the inferred wire type (uint, sint, fixed32, fixed64, string, bytes).
omitempty skips encoding when the field holds its zero value.

Plain Go Structs

type Peer struct {
	ID       uint64            `omnibuf:"1,name=peer_id"`
	Router   string            `omnibuf:"2,name=router,omitempty"`
	Active   bool              `omnibuf:"3,name=active"`
	Latency  uint32            `omnibuf:"4,wire=fixed32"`
	Prefixes []uint32          `omnibuf:"5,name=prefixes"`
	Labels   map[string]string `omnibuf:"6,omitempty"`
}

pkt, err := omnibuf.Marshal(&Peer{
	ID:     64512,
	Router: "rtr-01",
	Active: true,
})
var out Peer
err = omnibuf.Unmarshal(pkt, &out)

Existing .pb.go structs

Already generating Go code with protoc-gen-go? Pass your existing .pb.go structs directly to omnibuf.Marshal and omnibuf.Unmarshal with no extra plugin step.

OmniBuf reads the existing protobuf:"..." struct tags once per type, builds a cached pointer-offset table, and runs ~2.5x to 3x faster than standard Go Protobuf (and 4x to 11x faster than JSON and MessagePack) with zero heap allocations on reused structs.

Existing .pb.go Structs

// Encode any existing *pb.Peer (.pb.go) struct with zero extra codegen:
pkt, err := omnibuf.Marshal(msg)

// Decode back into a reused or new *pb.Peer struct:
var out pb.Peer
err = omnibuf.Unmarshal(pkt, &out)

Generated code with protoc-gen-omnibuf

Add protoc-gen-omnibuf to your buf.gen.yaml (supports both proto3 and Editions 2023/2024 schemas). The plugin attaches single-pass MarshalOmniBufAppend and UnmarshalOmniBuf methods plus lazy Opaque views directly onto your .pb.go structs.

Generated codecs run ~6x faster on encode and ~8x faster on decode than standard Go Protobuf, and beat vtprotobuf by over 50% while generating less than half the code.

buf.gen.yaml

version: v2
plugins:
  - local: protoc-gen-go
    out: gen
    opt: paths=source_relative
  - local: protoc-gen-omnibuf
    out: gen
    opt: paths=source_relative

Install Plugin & Generate

$ go install github.com/sudorandom/omnibuf/cmd/protoc-gen-omnibuf@latest
$ buf generate

Generated Single-Pass Methods

// Allocate a new slice, or append to a pooled buffer (buf[:0]):
pkt := msg.MarshalOmniBufAppend(nil)

// Decode back into *pb.Peer (pass true to alias strings zero-copy):
var out pb.Peer
err := out.UnmarshalOmniBuf(pkt, false)

Go benchmarks.

Measured on an 11-field message with scalars, strings, packed arrays, and nested messages. Shorter bars = Faster.

Full Message Encode Speed

This benchmark measures how long it takes to serialize an 11-field message into a reused byte buffer. OmniBuf performs extremely well with encoding because it writes forward in a single pass, reserves 2-byte length slots instead of pre-calculating sub-message sizes, and encodes integers without byte-by-byte varint loops.

OmniBuf (Generated)

1.0x (base)94 ns

OmniBuf (Go Struct Tags)

2.5x slower232 ns

Protobuf

6.0x slower561 ns

FlatBuffers

6.0x slower562 ns

Cap'n Proto

10.8x slower1016 ns

MessagePack

16.9x slower1590 ns

JSON (encoding/json)

18.1x slower1705 ns

JSON (encoding/json/v2)

18.6x slower1753 ns

Full Message Decode Speed

This benchmark measures how long it takes to decode the full 11-field message back into a struct. OmniBuf performs extremely well with decoding because the trailing zeros of the first byte reveal the exact integer width for a single-step read, booleans require zero value bytes, and arrays are decoded from a single length-prefixed block.

OmniBuf (Generated)

1.0x (base)120 ns

FlatBuffers

1.4x slower170 ns

OmniBuf (Go Struct Tags)

2.9x slower343 ns

Cap'n Proto

6.4x slower772 ns

Protobuf

8.0x slower960 ns

MessagePack

18.2x slower2185 ns

JSON (encoding/json/v2)

23.1x slower2778 ns

JSON (encoding/json)

32.4x slower3891 ns

Read a Single Nested Field

This benchmark is about extracting a string from a nested message without decoding the rest of the message. Although OmniBuf isn't optimized for single-field random access like pointer-table formats (FlatBuffers and Cap'n Proto), it still performs quite well by hopping over preceding field headers in a single addition per field.

Cap'n Proto (Unframed Shared Memory)

15.2x faster5 ns

FlatBuffers

5.8x faster13 ns

OmniBuf

1.0x (base)76 ns

Cap'n Proto (Framed Packet)

2.9x slower221 ns

MessagePack

7.8x slower590 ns

Protobuf

12.9x slower983 ns

Benchmark Environment

Hardware & Runtime

CPU: Apple M1 Pro (darwin/arm64)

Compiler: Go 1.27.1

Command: just bench

All benchmarks serialize and deserialize an identical 11-field message containing unsigned and ZigZag-signed integers, booleans, IEEE-754 floats, string lists, packed integer arrays, a 6-field nested sub-message, and a 4-element repeated sub-message list.

Source code for the harnesses and codec implementations is in omnibuf_bench_test.go, opaque_table_bench_test.go, jsonv2_bench_test.go, and peer_codecs_test.go. Raw output from GOEXPERIMENT=jsonv2 go test -bench=. -benchmem is checked into testdata/benchmarks.txt.

Go questions.

When should I use Opaque lazy views instead of Unmarshal?

Use Unmarshal when your handler reads most fields of a message. Use generated Opaque views in routers, proxies, and ingestion pipelines that only inspect a header field or update a single value before forwarding the packet.

Getters hop directly to the target field with zero heap allocations, and setters splice updated fields onto the raw byte slice without decoding the rest of the message.

Lazy Read & In-Place Update

view := pb.PeerOpaque(pkt)
router, ok := view.Router() // Zero-alloc header hop

// Update field #3 (Active) in-place without decoding the message:
updated := view.SetActive(false)

How do I enable Field #0 inline field names or decode without a schema?

Pass omnibuf.MarshalOptions{SelfDescribing: true} when marshaling any struct or .pb.go message to prepend Field #0 with the root and nested field-name tables.

To inspect a packet without a target struct or schema file, call omnibuf.DecodeBlind(pkt). It returns a map keyed by field numbers, or by field names automatically when Field #0 is present.

Field #0 & Schema-Less Decode

// Embed Field #0 (SchemaMeta) inline with the packet:
pkt, err := (omnibuf.MarshalOptions{SelfDescribing: true}).Marshal(&peer)

// Decode any OmniBuf packet without a schema (uses Field #0 names if present):
fields, err := omnibuf.DecodeBlind(pkt)

What does the zeroCopy flag do, and how do I pool buffers?

The second argument to UnmarshalOmniBuf(pkt, zeroCopy) controls string allocation. Passing false copies strings onto the heap so the packet buffer can be reused right away; passing true points string headers directly into the packet buffer for zero-allocation reads.

On the encode path, pass a pooled slice via MarshalOmniBufAppend(buf[:0]) to serialize messages without allocating a new buffer.

Buffer Reuse & Zero-Copy Strings

// Reuse an existing byte slice capacity across encodes (0 allocs):
buf = msg.MarshalOmniBufAppend(buf[:0])

// Pass zeroCopy=true when pkt outlives out to alias strings directly:
err := out.UnmarshalOmniBuf(buf, true)

How does protoc-gen-omnibuf compare to vtprotobuf?

vtprotobuf speeds up Go Protobuf by generating unrolled varint loops and a separate size-calculation pass before writing backwards into the buffer, because Protobuf requires knowing every sub-message length up front.

OmniBuf reserves a 2-byte length slot and fills it in after writing, and reads integers in one step instead of a byte-by-byte loop. As a result, protoc-gen-omnibuf runs over 50% faster on Marshal and over 2x faster on Read-Modify-Write while generating less than half the code.

Explore the wire layout, format comparisons & specification

How It Works →TypeScript / JS →Wire Specification →

OmniBuffor Go.

Three ways to use OmniBuf in Go.

Plain Go structs

Existing .pb.go structs

Generated code with protoc-gen-omnibuf

Go benchmarks.

Benchmark Environment

Go questions.

When should I use Opaque lazy views instead of Unmarshal?

How do I enable Field #0 inline field names or decode without a schema?

What does the zeroCopy flag do, and how do I pool buffers?

How does protoc-gen-omnibuf compare to vtprotobuf?

OmniBuf
for Go.