Skip to main content

Protocol Codecs

Annotate a data class, get a type-safe codec at compile time — with batch-optimized reads, round-trip testing, and zero manual field wiring.

Why Protocol Codecs?​

Hand-written protocol parsers break silently as protocols grow:

  • Field order mismatches between encode and decode go undetected until runtime
  • Missing fields in one direction cause subtle data corruption
  • No round-trip guarantee without manual test discipline
  • Tedious boilerplate — every field needs matching read/write calls in exact order

Installation​

plugins {
    id("com.google.devtools.ksp") version "<ksp-version>"
}

dependencies {
    implementation("com.ditchoom:buffer-codec:<latest-version>")
    ksp("com.ditchoom:buffer-codec-processor:<latest-version>")
}

Quick Start​

Define your wire format as a data class:

import com.ditchoom.buffer.codec.annotations.ProtocolMessage
import com.ditchoom.buffer.codec.annotations.LengthPrefixed

@ProtocolMessage
data class DeviceReport(
    val protocolVersion: UByte,       // 1 byte
    val deviceType: UShort,           // 2 bytes
    val sequenceNumber: UInt,         // 4 bytes
    val timestamp: Long,              // 8 bytes
    val latitude: Double,             // 8 bytes
    val longitude: Double,            // 8 bytes
    val altitude: Float,              // 4 bytes
    val batteryLevel: UByte,          // 1 byte
    val signalStrength: Short,        // 2 bytes
    @LengthPrefixed val deviceName: String,
)

That's it. The KSP processor generates DeviceReportCodec with decode, encode, and sizeOf — all type-safe and batch-optimized:

val buffer = DeviceReportCodec.encodeToBuffer(report)
val decoded = DeviceReportCodec.decode(buffer)
DeviceReportCodec.testRoundTrip(report) // verify correctness in one call

Note: Generated code appears after ./gradlew build. IDE autocomplete for generated codecs requires an initial build.

What you'd have to write manually​

Without code gen, this 10-field message requires writing every field in exact order — twice. One mismatch (e.g., swapping latitude/longitude, or reading a Float where you wrote a Double) silently corrupts data:

// 30+ lines of error-prone boilerplate
object DeviceReportCodec : Codec<DeviceReport> {
    override fun decode(buffer: ReadBuffer) = DeviceReport(
        protocolVersion = buffer.readUnsignedByte(),
        deviceType = buffer.readUnsignedShort(),
        sequenceNumber = buffer.readUnsignedInt(),
        timestamp = buffer.readLong(),
        latitude = buffer.readDouble(),
        longitude = buffer.readDouble(),
        altitude = buffer.readFloat(),
        batteryLevel = buffer.readUnsignedByte(),
        signalStrength = buffer.readShort(),
        deviceName = buffer.readLengthPrefixedUtf8String().second,
    )

    override fun encode(buffer: WriteBuffer, value: DeviceReport) {
        buffer.writeUByte(value.protocolVersion)
        buffer.writeUShort(value.deviceType)
        buffer.writeUInt(value.sequenceNumber)
        buffer.writeLong(value.timestamp)
        buffer.writeDouble(value.latitude)
        buffer.writeDouble(value.longitude)
        buffer.writeFloat(value.altitude)
        buffer.writeUByte(value.batteryLevel)
        buffer.writeShort(value.signalStrength)
        buffer.writeLengthPrefixedUtf8String(value.deviceName)
    }

    override fun sizeOf(value: DeviceReport): Int {
        val nameBytes = value.deviceName.encodeToByteArray().size
        return 1 + 2 + 4 + 8 + 8 + 8 + 4 + 1 + 2 + 2 + nameBytes
    }
}

The generated version is identical but also applies batch optimization — consecutive fixed-size fields are grouped into bulk reads, reducing the number of read/write calls in the hot path.

Round-Trip Testing​

Verify that encode → decode produces the original value:

@Test
fun deviceReportRoundTrip() {
    val report = DeviceReport(
        protocolVersion = 1u,
        deviceType = 42u.toUShort(),
        sequenceNumber = 1000u,
        timestamp = 1710000000000L,
        latitude = 37.7749,
        longitude = -122.4194,
        altitude = 15.5f,
        batteryLevel = 87u,
        signalStrength = -65,
        deviceName = "sensor-north-1",
    )
    val decoded = DeviceReportCodec.testRoundTrip(report)
    assertEquals(report, decoded)
}

// Or verify exact wire bytes
DeviceReportCodec.testRoundTrip(report, expectedBytes = wireBytes)

Annotation Reference​

@LengthPrefixed — Length-Prefixed Data​

Reads/writes a length prefix followed by that many bytes. Works on both String fields and @Payload type parameters. Default is 2-byte big-endian prefix.

@ProtocolMessage
data class GreetingMessage(
    @LengthPrefixed val name: String,                          // 2-byte prefix (default)
    @LengthPrefixed(LengthPrefix.Byte) val nickname: String,   // 1-byte prefix (max 255 bytes)
    @LengthPrefixed(LengthPrefix.Int) val bio: String,         // 4-byte prefix
)

// Also works with @Payload — the codec reads the prefix, then passes
// that many bytes to the caller's decode lambda
@ProtocolMessage
data class TlvMessage<@Payload P>(
    val tag: UByte,
    @LengthPrefixed val value: P,  // 2-byte length prefix + payload bytes
)

@RemainingBytes — Consume Remaining Bytes​

Reads all remaining bytes as a UTF-8 string (or passes them to a @Payload decode lambda). Must be the last constructor parameter.

@ProtocolMessage
data class LogEntry(
    val level: UByte,
    @RemainingBytes val message: String,  // reads everything after level byte
)

// With @Payload — all remaining bytes are passed to the decode lambda
@ProtocolMessage
data class RawPacket<@Payload P>(
    val header: UInt,
    @RemainingBytes val body: P,
)

@LengthFrom("field") — Length from Preceding Field​

The byte length is determined by a preceding numeric field instead of a prefix in the wire format. Works on both String fields and @Payload type parameters.

@ProtocolMessage
data class NamedRecord(
    val nameLength: UShort,
    @LengthFrom("nameLength") val name: String,  // reads nameLength bytes as UTF-8
    val value: Int,
)

// With @Payload — reads payloadLength bytes and passes to decode lambda
@ProtocolMessage
data class DataPacket<@Payload P>(
    val sequenceId: UInt,
    val payloadLength: Int,
    @LengthFrom("payloadLength") val payload: P,
)

@WireBytes(n) — Custom Wire Width​

Override the default wire size of a numeric field. Useful for protocols that use 3-byte, 5-byte, 6-byte, or 7-byte integers.

@ProtocolMessage
data class CompactHeader(
    @WireBytes(3) val length: Int,    // 3 bytes on the wire, read into Int
    @WireBytes(6) val offset: Long,   // 6 bytes on the wire, read into Long
)

Only applies to integer types. Not allowed on Float, Double, or Boolean.

@WireOrder(order) — Per-Field Byte Order​

Overrides the byte order for a single field, taking precedence over the message-level @ProtocolMessage(wireOrder = ...) setting. Use when a protocol mixes byte orders within a single message (e.g., big-endian magic number + little-endian length fields).

@ProtocolMessage(wireOrder = Endianness.Little)
data class MixedHeader(
    @WireOrder(Endianness.Big) val magic: UInt,  // overrides to big-endian
    val length: UInt,                              // inherits little-endian
)

Works with @WireBytes for custom-width fields:

@ProtocolMessage
data class BleAttHeader(
    val tag: UByte,
    @WireOrder(Endianness.Little) @WireBytes(3) val leLength: UInt,  // 3-byte little-endian
    @WireOrder(Endianness.Little) @WireBytes(2) val leFlags: Int,    // 2-byte little-endian
)

Only applies to multi-byte numeric types (Short, UShort, Int, UInt, Long, ULong, Float, Double). Single-byte types (Byte, UByte, Boolean) are unaffected.

The message-level wireOrder parameter on @ProtocolMessage sets the default for all fields. On sealed interfaces, variants inherit the parent's wireOrder unless they override it.

@UseCodec(codec) — Custom Codec Reference​

Delegates field encoding/decoding to an existing Codec object, without needing an SPI module. The referenced codec must be a Kotlin object implementing Codec<T>.

Without a length annotation — the codec reads directly from the buffer:

object RgbCodec : Codec<Rgb> {
    override fun decode(buffer: ReadBuffer) = Rgb(
        buffer.readUnsignedByte(), buffer.readUnsignedByte(), buffer.readUnsignedByte()
    )
    override fun encode(buffer: WriteBuffer, value: Rgb) {
        buffer.writeUByte(value.r); buffer.writeUByte(value.g); buffer.writeUByte(value.b)
    }
    override fun sizeOf(value: Rgb) = 3
}

@ProtocolMessage
data class ColoredPoint(
    val x: Int,
    val y: Int,
    @UseCodec(RgbCodec::class) val color: Rgb,
)
// Generated: val color = RgbCodec.decode(buffer)

With a length annotation — the codec receives a size-limited slice:

@ProtocolMessage
data class TaggedValue(
    val tag: UByte,
    val dataLength: Int,
    @UseCodec(RgbCodec::class) @LengthFrom("dataLength") val color: Rgb,
)
// Generated: val color = RgbCodec.decode(buffer.readBytes(dataLength))

Composes with @LengthPrefixed, @RemainingBytes, and @LengthFrom. This replaces the SPI CodecFieldProvider approach for most use cases — no extra Gradle module, no META-INF/services, no ServiceLoader.

@WhenTrue("expression") — Conditional Fields​

A field that is only present on the wire when a preceding Boolean field is true. The annotated field must be nullable with a default of null.

@ProtocolMessage
data class OptionalPayload(
    val hasExtra: Boolean,
    @WhenTrue("hasExtra") val extra: Int? = null,
)

@Payload + Type Parameter — Generic Payloads​

Mark a type parameter with @Payload to create a codec that is generic over its payload. A context class is generated to carry the non-payload fields needed for payload decoding.

@ProtocolMessage
data class Packet<@Payload P>(
    val version: UByte,
    val payloadLength: UShort,
    val payload: P,
)

@PacketType + Sealed Interfaces — Auto-Dispatched Decode​

Annotate a sealed interface with @ProtocolMessage and each variant with @PacketType(value) to generate a dispatch codec. The processor reads the type discriminator and delegates to the correct variant codec. Duplicate values are rejected at compile time.

@ProtocolMessage
sealed interface Command {
    @ProtocolMessage
    @PacketType(0x01)
    data class Ping(val timestamp: Long) : Command

    @ProtocolMessage
    @PacketType(0x02)
    data class Echo(
        @LengthPrefixed val message: String,
    ) : Command
}

The processor generates:

  • PingCodec and EchoCodec for each variant
  • CommandCodec that reads one byte, dispatches to the correct variant codec, and writes the type byte + variant on encode
val cmd: Command = CommandCodec.decode(buffer)
CommandCodec.encode(outputBuffer, Command.Ping(System.currentTimeMillis()))

@DispatchOn + @DispatchValue — Custom Discriminator Dispatch​

Many protocols don't use a plain byte as their type discriminator. MQTT packs the packet type into the top 4 bits of a byte. PNG puts the chunk length before the chunk type. RIFF uses 4-byte ASCII chunk IDs. @DispatchOn handles all of these by letting you define a custom discriminator type.

Step 1: Define the discriminator as a @ProtocolMessage value class (or data class) with one @DispatchValue property that returns Int:

@JvmInline
@ProtocolMessage
value class MqttFixedHeader(val raw: UByte) {
    @DispatchValue
    val packetType: Int get() = raw.toUInt().shr(4).toInt()

    val flags: UByte get() = (raw and 0x0Fu)
}

Step 2: Annotate the sealed interface with @DispatchOn and use @PacketType(value, wire) on each variant. value is what @DispatchValue returns; wire is the raw byte(s) written during encode:

@DispatchOn(MqttFixedHeader::class)
@ProtocolMessage
sealed interface MqttPacket {
    @ProtocolMessage @PacketType(value = 1, wire = 0x10)
    data class Connect(val header: MqttFixedHeader, val protocolLevel: UByte, val keepAlive: UShort) : MqttPacket

    @ProtocolMessage @PacketType(value = 2, wire = 0x20)
    data class ConnAck(val header: MqttFixedHeader, val sessionPresent: UByte, val returnCode: UByte) : MqttPacket
}

Decode flow: Read discriminator → evaluate @DispatchValue → when match on value → delegate to variant codec. The discriminator is forwarded via CodecContext so variants can access it (e.g., flags) without re-reading from the buffer.

Encode flow: Construct discriminator from wire value → encode via discriminator codec → encode variant fields.

Multi-byte discriminators​

For protocols with 4-byte type IDs (RIFF, PNG), the discriminator's inner type determines the wire width. The processor validates at compile time that wire fits the type — @PacketType(wire=300) on a UByte discriminator is a compile error.

@JvmInline
@ProtocolMessage
value class RiffChunkId(val id: UInt) {
    @DispatchValue
    val chunkType: Int get() = id.toInt()
}

@DispatchOn(RiffChunkId::class)
@ProtocolMessage
sealed interface RiffChunk {
    @ProtocolMessage @PacketType(value = 0x666D7420, wire = 0x666D7420)  // "fmt "
    data class Fmt(val chunkId: RiffChunkId, val audioFormat: UShort, ...) : RiffChunk
}

Prefix-before-type (data class discriminators)​

Some formats (PNG) put the length before the type on the wire. Use a data class discriminator that reads both fields:

@ProtocolMessage
data class PngChunkHeader(val length: UInt, val type: UInt) {
    @DispatchValue
    val chunkType: Int get() = type.toInt()
}

@DispatchOn(PngChunkHeader::class)
@ProtocolMessage
sealed interface PngChunk {
    @ProtocolMessage @PacketType(value = 0x49484452, wire = 0x49484452)  // "IHDR"
    data class Ihdr(val header: PngChunkHeader, val width: UInt, val height: UInt, ...) : PngChunk
}

When a variant has a field whose type matches the @DispatchOn discriminator, the generated codec populates it from context during decode (no double-read) and encodes it normally during encode.

Note: The lambda-based decode(buffer, ...) overload does not accept or forward an outer DecodeContext. If you need to pass runtime context through sealed dispatch, use the context-based decode(buffer, context) overload instead.

peekFrameSize — Stream Framing Without Boilerplate​

The processor automatically generates peekFrameSize(stream, baseOffset): Int? on every codec where the frame size is determinable from the wire format. This method peeks at a StreamProcessor to determine the total number of bytes the codec will read, without consuming any data.

// Generated on every codec that supports it:
object PacketCodec {
    const val MIN_HEADER_BYTES: Int = 3  // minimum bytes to determine frame size

    fun peekFrameSize(stream: StreamProcessor, baseOffset: Int = 0): Int?
    suspend fun peekFrameSize(stream: SuspendingStreamProcessor, baseOffset: Int = 0): Int?
}

This eliminates the manual peek boilerplate in streaming decode loops:

// Before: manual offset math, easy to get wrong
while (stream.available() >= 3) {
    val type = stream.peekByte(0)
    val length = stream.peekShort(1).toInt() and 0xFFFF  // hope the offset is right...
    val frameSize = 3 + length
    if (stream.available() < frameSize) break
    val msg = stream.readBufferScoped(frameSize) { PacketCodec.decode(this) }
}

// After: generated, always correct
while (stream.available() >= PacketCodec.MIN_HEADER_BYTES) {
    val frameSize = PacketCodec.peekFrameSize(stream) ?: break
    if (stream.available() < frameSize) break
    val msg = stream.readBufferScoped(frameSize) { PacketCodec.decode(this) }
}

What the generator handles:

  • Fixed-size fields — accumulated into constant offsets
  • @LengthPrefixed — peeks the prefix (1/2/4 bytes), always treated as unsigned
  • @LengthFrom("field") — peeks the referenced field's value at its known offset
  • @WhenTrue conditions — peeks the boolean (or value class property), branches to include/exclude conditional fields
  • @Payload with length annotation — same as the corresponding length type
  • @UseCodec with length annotation — same
  • @WireBytes custom widths (3, 5, 6, 7 bytes) — byte-by-byte assembly
  • Multiple variable-length fields — sequential peek with dynamic offset tracking
  • Sealed dispatch — peeks discriminator, branches per variant, delegates
  • @DispatchOn with value class — peeks inner type, constructs value class, calls @DispatchValue
  • @DispatchOn with data class — peeks all constructor parameters, constructs discriminator
  • Nested @ProtocolMessage — delegates to nested codec's peekFrameSize

When generation is skipped (the codec compiles but without peekFrameSize):

  • @RemainingBytes at top level — no length on wire
  • @UseCodec without a length annotation — opaque codec
  • @CollectionField — element iteration can't be peeked

Value Classes — Zero-Overhead Typed Wrappers​

Value classes wrapping a primitive type are supported as fields. The generated codec reads/writes the inner primitive directly with no boxing overhead:

@JvmInline
value class PacketId(val raw: UShort)

@JvmInline
value class Flags(val raw: UByte) {
    val sessionPresent: Boolean get() = raw.toInt() and 1 == 1
}

@ProtocolMessage
data class Acknowledgement(
    val flags: Flags,       // reads 1 byte, wraps in Flags
    val packetId: PacketId, // reads 2 bytes, wraps in PacketId
)

Supported Types​

Kotlin TypeDefault Wire SizeSigned
Byte1Yes
UByte1No
Short2Yes
UShort2No
Int4Yes
UInt4No
Long8Yes
ULong8No
Float4—
Double8—
Boolean1—
StringVariable—

Batch Optimization​

The generated code automatically groups consecutive fixed-size fields into single bulk reads where possible (e.g., reading a Long and splitting it into two Int fields), reducing the number of read/write calls in the hot path.

Custom Annotations (SPI)​

When the built-in annotations don't cover your protocol's encoding (e.g., variable-byte integers, repeated fields, TLV property bags), register custom annotations via the codec SPI.

The SPI lets you:

  • Define your own annotation (e.g., @VariableByteInteger)
  • Write the read/write/sizeOf functions in plain Kotlin
  • Register a CodecFieldProvider that maps the annotation to those functions
  • The generated codec calls your functions — type-safe, debuggable, no string templates

Step 1: Define Your Annotation​

package com.example.myprotocol.annotations

@Target(AnnotationTarget.VALUE_PARAMETER)
@Retention(AnnotationRetention.BINARY)
annotation class VariableByteInteger

Step 2: Write Your Functions​

package com.example.myprotocol.codec

import com.ditchoom.buffer.ReadBuffer
import com.ditchoom.buffer.WriteBuffer

fun ReadBuffer.readVariableByteInteger(): Int { /* ... */ }
fun WriteBuffer.writeVariableByteInteger(value: Int) { /* ... */ }
fun variableByteSizeInt(value: Int): Int { /* ... */ }

Step 3: Create a CodecFieldProvider​

package com.example.myprotocol.spi

import com.ditchoom.buffer.codec.processor.spi.*

class VariableByteIntegerProvider : CodecFieldProvider {
    override val annotationFqn = "com.example.myprotocol.annotations.VariableByteInteger"

    override fun describe(context: FieldContext): CustomFieldDescriptor {
        require(context.typeName == "kotlin.Int") {
            "@VariableByteInteger can only be applied to Int fields"
        }
        return CustomFieldDescriptor(
            readFunction = FunctionRef("com.example.myprotocol.codec", "readVariableByteInteger"),
            writeFunction = FunctionRef("com.example.myprotocol.codec", "writeVariableByteInteger"),
            sizeOfFunction = FunctionRef("com.example.myprotocol.codec", "variableByteSizeInt"),
        )
    }
}

Step 4: Create a KSP Provider Module​

Custom providers run at build time on the KSP processor classpath. Create a JVM-only module:

// my-protocol-ksp/build.gradle.kts
plugins {
    kotlin("jvm")
}
dependencies {
    implementation("com.ditchoom:buffer-codec-processor:<version>")
}

Register via META-INF/services/com.ditchoom.buffer.codec.processor.spi.CodecFieldProvider:

com.example.myprotocol.spi.VariableByteIntegerProvider

Step 5: Wire KSP Dependencies​

dependencies {
    ksp("com.ditchoom:buffer-codec-processor:<version>")
    ksp(project(":my-protocol-ksp"))  // your providers
}

Step 6: Use It​

@ProtocolMessage
data class FrameHeader(
    val packetType: UByte,
    @VariableByteInteger val remainingLength: Int,
)
// Generates FrameHeaderCodec with buffer.readVariableByteInteger() calls

Context Fields — Dependent Parsing​

Some custom fields need values from previously decoded fields. Use contextFields to pass them as extra arguments to your read/write functions:

class RepeatedShortsProvider : CodecFieldProvider {
    override val annotationFqn = "com.example.annotations.RepeatedShorts"

    override fun describe(context: FieldContext): CustomFieldDescriptor {
        val countField = context.annotationArguments["countField"] as String
        return CustomFieldDescriptor(
            readFunction = FunctionRef("com.example.codec", "readRepeatedShorts"),
            writeFunction = FunctionRef("com.example.codec", "writeRepeatedShorts"),
            contextFields = listOf(countField),
        )
    }
}

Generated code passes the context field:

// decode: buffer.readRepeatedShorts(count)
// encode: buffer.writeRepeatedShorts(value.items, value.count)

SPI Restrictions​

  • Cannot override built-in annotations (@LengthPrefixed, @WireBytes, etc.)
  • Custom fields break batch optimization (each custom field is read/written individually)
  • Two providers cannot register the same annotation FQN

CodecContext — Runtime Configuration​

Pass typed configuration through the entire codec chain without global state. Useful for allocator hints, compression config, security policies, or protocol version negotiation.

The Context Hierarchy​

interface CodecContext          // shared keys (version, endianness)
interface DecodeContext : CodecContext   // decode-only keys (allocator, size limits)
interface EncodeContext : CodecContext   // encode-only keys (compression, format)

Define Typed Keys​

object MyCodec : Codec<Bitmap> {
    data object AllocatorKey : CodecContext.Key<BufferAllocator>()
    data object MaxSizeKey : CodecContext.Key<Int>()

    override fun decode(buffer: ReadBuffer, context: DecodeContext): Bitmap {
        val allocator = context[AllocatorKey] ?: BufferAllocator.Default
        val maxSize = context[MaxSizeKey] ?: Int.MAX_VALUE
        require(buffer.remaining() <= maxSize) { "Payload too large" }
        return decodeBitmap(buffer, allocator)
    }
}

Pass Context at the Call Site​

val ctx = DecodeContext.Empty
    .with(MyCodec.AllocatorKey, hardwareAllocator)
    .with(MyCodec.MaxSizeKey, 1_000_000)

// Context flows automatically through sealed dispatch → @UseCodec → nested codecs
val result = TopLevelCodec.decode(buffer, ctx)

How Context Flows​

Context is forwarded automatically by generated code:

  • Sealed dispatch codecs forward context to every sub-codec
  • @UseCodec fields forward context to the referenced codec
  • Nested @ProtocolMessage fields forward context to the nested codec
  • @Payload lambdas access context via closure capture (no generated plumbing)

Codecs that don't read from context ignore it — the Codec interface defaults call the context-free overload:

interface Codec<T> {
    fun decode(buffer: ReadBuffer): T
    fun decode(buffer: ReadBuffer, context: DecodeContext): T = decode(buffer)  // default ignores
}

Shared Keys​

Keys defined with CodecContext.Key work in both DecodeContext and EncodeContext:

data object VersionKey : CodecContext.Key<Int>()

val dCtx = DecodeContext.Empty.with(VersionKey, 2)
val eCtx = EncodeContext.Empty.with(VersionKey, 2)

When to Use Context vs @Payload​

NeedUse
Caller provides decode/encode logic@Payload with lambdas
Caller provides runtime config (allocator, limits)CodecContext
Both@Payload lambda captures CodecContext via closure

Manual Codecs​

When you need full control — custom encoding logic, protocol-level optimizations, or formats that don't map to annotations — implement Codec<T> directly.

The Codec Interface​

interface Codec<T> {
    fun decode(buffer: ReadBuffer): T
    fun encode(buffer: WriteBuffer, value: T)
    fun sizeOf(value: T): Int? = null
}
  • decode reads fields from a ReadBuffer and constructs the value
  • encode writes fields to a WriteBuffer
  • sizeOf returns the encoded byte size if known, or null for variable-length encodings

Generated and manual codecs implement the same interface — they compose freely.

Simple Example​

data class SensorReading(val sensorId: UShort, val temperature: Int)

object SensorReadingCodec : Codec<SensorReading> {
    override fun decode(buffer: ReadBuffer) =
        SensorReading(buffer.readUnsignedShort(), buffer.readInt())

    override fun encode(buffer: WriteBuffer, value: SensorReading) {
        buffer.writeUShort(value.sensorId)
        buffer.writeInt(value.temperature)
    }

    override fun sizeOf(value: SensorReading) = 6
}

Length-Prefixed Strings​

Use readLengthPrefixedUtf8String() and writeLengthPrefixedUtf8String() for strings with a 2-byte big-endian length prefix:

data class ConnectMessage(val clientId: String, val username: String)

object ConnectMessageCodec : Codec<ConnectMessage> {
    override fun decode(buffer: ReadBuffer) = ConnectMessage(
        clientId = buffer.readLengthPrefixedUtf8String().second,
        username = buffer.readLengthPrefixedUtf8String().second,
    )

    override fun encode(buffer: WriteBuffer, value: ConnectMessage) {
        buffer.writeLengthPrefixedUtf8String(value.clientId)
        buffer.writeLengthPrefixedUtf8String(value.username)
    }
}

Variable-Length Integers​

readVariableByteInteger() and writeVariableByteInteger() handle variable-byte integers (1-4 bytes, max value 268,435,455):

data class PublishHeader(val topicLength: Int, val payloadLength: Int)

object PublishHeaderCodec : Codec<PublishHeader> {
    override fun decode(buffer: ReadBuffer) = PublishHeader(
        topicLength = buffer.readVariableByteInteger(),
        payloadLength = buffer.readVariableByteInteger(),
    )

    override fun encode(buffer: WriteBuffer, value: PublishHeader) {
        buffer.writeVariableByteInteger(value.topicLength)
        buffer.writeVariableByteInteger(value.payloadLength)
    }
}

Composing Codecs​

Use one codec inside another for nested structures:

data class Envelope(val version: UByte, val sensor: SensorReading)

object EnvelopeCodec : Codec<Envelope> {
    override fun decode(buffer: ReadBuffer) = Envelope(
        version = buffer.readUnsignedByte(),
        sensor = SensorReadingCodec.decode(buffer),
    )

    override fun encode(buffer: WriteBuffer, value: Envelope) {
        buffer.writeUByte(value.version)
        SensorReadingCodec.encode(buffer, value.sensor)
    }

    override fun sizeOf(value: Envelope): Int? {
        val sensorSize = SensorReadingCodec.sizeOf(value.sensor) ?: return null
        return 1 + sensorSize
    }
}

Manual Sealed Interface Dispatch​

sealed interface Message { val typeCode: Byte }

data class PingMessage(val timestamp: Long) : Message {
    override val typeCode: Byte = 0x01
}
data class DataMessage(val payload: String) : Message {
    override val typeCode: Byte = 0x02
}

object MessageCodec : Codec<Message> {
    override fun decode(buffer: ReadBuffer): Message =
        when (val type = buffer.readByte()) {
            0x01.toByte() -> PingMessage(buffer.readLong())
            0x02.toByte() -> DataMessage(buffer.readLengthPrefixedUtf8String().second)
            else -> throw IllegalArgumentException("Unknown type: $type")
        }

    override fun encode(buffer: WriteBuffer, value: Message) {
        buffer.writeByte(value.typeCode)
        when (value) {
            is PingMessage -> buffer.writeLong(value.timestamp)
            is DataMessage -> buffer.writeLengthPrefixedUtf8String(value.payload)
        }
    }
}

Compare this to the @PacketType approach above — the generated version eliminates the manual dispatch boilerplate and catches duplicate type codes at compile time.

PayloadReader​

When your protocol has a length-delimited payload section to hand off to application code:

val payloadReader = ReadBufferPayloadReader(payloadBuffer)
val id = payloadReader.readInt()
val name = payloadReader.readString(nameLength)

// Or copy/transfer
val copy = payloadReader.copyToBuffer(BufferFactory.managed())
payloadReader.transferTo(writeBuffer)

PayloadReader tracks release state — use-after-release throws IllegalStateException.

Streaming Integration​

For streaming scenarios where data arrives in chunks, use StreamProcessor to accumulate bytes, then decode with any codec:

withPool { pool ->
    val processor = StreamProcessor.create(pool)

    for (chunk in networkChannel) {
        processor.append(chunk)

        while (processor.available() >= HEADER_SIZE) {
            val headerBuffer = processor.readBuffer(HEADER_SIZE)
            val message = MessageCodec.decode(headerBuffer)
            handleMessage(message)
        }
    }
}

Handling Incomplete Data (Buffer Underflow)​

Generated codecs trust that their input buffer contains enough bytes — they don't validate remaining() before every read. This is by design: the framing layer owns the "do I have enough data?" question, not the codec.

StreamProcessor is the framing layer:

  • available() — reports total buffered bytes without consuming
  • peekInt(offset) — reads a value without consuming (e.g., to read a length field before committing)
  • readBuffer(size) — throws IllegalArgumentException if available() < size

The pattern: check available() first, then decode only when you have a complete message:

while (processor.available() >= 5) {  // 1 byte type + 4 byte length
    val type = processor.peekByte(0)
    val length = processor.peekInt(1)
    val totalSize = 5 + length
    if (processor.available() < totalSize) break  // wait for more data

    val frame = processor.readBuffer(totalSize)
    val message = MessageCodec.decode(frame)
    handleMessage(message)
}

For protocols where you want automatic refilling (e.g., reading from a socket), use AutoFillingSuspendingStreamProcessor — it calls your refill callback when more bytes are needed, eliminating the manual available() loop:

val processor = AutoFillingSuspendingStreamProcessor(delegate) {
    val chunk = socket.read()
    if (chunk == null) throw EndOfStreamException()
    delegate.append(chunk)
}
// Just decode — the processor auto-fills when it needs more bytes
val message = MessageCodec.decode(processor.readBuffer(frameSize))

Real-World Example: Zero-Copy RIFF Image Decoder​

This example decodes a custom RIFF-like image container and renders the bitmap in Jetpack Compose — without copying pixel data during parsing.

Wire Format​

┌────────────────────────────┐
│ FileHeader    (12 bytes)   │  magic("RIFF") + fileSize + format("IMG ")
├────────────────────────────┤
│ ChunkHeader   (8 bytes)    │  chunkId("meta") + chunkSize
│ ImageMetadata (variable)   │  width, height, depth, title
├────────────────────────────┤
│ ChunkHeader   (8 bytes)    │  chunkId("pxls") + chunkSize
│ Raw pixel data             │  ← zero-copy slice, never copied during parse
└────────────────────────────┘

Define the Codec Types​

Use @ProtocolMessage for all structured headers. The binary pixel payload is not a codec field — it's extracted manually with readBytes() for zero-copy.

@ProtocolMessage
data class FileHeader(
    val magic: Int,       // "RIFF" = 0x52494646
    val fileSize: UInt,   // total size after this field
    val format: Int,      // "IMG " = 0x494D4720
)

@ProtocolMessage
data class ChunkHeader(
    val chunkId: Int,     // 4-byte ASCII chunk ID
    val chunkSize: UInt,  // byte count of chunk data
)

@ProtocolMessage
data class ImageMetadata(
    val width: UInt,
    val height: UInt,
    val colorDepth: UShort,              // bits per pixel (32 = ARGB_8888)
    @LengthPrefixed val title: String,
)
Why not put pixel data in the codec?

@ProtocolMessage fields must be primitives, strings, or codec-composed types — not raw byte blobs. For binary payloads:

  • Hybrid (shown here): codec for headers, manual readBytes() for the blob — simplest approach
  • Custom SPI annotation: create a @BinarySlice("sizeField") annotation with a CodecFieldProvider that generates readBytes() calls — fully codec-managed, reusable across protocols (see Custom Annotations (SPI))
  • @Payload type parameter: for payloads that have their own structure and codec, not for raw bytes

Decode with Zero-Copy Pixel Extraction​

data class RiffImage(
    val metadata: ImageMetadata,
    val pixelData: ReadBuffer,  // zero-copy slice of the original buffer
)

object RiffImageDecoder {
    private const val MAGIC_RIFF = 0x52494646  // "RIFF"
    private const val FORMAT_IMG = 0x494D4720  // "IMG "
    private const val CHUNK_META = 0x6D657461  // "meta"
    private const val CHUNK_PXLS = 0x70786C73  // "pxls"

    fun decode(buffer: ReadBuffer): RiffImage {
        // Generated codecs parse the structured headers — type-safe, batch-optimized
        val header = FileHeaderCodec.decode(buffer)
        require(header.magic == MAGIC_RIFF && header.format == FORMAT_IMG)

        val metaChunk = ChunkHeaderCodec.decode(buffer)
        require(metaChunk.chunkId == CHUNK_META)
        val metadata = ImageMetadataCodec.decode(buffer)

        val pxlsChunk = ChunkHeaderCodec.decode(buffer)
        require(pxlsChunk.chunkId == CHUNK_PXLS)

        // Zero-copy: readBytes() returns a slice backed by the same memory
        val pixelData = buffer.readBytes(pxlsChunk.chunkSize.toInt())

        return RiffImage(metadata, pixelData)
    }
}

What's zero-copy here: readBytes() returns a ReadBuffer slice that shares the same underlying native memory as the original buffer. No pixel data is copied during parsing. The headers are tiny (28 bytes total) and parsed by the generated codec — only the multi-megabyte pixel payload gets the zero-copy treatment.

Render in Compose Multiplatform (Skia)​

Use nativeAddress to create a Skia Pixmap that wraps the buffer's memory directly — no copy:

@Composable
fun RiffBitmap(buffer: ReadBuffer, modifier: Modifier = Modifier) {
    // Hold a reference to the decoded image so the backing buffer stays alive
    val (imageBitmap, imageRef) = remember(buffer) {
        val img = RiffImageDecoder.decode(buffer)
        val w = img.metadata.width.toInt()
        val h = img.metadata.height.toInt()

        val nma = img.pixelData.nativeMemoryAccess
            ?: error("Use BufferFactory.Default for zero-copy Skia rendering")

        val info = ImageInfo(w, h, ColorType.RGBA_8888, ColorAlphaType.PREMUL)
        // Zero-copy: Pixmap wraps the buffer's native memory directly
        val pixmap = Pixmap(info, nma.nativeAddress, info.minRowBytes)
        Image.makeFromPixmap(pixmap).toComposeImageBitmap() to img
    }
    Image(bitmap = imageBitmap, contentDescription = null, modifier = modifier)
}
Buffer Lifetime

Pixmap does not own the memory — it borrows the buffer's native address. The original buffer (or pool) must stay alive while the image is in use. In the example above, imageRef keeps the RiffImage (and its backing ReadBuffer slice) alive alongside the ImageBitmap inside remember.

Android: Decompress Directly into HardwareBuffer (API 26+)​

For the fastest path to the GPU, lock a HardwareBuffer, wrap the locked pointer with BufferFactory.wrapNativeAddress, and decompress directly into GPU-accessible memory — zero intermediate buffers:

@Composable
fun RiffBitmap(buffer: ReadBuffer, modifier: Modifier = Modifier) {
    val imageBitmap = remember(buffer) {
        val img = RiffImageDecoder.decode(buffer)
        val w = img.metadata.width.toInt()
        val h = img.metadata.height.toInt()
        val pixelSize = w * h * 4

        val hwBuffer = HardwareBuffer.create(
            w, h, HardwareBuffer.RGBA_8888, 1,
            HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_CPU_WRITE_OFTEN,
        )

        // Lock HardwareBuffer → get GPU-mapped memory address (JNI)
        val gpuAddr = NativeRenderer.lockHardwareBuffer(hwBuffer)

        // Wrap the GPU memory as a PlatformBuffer — zero-copy, non-owning
        val dst = BufferFactory.wrapNativeAddress(gpuAddr, pixelSize)

        // Decompress directly from source buffer into GPU memory
        StreamingDecompressor.create(CompressionAlgorithm.Raw).use(
            onOutput = { chunk -> dst.write(chunk) }
        ) { decompress -> decompress(img.pixelData) }

        // Unlock → pixels are in GPU memory, ready to render
        NativeRenderer.unlockHardwareBuffer(hwBuffer)

        Bitmap.wrapHardwareBuffer(hwBuffer, ColorSpace.get(ColorSpace.Named.SRGB))!!
            .asImageBitmap()
    }
    Image(bitmap = imageBitmap, contentDescription = null, modifier = modifier)
}

The JNI helper is just lock/unlock — two small functions:

#include <android/hardware_buffer_jni.h>

JNIEXPORT jlong JNICALL
Java_com_example_NativeRenderer_lockHardwareBuffer(JNIEnv *env, jclass clazz, jobject hw_buffer) {
    AHardwareBuffer *ahb = AHardwareBuffer_fromHardwareBuffer(env, hw_buffer);
    void *dst = NULL;
    AHardwareBuffer_lock(ahb, AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN, -1, NULL, &dst);
    return (jlong)dst;
}

JNIEXPORT void JNICALL
Java_com_example_NativeRenderer_unlockHardwareBuffer(JNIEnv *env, jclass clazz, jobject hw_buffer) {
    AHardwareBuffer *ahb = AHardwareBuffer_fromHardwareBuffer(env, hw_buffer);
    AHardwareBuffer_unlock(ahb, NULL);
}

Writing the Container​

fun encodeRiffImage(metadata: ImageMetadata, pixels: ReadBuffer): PlatformBuffer {
    val metadataSize = ImageMetadataCodec.sizeOf(metadata)!!
    val pixelSize = pixels.remaining()
    val totalSize = 4 + 8 + metadataSize + 8 + pixelSize  // format + 2 chunk headers + data

    val buffer = BufferFactory.Default.allocate(12 + totalSize)

    // File header
    FileHeaderCodec.encode(buffer, FileHeader(
        magic = 0x52494646,
        fileSize = totalSize.toUInt(),
        format = 0x494D4720,
    ))

    // Metadata chunk
    ChunkHeaderCodec.encode(buffer, ChunkHeader(0x6D657461, metadataSize.toUInt()))
    ImageMetadataCodec.encode(buffer, metadata)

    // Pixel data chunk — bulk copy from source buffer
    ChunkHeaderCodec.encode(buffer, ChunkHeader(0x70786C73, pixelSize.toUInt()))
    buffer.write(pixels)

    buffer.resetForRead()
    return buffer
}

Copy Budget​

StepSkia (Pixmap)Android (HardwareBuffer + wrapNativeAddress)Naive (ByteArray)
Parse headers (codec)Zero-copyZero-copyZero-copy
Extract compressed data (readBytes)Zero-copy (slice)Zero-copy (slice)1 copy (readByteArray)
Decompress→ Direct buffer→ GPU memory (wrapNativeAddress)→ ByteArray
RenderZero-copy (Pixmap borrows memory)Zero-copy (GPU reads HardwareBuffer)1 copy (wrap + copyPixels)
Intermediate copies002

Both the Skia and HardwareBuffer paths achieve zero intermediate copies of pixel data. Decompressed pixels exist in exactly one place — either the Skia-wrapped Direct buffer or the GPU-mapped HardwareBuffer memory. BufferFactory.wrapNativeAddress is what makes the HardwareBuffer path possible from pure Kotlin.

Extension Functions Reference​

FunctionDescription
codec.encodeToBuffer(value)Allocate a new buffer, encode, and return it ready for reading
codec.testRoundTrip(value)Encode then decode, returning the decoded value
codec.testRoundTrip(value, expectedBytes)Same, but also verifies the wire bytes match