Serialization and encodings

Serialization means turning in-memory values into bytes for a file or a network, and parsing such bytes back. Many languages solve it centrally: in Rust you annotate a struct with #[derive(Serialize)] and a macro writes the encoder; in Python or Java the library inspects an object's fields at run time (reflection). kola has no macros, no traits and no reflection; its generics are monomorphic (Modules and the standard library). So it takes the other path. Every wire format in the standard library (bytes, hex, base64, DER, PEM, JSON, XML, TOML, YAML, HTTP/1.1 and HTTP/2 framing, DEFLATE and more) is a set of ordinary functions you can read from start to finish, with contracts sitting directly on the code that touches untrusted bytes.

Every parser starts from a byte slice, &[u8]: a pointer and a length describing a run of bytes someone else owns, much like a const uint8_t * plus a size_t in C. It is written &r [u8] when the result must point back into those bytes. What comes out has one of two shapes. A flat format such as DER or HTTP hands back plain data: small structs of usize offsets into the original buffer, called spans. Nothing is copied, and recovering the wire bytes is one slice, buf[start..end], for as long as you hold buf. A tree format such as JSON or XML allocates real nodes into a region you pass in. A region is an arena: one pool of memory that individual objects are carved from and that is released all at once, rather than freeing each node separately. So the whole parsed tree is freed at one closing brace (Memory: regions and references).

Result for the world, contracts for yourself

Wire bytes are adversarial. A certificate, a JSON body or an HTTP request line arrives from outside the program, and a malformed one is an expected outcome, not a bug. Every function here that consumes such bytes returns Result[T, E], a value that is either Ok with the answer or Err with a reason (kola has no exceptions), using a small, closed error enum. It never traps, which in kola means stopping the process on a failed check.

Accessors that work on an already-parsed value (der_content, json::span_bytes, http::header_at) follow the ordinary discipline from Contracts, traps, and tests instead: a req contract, a precondition written in the signature that the compiler either proves or checks at run time, which traps only when the caller hands in something the parser could not have produced. That is a programmer bug, not a hostile peer. The split shows in every signature: a Result return means "this reads untrusted bytes"; a plain return with a req means "this reads a value that was already validated".

bytes: fixed-width fields

Binary protocols store multi-byte numbers in a fixed byte order. Big-endian (also called network byte order) puts the most significant byte first: the 16-bit value 300, hex 0x012C, is the bytes 01 2C. Little-endian, which x86 and ARM use in memory, puts them the other way round. std::encoding::bytes reads and writes big-endian scalar fields at an offset you give it. It returns no Result at all; its contract is a req that the offset fits the buffer. A fixed-width field's size is not under the attacker's control, only its value is, so the job here is arithmetic rather than grammar. rd16/rd32/rd64 and wr16/wr32/wr64 (with 24-bit variants) read and write at an offset; hi8/lo8, hi16/lo16 and hi32/lo32 split values. std::encoding::bytes_le is the little-endian twin.

use std::encoding::bytes;

struct Point { x: u16, y: u16 }

fn write_point[r](buf: &r mut [u8], off: usize, p: Point) !{}
    req off + 4 <= buf.len
{
    bytes::wr16(buf, off, p.x);
    bytes::wr16(buf, off + 2, p.y);
}

fn read_point[r](buf: &r [u8], off: usize) -> Point !{}
    req off + 4 <= buf.len
{
    Point { x: bytes::rd16(buf, off), y: bytes::rd16(buf, off + 2) }
}

test point_wire_roundtrip {
    region r {
        let buf = alloc in r mut [0u8; 4];
        write_point(buf, 0, Point { x: 300, y: 40000 });
        let p = read_point(buf, 0);
        assert p.x == 300;
        assert p.y == 40000;
        // 300 is 0x012C, high byte first
        assert buf[0] == 0x01 && buf[1] == 0x2C;
    }
}

The same module carries the bulk operations for byte buffers: push_bytes, resize, copy (slice to slice, forward), xor, and extend_within, which appends bytes taken from earlier in the same buffer. That last one is the core step of LZ77 decompression (used by DEFLATE and gzip), which encodes repeated data as "copy n bytes from k bytes back", and the overlap is the point: with from = b.len - 1 it repeats one byte n times, which is neither memcpy nor memmove. On a Vec, prefer the methods v.extend_from_slice(s), v.resize(n, x) and v.copy_from_slice(s).

hex and base64

hex writes each byte as two characters 0-9a-f; base64 packs every three bytes into four printable characters, which is how binary data travels inside text formats such as email, JSON and PEM. Both codecs have the same shape. Encoding is infallible, since any bytes are valid input. Decoding returns a Result whose error says exactly what is wrong: HexErr is Len (an odd length) or Char (a byte outside [0-9a-fA-F]); B64Err is Len, Char or Pad (a misplaced =). Neither decoder skips whitespace or quietly accepts a second alphabet.

use std::encoding::hex;

test hex_roundtrip {
    region r {
        let msg = alloc in r b"kola";
        let enc = hex::encode(msg);
        assert enc == &b"6b6f6c61";
        match hex::decode(enc) {
            Ok(dec) => { assert dec == msg; }
            Err(_) => { assert false; }
        }
        match hex::decode(&b"zz") {
            Err(hex::HexErr::Char) => {}
            _ => { assert false; }
        }
    }
}

hex::encode_upper writes capital digits. base64 has encode/decode for the standard alphabet and encode_url/decode_url (plus encode_url_nopad) for the URL-safe one. The two alphabets differ in only two positions, and each decoder refuses the other's characters:

use std::encoding::base64;

test base64_roundtrip {
    region r {
        let msg = alloc in r b"kola!!";
        let enc = base64::encode(msg);
        assert enc == &b"a29sYSEh";
        match base64::decode(enc) {
            Ok(dec) => { assert dec == msg; }
            Err(_) => { assert false; }
        }
        // a byte outside the alphabet is an Err, not a trap
        match base64::decode(&b"@@@@") {
            Err(base64::B64Err::Char) => {}
            _ => { assert false; }
        }
    }
}

DER: pull-style TLV, and the bytes a signature covers

DER is the binary encoding under X.509 certificates and most key formats. Every DER value is three parts, called TLV: a tag byte saying what kind of value it is (an integer, a sequence, an object identifier), its length, and the content bytes. A SEQUENCE's content is more TLVs, so a certificate is a tree of them. der::der_read_tlv reads one header and returns a Tlv: five plain usize fields and a tag byte, no references, freely copyable. Walking the children of a SEQUENCE is pull-style: der_first_child, then der_more_siblings and der_next_sibling in a loop. There is no visitor callback, because kola has no closures; a parser for a fixed schema walks the known shape by hand.

A digital signature covers exact bytes, and for verification the important function is der_raw. It returns the tag, length and content exactly as they appeared on the wire. Verifying a signature over a re-encoding of the parsed value, instead of over the original bytes, is a well-known class of vulnerability. Here it cannot happen by accident, because the raw span is a slice of the original buffer.

use std::encoding::der;

test der_walks_a_sequence {
    region r {
        // SEQUENCE { OID 1.2.840.10045.2.1, NULL }: an AlgorithmIdentifier
        let buf = alloc in r [
            0x30u8, 0x0Bu8,
            0x06u8, 0x07u8, 0x2Au8, 0x86u8, 0x48u8, 0xCEu8, 0x3Du8, 0x02u8, 0x01u8,
            0x05u8, 0x00u8,
        ];
        match der::der_read_tlv(buf, 0) {
            Ok(seq) => {
                assert seq.tag == der::TAG_SEQUENCE;
                assert der::der_is_constructed(seq);
                match der::der_first_child(buf, seq) {
                    Ok(oid) => {
                        assert oid.tag == der::TAG_OID;
                        // tag, length, content, exactly as on the wire
                        let raw = der::der_raw(buf, oid);
                        assert raw.len == 9;
                        match der::der_next_sibling(buf, seq, oid) {
                            Ok(null_tlv) => {
                                assert null_tlv.tag == der::TAG_NULL;
                                assert !der::der_more_siblings(seq, null_tlv);
                            }
                            Err(_) => { assert false; }
                        }
                    }
                    Err(_) => { assert false; }
                }
            }
            Err(_) => { assert false; }
        }

        match der::der_read_tlv(&b"\x30", 0) {
            Err(der::DerErr::Truncated) => {}
            _ => { assert false; }
        }
    }
}

DerErr has six variants: Truncated, BadLength, IndefiniteLength (BER's 0x80, not legal DER), BadTag, BadInteger (empty or non-minimal) and UnexpectedTag. Helpers such as der_expect, der_oid_eq, der_int_bytes, der_bit_string_content and der_int_to_fixed[N] cover the common schema steps, and a family of der_write_* functions builds DER (der_write_sequence, der_write_integer, der_write_utc_time and so on) into a region-backed Vec[u8].

PEM: the text envelope around DER

DER is binary; the certificate and key files people open are text. PEM (RFC 7468) is the envelope: a -----BEGIN LABEL----- line, the DER bytes as base64 wrapped at 64 columns, and a matching -----END LABEL-----. std::encoding::pem encodes only, with one function:

pub fn pem_encode[r](label: &str, der: &r [u8]) -> &r mut Vec[u8] !{alloc(r)}

The label is a parameter rather than a fixed "CERTIFICATE", so the same function writes PRIVATE KEY, PUBLIC KEY, CERTIFICATE REQUEST and the other labels RFC 7468 defines. The standard library has no PEM parser.

use std::encoding::pem;

test pem_wraps_der_for_humans {
    region r {
        let der: &r [u8] = alloc in r b"fake-der-bytes-0123";
        let out = pem::pem_encode("CERTIFICATE", der);
        let want = b"-----BEGIN CERTIFICATE-----\nZmFrZS1kZXItYnl0ZXMtMDEyMw==\n-----END CERTIFICATE-----\n";
        assert out == &want;
    }
}

JSON: a tree in your region, and a writer that cannot leave a brace open

json::parse builds a JVal[r] tree whose variants are Null, Bool, Num, Str, Arr(&r Vec[&r JVal[r]]) and Obj(&r Vec[JMember[r]]), with every node allocated in the region you are in. Strings and numbers are kept as spans into the input. A number is checked against the full RFC 8259 grammar, so a valid document is never rejected for containing 1.5e3, but it is not converted to anything. num_i64 is the numeric accessor, and it is strict: a fraction, an exponent or a value outside i64 answers Err(NotInt) instead of truncating. obj_get searches an ordered member list and the last matching key wins; duplicate keys are legal JSON and the tree keeps them.

parse takes its nesting limit as a parameter and answers Err(JsonErr::TooDeep) past it. The recommended value is json::MAX_DEPTH (32), chosen to stay well clear of the interpreter's own stack limit on deeply nested input. A service parsing hostile input passes that or a smaller number of its own. JsonErr has thirteen variants, from Eof and BadNumber to Trailing and NotInt.

use std::encoding::json;

test json_tree_in_a_region {
    region r {
        let doc = b"{\"name\": \"kola\", \"stable\": false, \"stars\": 42}";
        match json::parse(&doc, json::MAX_DEPTH) {
            Ok(v) => {
                match json::obj_get(&doc, v, &b"stars") {
                    Some(n) => {
                        match json::num_i64(&doc, n) {
                            Ok(i) => { assert i == 42; }
                            Err(_) => { assert false; }
                        }
                    }
                    None => { assert false; }
                }
                match json::obj_get(&doc, v, &b"name") {
                    Some(s) => {
                        match json::str_span(s) {
                            Some(sp) => { assert json::span_bytes(&doc, sp) == &b"kola"; }
                            None => { assert false; }
                        }
                    }
                    None => { assert false; }
                }
                assert json::obj_get(&doc, v, &b"missing").is_none();
            }
            Err(_) => { assert false; }
        }

        // a trailing comma is not RFC 8259 grammar
        match json::parse(&b"{\"a\": 1,}", json::MAX_DEPTH) {
            Err(_) => {}
            Ok(_) => { assert false; }
        }
    }
}

A string span holds the raw, still-escaped bytes; json::str_unescape decodes them into a buffer. arr_len, arr_get and obj_len walk arrays and objects, and write_val serializes a parsed tree back out.

For producing JSON there is a streaming writer, which writes the document piece by piece into a buffer instead of building a tree first. json::writer() returns a JsonW, which is a resource. You call obj_begin, key, str_val, u64_val and the rest; the writer inserts commas and colons and escapes strings. Making it a resource closes the one bug a plain writer struct cannot: the forgotten closing brace. The normal way to consume the writer is json::finish(w), whose req demands a complete document, and a function that forgets to consume it at all does not compile. On an error path, json::discard(w) ends it explicitly.

use std::encoding::json;

fn render[o](out: &o mut Vec[u8]) !{alloc(o)} {
    let mut w = json::writer();
    json::obj_begin(&mut w, out);
    json::key(&mut w, out, &b"name");
    json::str_val(&mut w, out, &b"kola");
    json::key(&mut w, out, &b"stars");
    json::u64_val(&mut w, out, 42);
    json::obj_end(&mut w, out);
    json::finish(w);
}

test writer_emits_a_whole_document {
    region r {
        let out: &r mut Vec[u8] = alloc in r mut Vec::new();
        render(out);
        assert out == &b"{\"name\":\"kola\",\"stars\":42}";
    }
}

Delete the json::finish(w); line and the checker answers with a card §2 error: "resource w is not consumed on this path".

XML: the same tree shape

std::encoding::xml repeats the JSON shape (elements, attributes and text as region-allocated data) over a deliberately narrow grammar. A DOCTYPE or DTD is refused outright with XmlErr::Doctype. A DTD can define entities that expand into other entities, so a few hundred bytes can expand into gigabytes (the "billion laughs" attack); with no DTD support, that attack has nothing to work with. Namespaces are not resolved; : is an ordinary name byte. Nesting is bounded by xml::MAX_DEPTH (64) and answers TooDeep beyond it, so hostile input gets an Err and never exhausts the stack.

use std::encoding::xml;

test xml_walks_an_element_tree {
    region r {
        let doc = b"<pack name=\"gossip\"><version>3</version></pack>";
        match xml::parse(&doc) {
            Ok(root) => {
                assert xml::span_eq(&doc, root.name, &b"pack");
                match xml::attr_get(&doc, root, &b"name") {
                    Some(sp) => { assert xml::span_eq(&doc, sp, &b"gossip"); }
                    None => { assert false; }
                }
                match xml::child_elem(&doc, root, &b"version") {
                    Some(v) => { assert v.kids.len == 1; }
                    None => { assert false; }
                }
            }
            Err(_) => { assert false; }
        }

        match xml::parse(&b"<!DOCTYPE x><a/>") {
            Err(xml::XmlErr::Doctype) => {}
            _ => { assert false; }
        }
    }
}

xml::unescape decodes a text or attribute span, and write_elem and write_node serialize a tree.

From text to bytes and back

Two pieces of the language make these codecs comfortable to call. A byte-string literal b"GET / HTTP/1.1\r\n" is an ordinary [u8; N] array of static data, so &b"..." is a &heap [u8] that fits any region. Its content is printable ASCII plus the escapes \n \t \r \\ \" \0 \xNN. And every str has a .bytes view: the same UTF-8 bytes as &[u8], in the same region, with nothing copied.

use std::encoding::hex;

test byte_string_and_str_bytes_bridge {
    region r {
        let wire = b"kola";
        let name = "kola";
        // a str's bytes and a byte string compare element by element
        assert name.bytes == &wire;
        let enc = hex::encode(name.bytes);
        assert enc.len == 8;
    }
}

Going the other way is narrower. Every codec on this page hands back bytes or a span, never a str, and std::text::utf8::to_str is the only way to make a str out of bytes. kola has no cast that could do it.

pub fn to_str[r](b: &r [u8]) -> Option[&r str] !{}

to_str validates strict UTF-8 (overlong forms, surrogates and out-of-range code points are refused) and answers None on failure. The result is a view of b: nothing is allocated, and the str lives exactly as long as b. For text that must outlive its buffer, utf8::to_str_copy copies it into a region you choose. A NUL byte is an ordinary character: a str is a pointer and a length, so "a\0b".len is 3. Where a str crosses into a C-string API such as a path, the operation refuses it as Invalid rather than act on a truncated string.

use std::text::utf8;

test to_str_is_the_only_door {
    region r {
        let good: &r [u8] = alloc in r b"kola";
        match utf8::to_str(good) {
            Some(s) => { assert s == "kola"; }
            None => { assert false; }
        }

        // a lone continuation byte is not UTF-8
        let bad: &r [u8] = alloc in r [0x80u8];
        assert utf8::to_str(bad) == None;

        // U+0000 is valid UTF-8 and an ordinary character
        let nul: &r [u8] = alloc in r [107u8, 0u8, 108u8];
        match utf8::to_str(nul) {
            Some(s) => { assert s.len == 3; }
            None => { assert false; }
        }
    }
}

HTTP/1.1: framing without sockets

std::encoding::http parses request and response heads and chunked bodies for RFC 9110 and 9112. It holds no capability and knows nothing about sockets. parse_request_head and parse_response_head take a byte slice and two budgets, max_headers and max_head_bytes (http::MAX_HEAD_BYTES is 8192), and hand back the head plus how many bytes it used. A program that wants a server connects the codec to a real socket itself.

Being capability-free is what makes the codec fit a real socket. TCP delivers a stream of bytes, not messages, so one read may hand you half a request, or one and a half. So parse_request_head answers Result[Option[(&r RequestHead[r], usize)], HttpErr]. Err means the input is malformed. Ok(None) means it is well-formed so far but incomplete: call again when more bytes arrive. Ok(Some((head, consumed))) means the head is complete. The parser keeps no state between calls; it reparses the buffer from byte 0 each time.

use std::encoding::http;
use std::encoding::http::HttpVersion;

test http_incremental_parse {
    region r {
        let rq = b"POST /upload HTTP/1.1\r\nHost: kola.dev\r\nContent-Length: 5\r\n\r\nhello";

        // First read: only the request line has arrived.
        match http::parse_request_head(rq[0..24], 16, http::MAX_HEAD_BYTES) {
            Ok(opt) => {
                match opt {
                    None => {}
                    Some(_) => { assert false; }
                }
            }
            Err(_) => { assert false; }
        }

        // Second read: the whole head is here.
        match http::parse_request_head(&rq, 16, http::MAX_HEAD_BYTES) {
            Ok(opt) => {
                match opt {
                    Some(t) => {
                        let head = t.0;
                        let consumed = t.1;
                        assert head.version == HttpVersion::Http11;
                        match http::determine_request_body_framing(&rq, head) {
                            Ok(bf) => {
                                match bf {
                                    http::BodyFraming::Fixed(n) => { assert n == 5u64; }
                                    _ => { assert false; }
                                }
                            }
                            Err(_) => { assert false; }
                        }
                        // everything after the head is body
                        assert rq[consumed..].len == 5;
                    }
                    None => { assert false; }
                }
            }
            Err(_) => { assert false; }
        }
    }
}

BodyFraming is NoBody, Fixed(u64), Chunked or UntilClose. The codec also closes a known attack. In request smuggling, an attacker sends a request whose length two servers read differently (one trusts Content-Length, the other Transfer-Encoding), so a proxy and the server behind it disagree about where one request ends and the next begins. RFC 9112 §6.3 calls this ambiguous, and the codec refuses it: a request that carries both Transfer-Encoding: chunked and Content-Length is Err(ConflictingFraming), and there is no path that quietly prefers one header. HttpErr has ten variants in all. On the writing side, write_request_line, write_status_line, write_header, end_headers and the chunk writers build a head into a Vec[u8].

The whole family

The decompressors take a max_out bound on the output they will produce, so a small hostile input cannot expand into unbounded memory.

What is deliberately absent

There is no derive. kola has no macros, so a codec is as many lines as the format has fields, all of them ordinary function bodies. There is no serde-style framework either: with no traits there is no Serializer interface for formats to plug into, and each codec is its own module with its own error enum, so a format-specific decision (JSON's numbers kept as spans, DER's raw bytes for signatures) is not hidden behind a shared abstraction. And there is no runtime reflection: every field access in every codec is a name written in the source and checked by the compiler.

The reason is review. Writing a codec is cheap; a model can produce rd16 and wr16 calls from an RFC in seconds. Checking that a codec agrees with the wire format, byte for byte, is the expensive part, and it is exactly the part a derive macro or a reflective serializer moves out of sight. Written out, with the compiler holding every offset to a contract, that code stays where a reviewer reads it.