What is Apache Avro?

Apache Avro is a data serialization system developed within the Hadoop ecosystem. It uses a compact binary encoding where field names are not stored in the binary data — the schema carries all structural information, making payloads significantly smaller than JSON or XML.

The avro-c library is the reference C implementation. It supports both binary and JSON encoding from the same schema, dynamic typing (no code generation required), and the full Avro type system including records, enums, arrays, maps, unions, and fixed-width fields.

The defining feature of Avro is schema evolution: producers and consumers can use different versions of a schema as long as evolution rules are respected. This makes Avro the dominant serialization format in the Kafka ecosystem, where producers and consumers are deployed independently.

Schema Definition

Avro schemas are written in JSON. Each schema defines a type, and record schemas list their fields with names, types, and optional defaults.

// SensorReading.avsc
{
  "type":   "record",
  "name":   "SensorReading",
  "fields": [
    {"name": "sensor_id",  "type": "string"},
    {"name": "value",      "type": "double"},
    {"name": "timestamp",  "type": "long"},
    {"name": "unit",       "type": ["null", "string"], "default": null}
  ]
}

The unit field uses a union type ["null", "string"] with a default of null, making it optional. This is the standard Avro pattern for nullable fields.

Schema Evolution Rules

Avro resolves schemas at read time by matching the writer schema (used to serialize) against the reader schema (used to deserialize). The following rules govern compatibility:

Avro vs Protobuf vs Parquet

Kafka Schema Registry Integration

The avro-c library handles the Avro binary encoding. Combine it with librdkafka for the Kafka transport layer and implement the 5-byte schema ID prefix in your producer/consumer glue code.

Types

Handle Types

TAvroType Enum

TAvroCodec Enum

API Reference

Schema

Value Building

Field Setters

Field Getters

Serialization

Object Container Files

Avro Object Container Files (.avro) embed the schema and support block-level compression. They are the standard file format for Avro data at rest.

Code Example — Write and Read an .avro File

uses avro_c;

var
  Schema : TAvroSchema;
  V      : TAvroValue;
  F      : TAvroFile;
  Read   : TAvroValue;

begin
  { Parse schema inline }
  Schema := ParseSchema('{"type":"record","name":"Sensor",' +
    '"fields":[{"name":"id","type":"string"},' +
    '{"name":"temp","type":"double"}]}');

  { Open file for writing with Snappy compression }
  F := OpenFileWrite('/data/sensors.avro', Schema, acSnappy);

  { Write first record }
  V := NewValue(Schema);
  SetString(V, 'id', 'sensor-001');
  SetDouble(V, 'temp', 23.7);
  FileAppend(F, V);

  { Reuse value for second record }
  SetString(V, 'id', 'sensor-002');
  SetDouble(V, 'temp', 24.1);
  FileAppend(F, V);

  FreeValue(V);
  CloseFile(F);

  { Read back }
  F := OpenFileRead('/data/sensors.avro');
  Schema := FileSchema(F);

  repeat
    Read := FileRead(F, Schema);
    if Read <> 0 then begin
      WriteLn(GetString(Read, 'id'), ': ', GetDouble(Read, 'temp'):5:1, ' C');
      FreeValue(Read);
    end;
  until Read = 0;

  CloseFile(F);
end.