What is compaction?

Compaction is where you take a chunk of messages and replace them with a single message.

For example, one of the DriftDB demos is a counter (https://demos.driftdb.com/counter). State is synchronized by putting increment/decrement events into a stream. When a new user connects, their client get all the messages in the stream, plays them back, and arrives at the same state as everyone else.

If that’s all we did, over time, the stream would grow unruly. It would take ages to load the page because we’d have to load every state change. But we only really care about a single numeric value. Compaction takes a chunk of messages that look like this:

    {"apply":"increment"}
    {"apply":"increment"}
    {"apply":"decrement"}
    {"apply":"increment"}

And replaces them with a message that looks like this:

    {"reset":2}

DriftDB doesn’t know how to compute the compaction, it relies on clients to do that. When a client does something that increases the length of the stream, the server sends back the new length of the stream, so that the client can decide whether to compact it (i.e. if it passes some threshold).

The important part that I haven’t seen elsewhere is that when a client compacts the stream, it includes a sequence number of the last message that’s part of the compaction. The server will preserve messages greater than that sequence number, since they are not part of the compaction.