CockroachDB: Why They Threw Away RocksDB

February 2016. A CockroachDB engineer runs a benchmark that inserts random rows. In the first second it writes about 1,944 rows a second. One minute later it is down to 105. He opens Go’s profiler to find the slow loop, and it shows almost nothing: most of the time is spent inside RocksDB, on the far side of a boundary the Go profiler cannot cross.

The usual story is that CockroachDB rewrote RocksDB in Go because Go is nicer. Both real reasons are already in that bug. In an LSM tree a delete is a write, and data only leaves the disk during compaction. And every call from Go into C++ pays a toll. Four years later CockroachDB shipped its own engine, Pebble.

Some background helps. An LSM tree buffers writes in memory, flushes them to disk as sorted files called runs, and merges those runs in the background (compaction). RocksDB is Facebook’s fork of LevelDB, an LSM engine written in C++ that many databases embed (its design). CockroachDB used it as its storage layer.

A delete is a write

LSM files are never edited in place. An update is a new entry, and the newest entry for a key wins. So a delete cannot remove anything either. It writes a new entry that says the key is gone, called a tombstone. As the engineer who found the bug put it: “When you delete a key you add a record (a deletion tombstone) that shadows the existence of the old key.”

A read that finds the tombstone first stops there and treats every older value as deleted. The old value is still on disk, underneath.

The old value only leaves when compaction merges the two runs: the tombstone wins, and the old value is not written out again. The tombstone itself can go only once nothing older is left below it to hide. In Pebble’s compaction iterator you can see both rules: a key covered by a range deletion is skipped, and point deletions are dropped only when the engine can prove nothing older sits beneath them.

An empty pizza box in the fridge works as a mental model. Anyone who looks finds the box first and knows there is nothing left, but the box still takes the shelf. It leaves on trash day, and in an LSM, trash day is compaction.

Until then a tombstone costs space, and it costs reads. A scan has to step over every tombstone in its path, because it cannot know a key is dead without looking at its newest entry. That is the loop from the cold open.

Every write is a version

CockroachDB adds one more layer on top. It never overwrites a key at all. Every write is a new version stamped with a timestamp, which is MVCC (multi-version concurrency control). The timestamp is stored right after the key, and the comparer flips it, so versions sort newest first. The 2016 post says the same: “the keys are sorted by descending timestamp.”

A read at time 4 seeks to key@4, and the first version it lands on is the answer. A delete is just one more version, written with an empty value.

Older versions stay on disk so that reads in the past still work. They go away through garbage collection, after four hours by default in current versions (it was 25 hours until v23.1 lowered it for new clusters).

The 2016 bug, step by step

Now the benchmark. Peter Mattis’s write-up traces it to a simple pattern. Every write transaction also writes a transaction record, and deletes it when it commits. Random inserts left a growing stretch of deleted records sitting side by side in the memtable.

Each new transaction began by checking whether its record existed. That seek landed in the stretch of tombstones, and the seek itself took microseconds. The slow part was what came next: the iterator stepped over deleted records one by one, looking for the next live key, “upwards of 20,000 times” by the time performance had visibly degraded.

The fix was an option RocksDB already had: iterate_upper_bound. With a bound, the scan stops at the end of the range it cares about instead of walking on to the next live key. The same benchmark afterwards started at 3,028 rows a second and was still at 2,579 after a minute.

It is like finding the snack cupboard empty. Without a bound you keep walking, through the kitchen and into the neighbor’s, until you find food somewhere. With a bound, the cupboard was the whole search.

That bug came from deleting one key at a time. The bigger problem was deleting millions at once.

Deleting a whole range

Dropping a table with a hundred million rows would take a hundred million point tombstones. The Pebble range keys RFC calls the MVCC version of that “prohibitively expensive”. So LSMs have a second kind of delete: a range tombstone, one entry that covers every key from a start to an end. (RocksDB has this too, as DeleteRange. It is not a Pebble invention.)

A read that lands inside the span sees the marker and knows the answer. A scan does not step through the covered keys either: Pebble’s merging iterator seeks the lower levels straight to the end of the span.

But the space does not come back by itself. The covered data is still on disk in every level below the marker, and only compaction can drop it, when it rewrites the files the marker covers.

A picker that didn’t look

This is where CockroachDB got hurt. The Pebble announcement says it plainly: the need for the Compactor “stems from RocksDB not taking range deletion operations into consideration in its compaction decisions.”

So in December 2017 Spencer Kimball added a compactor to CockroachDB. After each range delete it recorded a suggested compaction. Once the suggestions for a span could free more than 256 MiB, or 10% of the logical bytes in use (or 10% of the space still available), it asked RocksDB to compact that span.

Think about what that means. A database was scheduling its storage engine’s compactions from the outside. The logic it needed lived in someone else’s code, on the far side of the same boundary the profiler could not cross.

RocksDB has since fixed this part: from version 7.10 (released January 2023) a file’s size for compaction picking includes the bytes its range tombstones delete. The 2020 complaint was true then. It is not true of RocksDB today.

The cgo toll

Deletes were one cost of the language boundary. Reads paid the other, on every query.

An MVCC read is a loop. For each key it steps past the versions newer than its timestamp and takes the first one it is allowed to see, watching on the way for write intents from transactions that have not committed yet. In CockroachDB that loop ran in Go, and every step was a separate call into RocksDB.

CockroachDB is written in Go and RocksDB in C++. Go reaches C code through a bridge called cgo, and the bridge charges a toll on every crossing. In 2015 the team measured it: 171 ns for an empty call through cgo against 1.83 ns for a plain Go call, “approximately a factor of 100”. The same post is careful to add that in absolute time, 171 ns is often a perfectly acceptable price.

Cgo has also gotten cheaper since. I reran the same shape of benchmark (Go 1.25.5, Ryzen 7 3700X) and got about 52 ns against 1.8 ns, roughly 29 times instead of 100. Different hardware, so compare the ratios, not the raw nanoseconds.

One crossing is nothing. A scan that calls into RocksDB once per key pays on every key. At the ~200 ns a call the team was working with in 2016 (an illustrative figure), a million keys is a fifth of a second spent only on crossing. At today’s 52 ns it would be about a twentieth.

So in 2018 Peter Mattis moved the whole scan loop into C++: one crossing per batch of keys. The commit message reports a COUNT(*) over 5 million rows falling from 4.3 s to 2.3 s. Buying bread across the street works the same way. Crossing once per roll costs more in walking than in bread, so you buy ten and cross once.

The toll was now paid in code instead. The Pebble announcement says the team wrote “significant amounts of logic in C++ in order to avoid the performance overhead of frequent crossings from Go to C++, at times duplicating logic that already existed in Go.”

The profiler still stopped at the bridge, and values read from RocksDB were copied from C memory into Go memory. In January 2019 the team called engineering around the cgo overhead “manageable so far”, while “keeping an eye on when this calculation changes.”

There was a human cost too. From the same announcement:

While the absolute number of bugs we’ve encountered in RocksDB is modest, their severity is often high, and the urgency to fix them is frequently House Is On Fire.

and, about the 350k+ lines of C++ behind them: “doable (we’ve done it), yet hardly what could be described as a good time.” The post adds that “the barrier between Go and C++ is psychologically real.”

Starting over: Pebble

Pebble’s first commit, by Peter Mattis in June 2018, is titled “Initial fork (leveldb -> pebble)”. It started from Go’s own port of LevelDB, not from RocksDB, and borrowed RocksDB’s design where CockroachDB needed it.

A quick detour: Mattis and Spencer Kimball, two of CockroachDB’s founders, also announced the first version of GIMP together in 1995, as Berkeley students.

Only what CockroachDB needs

The README says Pebble intentionally does not aspire to include every feature in RocksDB, and lists the features it leaves out: fourteen of them, including column families, universal compaction, transactions and FIFO compaction. Column families (separate keyspaces with their own settings inside one database) and universal compaction (an alternative compaction strategy) are RocksDB features. Neither made the cut.

The size tells the same story. In 2020 the announcement counted a bit over 45k lines of code and another 45k of tests. Counting non-test Go lines myself at a 2026 commit gives about 174k (the same count gives about 52k at the end of 2020). An engine you own keeps growing too. Owning an engine is like owning a house instead of renting: nothing breaks on someone else’s schedule, but every repair and every new room is yours to build.

Deletes became first-class

The first payoff was for deletes, and it is the opposite of the Compactor.

• Delete-only compactions. In 2020 Pebble added them: a file that sits entirely under a range tombstone is dropped from the file list, by “simply constructing a manifest edit that drops the files without reading them”.
• Tombstone-weighted file sizes. The picker inflates a file’s size by the space its deletions should free (compensated size). Files full of deletes look bigger, so they are compacted sooner.
• No more Compactor. In June 2020 the team switched it off for Pebble, because Pebble’s compaction picking “is aware of range deletions and will prioritize reclamation of disk space.”

The scan loop that had moved into C++ came back as Go: a comment in it calls it a Go port of the C++ scanner. Now one profiler can see the whole path, from the query down to the disk.

Opt in, then default

Pebble shipped as an option in CockroachDB v20.1 (May 2020) and became the default in v20.2, that November, per its README. RocksDB was removed from the code in October 2020, so the next release, v21.1 in May 2021, shipped with Pebble only. Counted in releases, the two engines overlapped for about a year, not two. On the six standard YCSB workloads, the team reported that Pebble “meets or exceeds” RocksDB.

Deleting data while keeping history

Owning the engine also meant it could learn things only CockroachDB needed.

A plain range delete removes the data and its history with it. A backup, or a read in the past, cannot see what was there or when it went. The MVCC way, one empty version per key, keeps the history but costs one write for every key in the table.

So Pebble added range keys: one entry that covers a span of keys and carries a timestamp (RFC). In CockroachDB it means “everything in this span was deleted at time 6”.

A read after time 6 sees the range key and hides everything older under it. A read before time 6 sees the data as it was, so backups and history keep working. Each data block also records the range of timestamps inside it, so a read that is masking can skip a whole block when every version in it is older than the range key (the masking option is what enables it).

CockroachDB’s tech note sums it up: the same effect as point tombstones across every key, “with a constant rather than linear write/storage cost”. It arrived in version 22.2, off by default there, was switched on by default on the development branch in November 2022, and is always on from v23.1.

Who runs Pebble today, and when not to

Pick Pebble when your program is written in Go and embeds its storage. That is why go-ethereum defaults to it. It is not only Ethereum’s main client:

• Arbitrum Nitro creates new databases on Pebble by default.
• Polygon’s Bor defaults to it.
• CometBFT, the consensus engine under Cosmos, changed its default database to Pebble in v1.0.
• Bluesky’s rainbow firehose service keeps its backfill window in Pebble, and its Jetstream store uses it too.
• PingCAP’s TiDB Lightning sorts bulk imports in Pebble before ingesting into TiKV, which itself runs on RocksDB.

Nearly every one of them is a Go program that embeds its storage.

Stay with RocksDB when you need what Pebble left out, such as column families, universal compaction or transactions, or when your program is not written in Go.

What Pebble comes down to

Buffer in memory, flush sorted runs to disk, compact in the background. That last step is where deletes finally happen, so the engine that runs it has to know what a delete means. CockroachDB’s answer was a branch of its own, grown from Go’s port of LevelDB, with the tombstone handling and the language boundary it needed and little else.

Sources and further reading

Sources

1. Cockroach Labs, Adventures in performance debugging, Peter Mattis, March 2016, and issue #4196, February 2016
2. Cockroach Labs, The cost and complexity of Cgo, December 2015
3. Peter Mattis, Reimplement MVCC{Scan,Get} in C++ (PR #21395), January 2018
4. Cockroach Labs, Introducing Pebble: A RocksDB-inspired key-value store written in Go, September 2020
5. Cockroach Labs, Why we built CockroachDB on top of RocksDB, January 2019
6. Spencer Kimball, Compactor PR #20607, December 2017
7. PR #50508, compactor disabled for Pebble, June 2020
8. Jackson Owens, Pebble PR #784, delete-only compactions, July 2020
9. Pebble, compaction_picker.go, compensated file sizes
10. Pebble, README, non-goals and release timeline
11. Pebble, first commit, June 2018
12. GIMP, Prehistory, and Cockroach Labs, About
13. Cockroach Labs, PR #55509, Remove RocksDB, October 2020
14. Sumeer Bhola and Jackson Owens, Pebble range keys RFC, October 2021
15. Erik Grinaker, MVCC range tombstones tech note, July 2022
16. Cockroach Labs, Replication controls: gc.ttlseconds
17. RocksDB, PR #10734, include range tombstone bytes in compensated file size, merged December 2022
18. Source for each Pebble user: go-ethereum, Arbitrum Nitro, Polygon Bor, CometBFT, Bluesky rainbow and Jetstream, TiDB Lightning (links inline above)

Further reading

• CockroachDB, issue #6739, the ~200 ns cgo call figure (2016)
• RocksDB Wiki, DeleteRange
• Pebble, merging iterator and compaction iterator
• CockroachDB, MVCC key encoding and MVCCDelete
• Pebble, block property filters and range key masking
• Previous: How RocksDB became the SQLite of storage engines, and the basics in What is an LSM tree?