How Discord Scaled to 11M+ Concurrent Users: Why Cassandra Failed & ScyllaDB Won

In 2015, Discord launched on a single MongoDB replica set. Within months, as voice and text chat skyrocketed across gaming communities, MongoDB began choking under the write load: data and indexes exceeded RAM, and page faults skyrocketed.

In 2017, Discord migrated its core message storage to Apache Cassandra. Cassandra seemed like the ideal fit: a distributed, masterless, LSM-tree based NoSQL database optimized for heavy sequential writes and linear horizontal scaling.

Despite massive engineering investments into JVM tuning (CMS, G1GC, ZGC) and OS page cache optimization, Cassandra began exhibiting catastrophic tail latencies. In a real-time chat application where users expect instant message delivery, Discord observed P99 latencies spiking past 1,000ms, and in worst cases, 5,000ms+ timeouts.

Three structural flaws caused the breakdown at scale:

• Stop-the-World JVM Garbage Collection: With multi-gigabyte heaps, Java GC cycles periodically paused threads for 1 to 3 seconds. Nodes froze, gossip heartbeats failed, and neighboring nodes erroneously marked them as down.
• Tombstone Accumulation: When users or automated bots delete messages or channels, Cassandra doesn't immediately remove the row from disk. Instead, it writes a 'tombstone' marker. Reading through channels with high bot activity required scanning tens of thousands of tombstones, triggering severe read timeouts.
• SSTable Compaction Storms: As writes flooded the LSM-trees, Cassandra background compaction threads starved read threads of disk IOPS and CPU, driving cascading node degradation.

Instead of rewriting their entire data access layer, Discord evaluated ScyllaDB—a complete reimplementation of Apache Cassandra in C++20 built on top of the Seastar asynchronous framework.

ScyllaDB fundamentally eliminates the performance bottlenecks of Cassandra through three architectural principles:

In wide-column stores, unbounded partition sizes are fatal. If a Discord channel has 50 million messages over 5 years, putting all messages into a single partition `(channel_id)` creates a multi-gigabyte row that overwhelms memory buffers and destroys read performance.

Discord solved this by bucketing messages into discrete time slices using their 64-bit Snowflake ID generation scheme. A Discord Snowflake ID encodes an epoch timestamp in its most significant 42 bits.

By taking the timestamp component and grouping messages into roughly 10-day buckets, Discord defined a composite primary key that guarantees no partition ever exceeds ~100MB, allowing ScyllaDB to query message history with pinpoint accuracy.

CREATE TABLE messages (
    channel_id bigint,
    bucket int, -- (timestamp >> 22) / (10 * 86400 * 1000)
    message_id bigint,
    author_id bigint,
    content text,
    attachments list<text>,
    PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

-- Fetching the latest 50 messages in a channel:
SELECT * FROM messages 
WHERE channel_id = 987654321 
  AND bucket = 204 
ORDER BY message_id DESC 
LIMIT 50;

The migration delivered startling efficiency improvements across Discord's infrastructure:

1. Node Reduction: Discord reduced their massive cluster footprint from 177 beefy Cassandra nodes down to just 72 ScyllaDB nodes, saving millions in annual cloud compute costs.

2. Latency Elimination: Tail P99 read latency dropped from unstable 1,000ms+ spikes to a rock-solid 15ms. Even P99.9 latencies remained below 30ms during global peak gaming hours.

3. Operational Simplicity: No more manual GC tuning, no more tombstone scan crashes, and automated self-throttling compactions that never starve active traffic.