Unveiling the Future: Disaggregated Systems and the Evolution of Cloud Computing
The Cloud's Evolution: A Shift Towards Efficiency and Flexibility
In a recent talk at QCon SF 2025, Murat Demirbas highlighted an exciting architectural transformation in cloud computing. Traditional cloud setups, with their tightly coupled compute and storage, are being challenged by the industry's need for more dynamic and cost-effective solutions.
The Shared-Nothing Dilemma
Traditional cloud architectures, often referred to as 'shared-nothing', face a fundamental mismatch. Compute resources are volatile and expensive, while storage is stable and cheap. Coupling these two leads to inefficiencies, forcing users to pay for resources they may not fully utilize, which goes against the cloud's promise of flexible, pay-per-use scalability.
The Rise of Disaggregation
Demirbas emphasizes that the industry is rapidly moving towards disaggregated architectures. This approach decouples compute, storage, and even logging, allowing for independent scaling and simplified operations. Cloud databases like Amazon Aurora, Google AlloyDB, Microsoft Socrates, and Snowflake are leading this trend, offering a more efficient and straightforward approach to cloud management.
Key Benefits of Disaggregation
- Elastic Scalability: Compute nodes can adjust their capacity instantly without moving vast amounts of data.
- Fault Isolation: Failures in isolated components, like a compute node, allow for faster recovery.
- Simplified Operations: Shared storage simplifies complex maintenance and backup processes.
- Pay-Per-Use Models: Customers pay only for what they use, optimizing costs.
Enablers of This Transformation
Advancements in high-speed networking, such as Remote Direct Memory Access (RDMA) and Compute Express Link (CXL), have made this shift possible. These technologies provide the low-latency connections needed to replace traditional local I/O, enabling the disaggregation of compute and storage.
A New Approach to Database Tasks
Disaggregation fundamentally changes how core database operations are executed. In systems like Amazon Aurora, the primary compute node sends only the redo log to a storage quorum. The storage nodes then create the database state, allowing the compute node to acknowledge writes quickly. This approach shifts the burden of replication and consensus away from the compute cluster, simplifying the process.
Connecting to Distributed Systems Theory
Demirbas draws a parallel between these modern designs and Leslie Lamport's Paxos protocol. This connection suggests that disaggregation creates a natural separation of database roles, with compute nodes acting as proposers, the shared log/storage quorum as acceptors, and the shared page store as learners. This structural separation allows for graceful scaling, failure, and recovery.
Trade-offs and Innovations
While disaggregation offers many benefits, it also shifts the performance bottleneck from the CPU to the network. To overcome this, aggressive data buffering and prefetching techniques are employed. Despite this challenge, disaggregation is pioneering new frontiers in database design, such as:
- Pushdown Computation: Executing complex queries directly at the storage nodes to minimize data movement.
- Memory Disaggregation: Using CXL to separate memory from compute, allowing for independent scaling.
- Unification of Workloads: Merging transactional and analytical workloads on shared storage, as seen in Google AlloyDB.
A Thought-Provoking Conclusion
Demirbas concludes with a quote from Ralph Waldo Emerson: "There are many methods, but few principles. If you master the principles, you can choose your methods." He emphasizes the importance of understanding distributed systems principles as new hardware and failure modes emerge. This knowledge will be crucial in designing the future of cloud data, where databases self-assemble and data centers become giant, interconnected computers.
This talk by Murat Demirbas offers a fascinating glimpse into the future of cloud computing, where efficiency, flexibility, and innovation are driving the industry forward.
