Introduction

Why we built Amza

“There are no right choices, only compromises.”

Every key-value store caters to a specific use case and approaches the problem space from a unique angle to make that use case as efficient as possible. Choosing a key-value store is about finding the right compromise that best fits your needs. After making this choice a dozen times in half as many years, we found that (1) developers often leak technology-specific, specialized behavior into their applications; and (2) changing requirements can dramatically alter an application’s performance profile. Inevitably there comes a moment when you wish you had gone with “the other guy.” The net result is either a less-than-ideal service stack, or unexpected rewrites and tedious data migrations.

Instead of following the same, tired road, we elected to build a key-value store that would let us choose our compromises up front, and then renegotiate if we changed our minds down the line in response to unanticipated client behavior or emerging requirements. By incorporating a number of industry-proven ideas related to ordered logs, election, replication and indexing, we devised a system capable of sliding between behaviors without having to migrate or alter the underlying data structure.

Moreover, supporting multiple common behaviors in a single service dramatically reduced the size and complexity of our service stack. Applications that might selectively have utilized HBase, Cassandra, Kafka and Zookeeper could instead rely exclusively on Amza. This increased code and client reuse and simplified data modeling, with the added bonus of making it easier for operational and applications developers to achieve full-system expertise.