Great overview of techniques for testing distributed systems. Additional materials for this talk could be found in this Github repo
Colin Skott shares his viewpoint from academia on testing distributed systems.
- Technologies for Testing Distributed Systems, Part I
- See also post Distributed Systems Testing: The Lost World by Crista Lopes
- Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems -- Great overview of how even simple testing can help a lot, you just need right focus.
- What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems -- study of actual bugs in different popular distributed systems (Hadoop MapReduce, HDFS, HBase, Cassandra, ZooKeeper and Flume)
- TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems -- comprehencive taxonomy of bugs in distributed systems (Cassandra, Hadoop MapReduce, HBase, ZooKeeper)
- Redundancy does not imply fault tolerance: analysis of distributed storage reactions to single errors and corruptions -- study of several distributed systems (Redis, ZooKeeper, MongoDB, Cassandra, Kafka, RethinkDB) on how fault tolerant they are to data corruption and read/write errors
- Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
- Velocity 2012: Richard Cook, "How Complex Systems Fail"
- How Complex Systems Fail
- Jepsen is used by Microsoft CosmosDB
- Kyle Kingsbury on InfoQ
- CockroachDB beta-20160829
- Jepsen: RethinkDB 2.1.5
- Jepsen: RethinkDB 2.2.3 reconfiguration
- Aphyr's Jepsen posts
- Jepsen Talks
- Call me maybe: Jepsen and flaky networks
- Test scaffolding -- start page if you want to write Jepsen test
- Comparisons of Alloy and Spin
- Verdi: Formally Verifying Distributed Systems
- Network Semantics for Verifying Distributed Systems
- The verification of a distributed system By Caitie McCaffrey also podcast and talk on InfoQ.com and accompanying materials on GitHub and a slidedeck
- Fuzzing Raft for Fun and Publication
- DNS parser, meet Go fuzzer
- Fuzz Testing with afl-fuzz (American Fuzzy Lop)
- Randomized testing for Go and talk on this tool GopherCon 2015: Dmitry Vyukov - Go Dynamic Tools
- Simple guided fuzzing for libraries using LLVM's new libFuzzer
- LibFuzzer – a library for coverage-guided fuzz testing
- How Heartbleed could've been found -- example of how fuzzing could be used for finding famous HeartBleed vulnerability
- Combining AFL and QuickCheck for Directed Fuzzing by Dan Luu
- Game Day Exercises at Stripe: Learning from kill -9
- Sometimes Kill -9 Isn’t Enough
- Comcast tool
- Slow tool
- Python API for Linux Traffic Control
- Blockade is a utility for testing network failures and partitions in distributed applications
- Your Load Generator Is Probably Lying To You
- Everything You Know About Latency Is Wrong -- great overview of Gil Tene`s "How NOT to Measure Latency" talk
- "How NOT to Measure Latency" by Gil Tene
- "Benchmarking: You're Doing It Wrong" by Aysylu Greenberg
- The Evolution of Testing Methodology at AWS: From Status Quo to Formal Methods with TLA+
- Use of Formal Methods at Amazon Web Services
- CACM Article "How Amazon Web Services Uses Formal Methods"
- Experience of software engineers using TLA+, PlusCal and TLC
- Debugging Designs by Chris Newcombie there is also a source bundle
On automated failure injection:
- Monkeys in Lab Coats: Applying Failure Testing Research @Netflix
- “Monkeys in Labs Coats”: Applied Failure Testing Research at Netflix
- Automated Failure Testing
- Automating Failure Testing Research at Internet Scale by P. Alvaro et.el
Random/manual failure injection testing:
- Netflix Simian Army
- Failure Injection Testing
- From Chaos to Control - Testing the resiliency of Netflix’s Content Discovery Platform
- Breaking Bad at Netflix: Building Failure as a Service
- GTAC 2014: I Don't Test Often ... But When I Do, I Test in Production -- Netflix different testing strategies
- Principles of Chaos Engineering
- Testing Apache Cassandra with Jepsen
- Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
- Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
- Jepsen Cassandra Testing on Git
- Netflix A STATE OF XEN - CHAOS MONKEY & CASSANDRA from Cassandra Summit 2015
- Testing Apache Cassandra with Jepsen: How to Understand and Produce Safe Distributed Systems by Joel Knighton presented at Devoxx UK 2016
They published series of blog posts on testing ScyllaDB:
- Scylla testing part 1: Cassandra compatibility testing
- Scylla testing part 2: Extending Jepsen for testing Scylla
- CharybdeFS: a new fault-injecting filesystem for software testing
- Testing part 4: Distributed tests
- Testing part 5: Longevity testing
- Fault-injecting filesystem cookbook
Series of post on testing at VoltDB:
- How We Test at VoltDB
- Testing at VoltDB: SQLCoverage Describes how they test SQL query functionality using 5 millions queries generated from templates and comparing results against HSQLDB.
- Testing VoltDB Against PostgreSQL
Additional resources:
- "All In With Determinism for Performance and Testing in Distributed Systems" by John Hugg and a slide deck Hugg-DeterministicDistributedSystems.pdf
- SelfCheck workload
- TPC-C implementation
- Running MemSQL’s 107 Node Test Infrastructure on CoreOS
- Practical Techniques to Achieve Quality in Large Software Projects
- How to Make a Believable Benchmark
- Building an Infinitely Scalable Testing System -- description of internal test system PsyDuck
- DIY Jepsen Testing CockroachDB -- great read about using Jepsen at Cockroach Labs
- MongoDB’s JavaScript Fuzzer: Creating Chaos (1/2)
- MongoDB’s JavaScript Fuzzer: Harnessing the Havoc (2/2)
- Quality Assurance at Cloudera: Fault Injection and Elastic Partitioning -- Cloudera describes their approach to fault injection testing
There is one talk from Sean T. Allen on testing stream processing system at Sendence
- Materials on Sean's blog "CodeMeshIO: How Did I Get Here?"
- Video from QCon NY 2016 on InfoQ
- Video from CodeMeshIO on YouTube
- Presentation on Speakerdeck
- Efficient Exploratory Testing of Concurrent Systems -- They don`t mention it but looks like they describe testing of Google Omega
- Exploratory Testing Architecture (ETA)
- Paxos Made Live - An Engineering Perspective has a section on testing
- 10 Years of Crashing Google describes some war stories from Disaster Recovery Testing (DiRT) team at Google
- Uncovering Bugs in Distributed Storage Systems during Testing (not in Production!)
- Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency describes "Pressure Point Testing" approach used for Azure Cloud Storage
- Inside Azure Search: Chaos Engineering
- Mysteries of Dropbox Property-Based Testing of a Distributed Synchronization Service -- example of how to use QuickCheck to test synchronisation in Dropbox and similar tools (Google Drive)
- Simoorg Failure inducer framework -- Failure inducer implemented in Python
- A Deep Dive into Simoo
- Finding bugs in SQLite, the easy way -- how fuzzing used in testing SQLite database
- How SQLite Is Tested
- Hermitage: Testing transaction isolation levels
- RapidCheck - QuickCheck port to C++
- Verdi - A framework for formally verifying distributed systems implementations in Coq
- Simulating shitty network connections so you can build better systems
- Muxy Simulating real-world distributed system failures
- faketime
- PolyConf 14: Testing the Hard Stuff and Staying Sane / John Hughes
- The Joy of Testing
- John Hughes on InfoQ
- Hansei: Property-based Development of Concurrent Systems
- QuickChecking Poolboy for Fun and Profit -- from Basho
- Combining Fault-Injection with Property-Based Testing
- Testing Telecoms Software with Quviq QuickCheck
- OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases
- OLTP Benchmark Wiki
- OLTP Benchmark on Github
- Py-TPCC
- Netflix Data Benchmark: Benchmarking Cloud Data Stores