The real world is complicated – especially true for distributed systems. This pushed Steadybit’s experiment engine to their limits when trying to represent the real world (AKA production) in a very simple experiment. In this blog post, we’ll unleash a set of new experiment features that address these shortcomings – and are already loved by many of our users.
As an attentive reader, you may be familiar with our other blog posts covering simple experiments to survive an AWS zone outage, testing exception handling of REST endpoints and revealing top 3 Kubernetes weak spots. Now we want to test more complicated examples, such as a rippling container outage, a slow network combined with package corruption, or a DNS failure that occurs at the exact moment a container restart is performed. In this blog post, we will verify the last point and start by simulating a simple DNS failure. For that, we use the online shopping demo application, which consists of a microservice gateway that periodically requests products from other microservices over HTTP (Hot-Deals, Fashion-Bestseller, and Toys-Bestseller). See Shopping Demo’s GitHub repository for more details.
Since Steadybit has already discovered our system, we can use the known wizard for experiments. Just follow the steps below to create this experiment yourself. If you don’t have Steadybit in use yet, let’s meet for a short demo call.
We start on the Steadybit dashboard to create an experiment, which leads us to the familiar wizard.
So, now we land for the first time in our new experiment editor! It is pretty neat and allows us to add attacks, checks, or actions to the experiment via drag and drop. We can decide for ourselves whether we want to run multiple attacks one after the other or simultaneously. We make more use of it in step 4.
For now, we will run the experiment we have created and see how our application behaves.
This brings us to the new Steadybit execution window (left side of the screenshot), which provides many details about the system. The system looks stable now, and everything seems to be working. We will look closer at the system in step 5 to analyze erroneous behavior.
However, opening up the demo shop in parallel (right side) we get the same feeling: the shop is still working – we are safe!
But wait, are we? As you probably know: DNS records are cached. This way, a DNS failure is not so bad as long as the DNS cache is up to date. But what happens to the system when the DNS cache isn’t there yet?
Let’s extend the scenario by adding more steps. First, we add a wait step of a few seconds before the “DNS failure” attack and a simultaneous HTTP call from the beginning. This will periodically request the HTTP endpoint of the online shop and ensure that the DNS entries are cached when the DNS failure is simulated.
The next step is to crash one of the gateway containers (using a simple stop container attack), immediately restarted by Kubernetes. We can also check if the pod has been successfully marked as unhealthy and is ready again afterward by using the pod count checks (see lane 3).
All right, ready to go! Let’s check how our system behaves and keep our fingers crossed that the user won’t notice any downtime. We’ll rerun the experiment.
Uh-oh…This doesn’t look so good. The experiment failed because the HTTP call received some erroneous responses. Let’s take a closer look and analyze what happened.
The execution view already shows all the necessary details to find out what happened here:
Putting the puzzle pieces together, we can conclude that the restarted container was immediately marked as ready again (as soon as the configured readiness probe was successful). Nevertheless, due to the DNS failure and the missing DNS cache, it could not reach any other microservices (Hot-Deals, Fashion-Bestseller, and Toys-Bestseller) via HTTP.
However, Kubernetes assumes it has two ready pods, which are both working fine, and it can schedule the HTTP requests to both. Thus, it sends the requests to the newly started container and has no idea it is not working correctly due to DNS cache errors.
We can even validate the hypothesis by re-running it a few times, as the routing of Kubernetes may also choose the healthy gateway pod to serve the request.
Now that the weak spot has been exposed, we have several options to fix it. The simplest approach would be to extend the gateway’s health endpoint to check the availability of the endpoints of other microservices (Hot-Deals, Fashion-Bestseller, and Toys-Bestseller).
However, a better approach would be to rethink the shop’s architecture in terms of a self-contained system. Right now, the gateway has a very tight coupling to other microservices, making it difficult to get it working reliably. Applying known patterns of eventual consistency and caches may help to reduce the coupling.
So, bottom line: Setting up more complex experiments with Steadybit’s new experiment features is pretty straightforward. We could set up a complex experiment in a few minutes and use it to test our system for specific turbulent conditions. This way, we could uncover a weak spot and think about possible fixes. However, the Experiment Editor has many more features waiting for you. In the coming weeks, we will present some of them in blog posts covering, e.g., the integration of monitoring solutions like Instana, New Relic, Data Dog, and Prometheus.