Terminating Application Instances

Inspecting the definition of terminate-pod.yaml#

We’re finally there. We can finally create some chaos. We are about to destroy stuff.

Let’s take a look at the first definition that we are going to use. It is located in the chaos directory, in the file terminate-pod.yaml.

The output is as follows.

version: 1.0.0
title: What happens if we terminate a Pod?
description: If a Pod is terminated, a new one should be created in its places.
tags:
- k8s
- pod
method:
- type: action
  name: terminate-pod
  provider:
    type: python
    module: chaosk8s.pod.actions
    func: terminate_pods
    arguments:
      label_selector: app=go-demo-8
      rand: true
      ns: go-demo-8

What do we have there? We have version, title, description, and tags. They’re all informative. They do not influence how an experiment is executed. They just provide additional information to whoever would like to know what that experiment is about.

title#

The title, for example, says what happens if we terminate a Pod?. That’s a valid question.

description#

The description is a statement that if a pod is terminated, a new one should be created in its place.

tags#

We also have tags that are, again, optional. They’re letting us know that the experiment is about k8s (Kubernetes), and it is about a pod.

method#

The real action is happening in the method. That section specifies the activities that we want to perform. They can be actions or probes. For now, we’re going to focus only on actions.

We have only one action that we’re going to perform. We’re calling it terminate-pod. Within that action, there is a provider that is of type python. There could be theoretically other types, but since Chaos Toolkit uses Python, in most cases, the type will be python.

The module matches the plugin that we installed before (chaosk8s). We’re also specifying that we want to do something with a pod and that something is actions. Hence, the full name of the module is chaosk8s.pod.actions.

The main thing is the function (func). Whenever you’re in doubt, consult the output of the discover command, and you’ll see types of functions you can use within that module. In our case, we’re going to use the function called terminate_pods.

In this, and in most other cases, functions require some arguments. We’re using label_selector which specifies that we’re going to terminate a Pod that matches the app=go-demo-8 label. It could be any other label as long as there is at least one pod that matches it.

We also have the argument rand, short for random, set to true. If more Pods match that label_selector, a random one will be terminated.

Finally, there is the ns argument, which specifies the Namespace.

All in all, we want to terminate a Pod in the go-demo-8 Namespace that has the matching label app=go-demo-8. If there is more than one Pod with these criteria, it should select a random one to destroy.

Running terminate-pod.yaml#

Now that we have seen the definition, we can run it and see what we’ll get.

Inspecting the output#

The output is as follows (timestamps are removed for brevity).

[... INFO] Validating the experiment's syntax
[... INFO] Experiment looks valid
[... INFO] Running experiment: What happens if we terminate a Pod?
[... INFO] No steady state hypothesis defined. That's ok, just exploring.
[... INFO] Action: terminate-pod
[... INFO] No steady state hypothesis defined. That's ok, just exploring.
[... INFO] Let's rollback...
[... INFO] No declared rollbacks, let's move on.
[... INFO] Experiment ended with status: completed

We can see from the output that there are certain events that started by validating that the syntax is correct.

After the initial validation, it started running the experiment called What happens if we terminate a Pod?. It found that there is no steady state hypothesis defined. We’re going to see what the steady-state hypothesis is. For now, just observe that there isn’t any.

Judging by the output, there is one action terminate-pod. We’ll see the effect of that action later, even though you can probably guess what it is.

Next, it went back to the steady state hypothesis and figured out there is none. Then, it tried to rollback, and it found out that it couldn’t. We did not specify any of those things. We did not set the steady state hypothesis, and we did not specify how to rollback. All we have done so far is execute an action to terminate a Pod. We can see the result in the last line that says that the experiment ended with status: complete. It’s finished successfully.

That is the simplest experiment I could come up with, even though it might not be useful in its current form.

Now, let’s output the exit code of the previous command.

We can see that the output is 0. If you’re not familiar with Linux, exit code 0 means success. Any other number would indicate a failure.

Why are we outputting the exit code? The reason is simple. Even though we’re running the experiments manually and we can see from the output whether they are successful or not, sooner or later, we will want to automate all that. Additionally, it is essential that whatever we execute returns proper exit codes. That way, no matter the mechanism we use to run experiments, those exit codes will tell the system whether it’s a failure or a success.

Checking the pods#

Now, let’s take a look at the Pods in our Namespace.

The output states that no resources were found in go-demo-8 namespace.

What happened is that we deployed the single Pod, and we ran an experiment that destroyed it. That experiment by itself is not that useful because we did not validate whether the state before we ran it was correct, nor did we check whether the state after the action was as expected. We did not do any validations. Instead, we only executed a single action meant to terminate a Pod, and we can see from the output that the Pod is no more. It was destroyed.

In the next lesson, we will define the steady-state hypothesis for our experiment.