Sending Experiment Notifications

At the time of this writing (March 2020), Chaos Toolkit allows us to send notifications to at least two different targets (there may be others by the time you’re reading this). We can send notifications to Slack, or to any HTTP endpoint.

Using Slack as the target for notifications#

We’ll focus on Slack. If that’s not your favorite communication tool, you should be able to change the configuration to use anything else, as long as that destination can receive HTTP requests. Since almost every tool designed in this century has an API exposed through HTTP, we should be able to send notifications to any destination. The question is whether we do that through a plugin, just like the one we’re about to use, or through the generic HTTP notification mechanism available in Chaos Toolkit.

We’re going to explore how to send notifications to Slack. Later on, if you decide that Slack is not something you want to use, you should be able to adapt the examples to HTTP notifications.

For you to be able to see the notifications we are about to send, you’ll have to do one of two things. You can join DevOps20 Slack workspace. If you don’t want to do that and, instead, you prefer to use your own Slack workspace, you will need to change a few commands. Specifically, you’ll have to change the Slack API token in some of the YAML definitions we’re about to explore.

So, either use my Slack workspace, and for that, you’ll have to join DevOps20 Slack workspace or modify some of the YAML definitions that I prepared.

Inspecting the settings.yaml file#

Let’s take a look at settings.yaml that contains a few things that we will need.

The output is as follows.

apiVersion: v1
kind: ConfigMap
metadata:
  name: chaostoolkit-settings
data:
  settings.yaml: |
    notifications:
    - type: plugin
      module: chaosslack.notification
      token: xoxb-@102298415591@-@902246554644@-@xa4zDR3lcyvleRymXAFSifLD@
      channel: tests

That’s a simple ConfigMap with a single key settings.yaml. Normally, we use ConfigMaps in Kubernetes for all sorts of configurations. The exceptions would be confidential information that can be stored as Kubernetes Secrets, in HashiCorp Vault, or in any other secrets management solution. But, we won’t dive into secrets since that would result in a big topic by itself and would not be directly related to chaos experiments. We’d need not only to create secrets (that part is easy) but also to figure out how to inject them into that ConfigMap. We’d probably end up with temporary settings (e.g.settings.yaml.tmp and a secret, and we’d combine them into the final settings.yaml file during boot of the container with Chaos Toolkit. Instead, we’re taking a shortcut by adding the token inside the ConfigMap.

Fixing the Slack token and applying the ConfigMap#

Let’s get back to the task at hand.

Chaos Toolkit expects .chaostoolkit/settings.yaml file inside the home directory of the current user. We’ll deal with that later. For now, what matters is that the ConfigMap defines the settings.yaml file with everything we’ll need to send notifications to Slack.

The settings.yaml file contains a single entry in the notification section that specifies that we want to use the chaosslack.notification module, which is available through the Slack plugin that is already available in the container image we explored earlier. Additionally, we have the token and the channel. The former is, as you can guess, used to authenticate to Slack, while the latter specifies which channel within the workspace should receive notifications.

If you already used Slack tokens, you might have noticed that the on in that YAML is a bit strange. I could not put the “real” token because GitHub would not allow me to push confidential information. Instead, I “tricked” the system by adding the @ character in a few places inside the token. That way, GitHub does not recognize it as a Slack token, while it’s still relatively easy to transform it into the “real” one. All we have to do is remove @ characters. We’ll use sed for that.

We output the content of the settings.yaml file and used it as the input to the sed command. It, in turn, replaced all occurrences of @ with an empty string. The transformed definition was sent to the kubectl apply command, which created the modified ConfigMap inside the go-demo-8 Namespace.

Inspecting the periodic-slack.yaml file#

Now that we have the ConfigMap with settings.yaml, which defines the target (Slack) that should receive the notifications, we can include it into our CronJob.

The output, limited to the relevant parts, is as follows.

---

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: go-demo-8-chaos
spec:
  ...
  jobTemplate:
    ...
    spec:
      ...
      template:
        ...
        spec:
          ...
          containers:
          - name: chaostoolkit
            ...
            volumeMounts:
            ...
            - name: settings
              mountPath: /root/.chaostoolkit
              readOnly: true
            ...
          volumes:
          - name: experiments
            configMap:
              name: chaostoolkit-experiments
          ...

The new definition is almost the same as the CronJob currently running in our cluster. The only novelty is the addition of the ConfigMap. The CronJob has a new entry in the volumeMounts section referencing the volume settings. It mounts it as the /root/.chaostoolkit directory, which is the location Chaos Toolkit expects to find settings.yaml. Further down, we can see those same settings defined as the volume that is the configMap with the name chaostoolkit-experiments.

Checking the difference between periodic.yaml and periodic-slack.yaml#

To be sure that we did not miss any other change when compared with the previous definition, we’re going to output the differences.

The output is as follows.

41a42,44
>  - name: settings
>    mountPath: /root/.chaostoolkit
>    readOnly: true
55a59,61
>  - name: settings
>    configMap:
>      name: chaostoolkit-settings

As you can see, the entries to mount the ConfigMap are the only changes between now and then.

Applying the new definition#

Let’s apply the new definition.

Now that the CronJob was reconfigured, we should start seeing notifications in the DevOps20 Slack workspace, unless you chose to use your own token.

Observing the notifications on Slack#

Go to the DevOps Slack workspace and choose the channel tests. You will see the output of your chaos experiment soon.

Don’t get confused if you don’t see only the results of your experiments. You might notice notifications from experiments run by other people that might be going through the same exercises as you. It all depends on how many people are exploring notifications at the same time. What matters, though, is that we are going to see our notifications popping up every five minutes and that they might be mixed with notifications coming from others.

You’ll see a notification when the next experiment starts (run-started), as well as when it ends (run-completed). You should be able to observe something similar to the image below.

I won’t insult your intelligence by going through the information we see in those notifications. They are the same pieces of data we’ve seen countless times before. The only difference is that they’re now in Slack.

If you keep the CronJob running, the same notification will pop up every five minutes. Just don’t get confused if you see an occasional unexpected notification. Others might be ahead of you.

What is interesting about the notifications is that they are not specific to a failure or a success. We are, right now, receiving notifications, no matter the outcome of the experiments. We’ll change that soon. We’ll try to figure out how to filter the notifications so that, for example, we receive only unsuccessful experiments. But, for now, what we are having in front of us are the results of the experiments, no matter whether they’re successful or failed.

Checking the pods#

We can confirm the same thing if we output the Pods.

The output, in my case, is as follows.

NAME                  READY   STATUS      RESTARTS   AGE
go-demo-8-chaos-...   0/1     Error       0          16m
go-demo-8-chaos-...   0/1     Completed   0          11m
go-demo-8-chaos-...   0/1     Completed   0          6m14s
go-demo-8-chaos-...   0/1     Completed   0          83s

Just as before, we can see that the new Pod was completed.

If you are curious to see more information about how to set up notifications, please go to the Get Notifications From The Chaos Toolkit Flow page in the documentation.

In the next lesson, we’ll go slightly deeper into notifications and try to filter them a bit more.