Creating a Multi-Experiment Report

Assuming that we want to see how we can generate reports based on multiple experiments, the first thing we need to do is to run a second experiment. Otherwise, we’d be left with data (journal) from a single experiment.

So, we are going to execute yet another experiment that will generate a second journal file. After that, we’ll try to figure out how to create a report based on both journals.

Using the definition of network-delay.yaml#

Let’s start by taking a quick look at yet another definition.

The output is as follows.

version: 1.0.0
title: What happens if we abort and delay responses
description: If responses are aborted and delayed, the dependant application should retry and/or timeout requests
tags:
- k8s
- istio
- http
configuration:
  ingress_host:
      type: env
      key: INGRESS_HOST
steady-state-hypothesis:
  title: The app is healthy
  probes:
  - type: probe
    name: app-responds-to-requests
    tolerance: 200
    provider:
      type: http
      timeout: 15
      verify_tls: false
      url: http://＄{ingress_host}?addr=http://go-demo-8
      headers:
        Host: repeater.acme.com
  - type: probe
    tolerance: 200
    ref: app-responds-to-requests
  - type: probe
    tolerance: 200
    ref: app-responds-to-requests
  - type: probe
    tolerance: 200
    ref: app-responds-to-requests
  - type: probe
    tolerance: 200
    ref: app-responds-to-requests
method:
- type: action
  name: abort-failure
  provider:
    type: python
    module: chaosistio.fault.actions
    func: add_abort_fault
    arguments:
      virtual_service_name: go-demo-8
      http_status: 500
      routes:
        - destination:
            host: go-demo-8
            subset: primary
      percentage: 50
      version: networking.istio.io/v1alpha3
      ns: go-demo-8
- type: action
  name: delay
  provider:
    type: python
    module: chaosistio.fault.actions
    func: add_delay_fault
    arguments:
      virtual_service_name: go-demo-8
      fixed_delay: 15s
      routes:
        - destination:
            host: go-demo-8
            subset: primary
      percentage: 50
      version: networking.istio.io/v1alpha3
      ns: go-demo-8
  pauses: 
    after: 1
rollbacks:
- type: action
  name: remove-abort-failure
  provider:
    type: python
    func: remove_abort_fault
    module: chaosistio.fault.actions
    arguments:
      virtual_service_name: go-demo-8
      routes:
        - destination:
            host: go-demo-8
            subset: primary
      version: networking.istio.io/v1alpha3
      ns: go-demo-8
- type: action
  name: remove-delay
  provider:
    type: python
    func: remove_delay_fault
    module: chaosistio.fault.actions
    arguments:
      virtual_service_name: go-demo-8
      routes:
        - destination:
            host: go-demo-8
            subset: primary
      version: networking.istio.io/v1alpha3
      ns: go-demo-8

That is the same experiment as one of those we ran while we were experimenting with networking. There’s nothing new here.

Running the experiment with the --journal-path argument#

We are reusing the experiment we’re familiar with, so I’ll skip explaining what that does. Instead, we’ll jump straight into running it.

The output, without timestamps, is as follows.

[... INFO] Validating the experiment's syntax
[... INFO] Experiment looks valid
[... INFO] Running experiment: What happens if we abort and delay responses
[... INFO] Steady state hypothesis: The app is healthy
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Steady state hypothesis is met!
[... INFO] Action: abort-failure
[... INFO] Action: delay
[... INFO] Pausing after activity for 1s...
[... INFO] Steady state hypothesis: The app is healthy
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Probe: app-responds-to-requests
[... INFO] Steady state hypothesis is met!
[... INFO] Let's rollback...
[... INFO] Rollback: remove-abort-failure
[... INFO] Action: remove-abort-failure
[... INFO] Rollback: remove-delay
[... INFO] Action: remove-delay
[... INFO] Experiment ended with status: completed

The experiment was successful, and you already know what it does. Even if your memory does not serve you well, you should be able to deduce what it’s about by reading the definition and comparing it with the outcome of the execution. What matters is that we stored the result in a different journal file.

Creating the Docker container and generating the report#

Let’s run another container to generate a report.

We created a container with almost the same arguments. The only difference is that we added the path to the second journal file (journal-network-delay.json).

We can generate a report based on as many experiments as we want, as long as each produces a different journal file. All we have to do is specify all those journals as arguments. It’s up to Chaos Toolkit to assemble the information from all the specified experiments and create a report.

Inspecting the generated report#

Let’s take a look at the report.

We can see that this time, the report contains the outcomes from both experiments.

As long as we store the results of each experiment in a separate journal file, we can combine them all into one big report and distribute it to whoever is interested.

In the next lesson, we will remove the resources that we have created.