Using hostPath Volume Type to Inject Configuration Files

Using Prometheus#

We are about to deploy Prometheus for the first time. We won’t go into details behind the application except to say that it’s fantastic and that you should consider it for your monitoring and alerting needs. We’re using it only to demonstrate a few Kubernetes concepts. We’re not trying to learn how to operate it.

Looking into the definition#

Let’s take a look at the application’s definition.

There’s nothing genuinely new in that YAML file. It defines an Ingress, a Deployment, and a Service. There is, however, one thing we might need to change.

Configuring the IP#

Prometheus needs a full external-url if we want to change the base path. For practicing on our platform the URL is set to Educative at line 49 in the following definition of prometheus.yml.

Once we created the application, we used the kubectl rollout status command to confirm that the deployment finished.

Testing prometheus#

Now we can open Prometheus in a browser.

At first glance, the application seems to be running correctly. However, since the targets are the crucial part of the application, we should check them as well. For those not familiar with Prometheus, it pulls data from targets (external data sources) and, by default, comes with only one target pre-configured: Prometheus itself. Prometheus will always pull data from this target unless we configure it otherwise.

Let’s take a look at its targets. While port-forward is still running open the following in the browser or press the link beside run button in the playground and add /targets at the end.

There’s something wrong. The default target is not reachable. Before we start panicking, we should take a closer look at its configuration.

The problem is with the metrics_path field. By default, it is set to /metrics. However, since we changed the base path to /prometheus, the field should have /prometheus/metrics as the value.

Changing Prometheus configuration#

Long story short, we must change the Prometheus configuration.

We could, for example, enter the container, update the configuration file, and send the reload request to Prometheus. That would be a terrible solution since it would last only until the next time we update the application, or until the container fails, and Kubernetes decides to reschedule it.

Let’s explore alternative solutions. We could, for example, use hostPath Volume for this as well. If we can guarantee that the correct configuration file is inside the cluster, the Pod could attach it to the prometheus container. Let’s try it out. The output of the prometheus-hostpath with relevant segments is shown below.

The only significant difference, when compared with the previous definition, is in the added volumeMounts and volumesfields. We’re using the same schema as before, except that, this time, the type is set to File. Once we apply this Deployment, the file /files/prometheus-conf.yml on the host will be available as /etc/prometheus/prometheus.yml inside the container.

Try it yourself#

A list of all the commands used in the lesson is given below.

Troubleshooting tips for minikube#

Following are some steps you may want to follow while working with minikube clusters.

Configuring IP#

Prometheus needs a full external-url if we want to change the base path. At the moment, it’s set to the IP of our Minikube VM. In your case, that IP might be different. We’ll fix that by adding a bit of sed “magic” that will make sure the IP matches that of your Minikube VM.

We output the contents of the volume/prometheus.yml file, we used sed to replace the hard-coded IP with the actual value of your Minikube instance, and we passed the result to kubectl create.

📝 Please note that, this time, the create command has dash (-) instead of the path to the file. That’s an indication that stdin should be used instead.

If you recall, we copied one file to the ~/.minikube/files directory, and Minikube copied it to the /files directory inside the VM.

In some cases, files might end up being copied to the VM’s root (/), instead of to /files. If this has happened to you, please enter the VM (minikube ssh), and move the files to /files, by executing the commands that follow (only if the /files directory does not exist or is empty).

We output the contents of the volume/prometheus.yml file, we used sed to replace the hard-coded IP with the actual value of your Minikube instance, and we passed the result to kubectl create.

The time has come to take a look at the content of the file. The output is as follows.

This configuration is almost identical to what Prometheus uses by default. The only difference is in the metrics_path, which is now pointing to /prometheus/metrics.