Recently, I realized that there is not much information available about how to monitor Portworx with Dynatrace. Therefore, I decided to write this article to guide you through the process. As you probably know, Portworx metrics are available in OpenMetrics format and are typically collected by the built-in Prometheus deployment of Portworx.
Moreover, Dynatrace supports scraping OpenMetrics format metrics from application endpoints. Therefore, to start collecting Portworx metrics with Dynatrace, you just need to follow the instructions on the page and meet the following requirements:
- ActiveGate version 1.217+.
- In Dynatrace, go to your Kubernetes cluster settings page and enable:
- Monitor Kubernetes namespaces, services, workloads, and pods.
- Monitor annotated Prometheus exporters.
- Annotated pod (or service) definitions.
After fulfilling the first two prerequisites, all you need to do is to annotate two services that are attached to the pods that share metrics. You can identify those services and needed ports by checking the currently deployed ServiceMonitor object in the Portworx installation namespace (I am using the kubectl neat plugin here to make the output clean). The required port names are highlighted below:
You can identify the needed port numbers by checking the manifests of services with the label “name=portworx” and checking the full manifests of those services. The port numbers are highlighted in the output below:
Now that we have identified the services and ports, all we need to do is to annotate the identified services as suggested by Dynatrace. To do this, we would add additional configuration to our “StorageCluster” object to annotate the needed services. Please see detailed instructions on this page.
This will annotate the needed services, and OneAgents would start scraping metrics from the specified endpoints.
If you would like to collect metrics of the portworx-operator, you can annotate its service by modifying it or by running the following command:
Finally, if you would like to monitor Stork, you can deploy an additional service with the required annotations. The deployment of the following manifest would do the work:
After completing all the steps above, your Portworx installation metrics will be available in Dynatrace, and all you need to do is set up dashboards and alerts.
To set up dashboards, a good starting point would be Grafana dashboard examples available on the Portworx documentation site.
For alert definitions, you can use existing Alertmanager alert definitions.
Ones related to Portworx can be found on this page here (just add labels "prometheus=portworx" to the Prometheus rule object), and alerting rules for Stork can be found on this page.
I hope this article helps you monitor your Portworx installation with Dynatrace. If you have any questions, feel free to reach out to the team here at Arctiq.