LXD Metrics

References

Introduction

LXD servers have a metrics endpoint as part of the REST API. Arguably the best way to take advantage of this is to use Prometheus to collect the metrics data, and Grafana to graph the collected data.

This page describes configuring LXD servers to enable metrics gathering, as well as setting up docker containers of Prometheus and Grafana to make use of the metrics data.

Metrics Certificate

We will need a certificate to provide authentication in order to gather metrics from LXD servers. The resultant files (metrics.key, metrics.crt) need to be copied to the metrics server. In addition, the metrics.crt file will need to be copied to each LXD server for which metrics data will be gathered.

$ openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 -sha384 -keyout metrics.key -nodes -out metrics.crt -days 3650 -subj "/CN=metrics.local"

LXD Server Configuration

Gathering metrics data from an LXD server require two configuration steps. First, the metrics data port needs to be configured. Next a trust needs to be added using the metrics public certificate. Finally, we need to obtain the LXD server certificate, which will be copied to the metrics server. These steps need to be performed on each LXD server that metrics will be gathered from.

$ lxc config set core.metrics_address :8444
$ lxc config trust add --type metrics --name prometheus metrics.crt
$ sudo cat /var/snap/lxd/common/lxd/server.crt >(server)-server.crt

Setting up the Docker Directory

The following sub-directories need to be created in the docker-compose run directory. The permissions and directory ownership need to be set as well.

$ mkdir prometheus_data prometheus_etc prometheus_etc/tls grafana_data grafana_etc 
$ chmod 777 prometheus_data
$ chmod 777 grafana_data
$ sudo chown 472:0 grafana_data

Next you need to generate the grafana.ini file. For some reason the grafana docker image doesn't create this on the initial run in this setup.

$ docker run --rm --entrypoint /bin/bash grafana/grafana -c 'cat $GF_PATHS_CONFIG' > grafana_etc/grafana.ini

Finally, the prometheus.yml file, suitably modified to match the current environment, is copied into prometheus_etc. All of the server cert files are copied into prometheus_etc/tls, as well as the metrics.key and metrics.crt files.

Once all of this is in place, copy the docker-compose.yml file into the run directory and start the servers with docker-compose up -d. Confirm everything is running correctly by checking the docker container logs.

Check Prometheus Data Sources

Confirm that Prometheus is gathering metrics data from the LXD servers by first opening the prometheus page in a web browser. Open "http://(metrics-server):9090" in a browser, and then select "Status -> Targets". All of the LXD server data sources should be marked as "up", with no errrors.

Setup Grafana Data Source and Dashboard

Finally, open the Grafana server in a web browser at "http://(grafana-server):3000". Select "Configuration -> Data Sources". Select "Add data source", and "Prometheus". In the "HTTP -> URL" field enter "http://prometheus:9090". Scroll down to the bottom of the page and select "Save & test".

Next select "Dashboards -> Import", and in the "Import via grafana.com" field, enter "19131", then click "Load" to the left of that. At the bottom of the next page, set "Data Source" to "Prometheus", then select "Import".

This will import the LXD dashboard created by "stgraber". This dashboard is configured to display the LXD instance metrics for the selected LXD server.

Conclusion

This is a very simple deployment of Prometheus and Grafana. There are many other Prometheus features, such as alerting, etc, that haven't been touched upon. There are also many features of Grafana that could be expanded upon as well.

However, this will provide a very functional metrics gathering and display system for LXD servers though, which was exactly what I was looking for when I set this up.

Created: 2023-01-17 14:19
Last update: 2023-09-11 20:08