Grafana: The API Magic

When we start using Grafana to monitor our environment, we may have many ways to pre-create dashboards in our environment. We can configure the Grafana docker image with dashboards in a Dockerfile or deploy it using a Grafana helm chart that can reference the JSON files that contain the dashboards. 

But what about a scenario where we have already created a Grafana environment, and after a while, we decided to add dashboards and other Grafana resources? We probably won’t want to delete the environment and recreate it. What can we do? One solution, which is not automated, is to manually add dashboards to the environment or import them from an existing one.

The other solution is to use the Grafana API.

What is Grafana API

Grafana API allows developers to access Grafana resources through the CLI, and you will be amazed at how many actions we can perform in Grafana from a local terminal window. A simple ‘curl’ command will give us access to almost all Grafana resources. Thanks to this functionality, Grafana API can be accessed from a wide range of platforms and technologies, such as Infrastructure as Code tools or CICD pipelines.

Advantages

Let’s explore the benefits of Grafana API to understand when and why to use this solution:

1. Automation: Using the Grafana API, you can automate various tasks, such as creating, updating, or deleting dashboards, data sources, users, or actually all your Grafana resources.
2. Customization: After creating Grafana, we can extend its functionality, add resources, and configure the platform in various ways.
3. Bulk Operations: The API supports bulk operations, allowing you to perform actions on multiple objects simultaneously. This can be particularly useful for efficiently making changes across a large number of dashboards or data sources.
4. Scalability: The API helps in scaling operations by providing programmatic access to manage various aspects of Grafana. This is essential in large or distributed environments where manual management may be impractical.

Authentication

As we already know, the Grafana API is accessible from every terminal that you can run a ‘curl’ command. So, in order to make changes in our Grafana environment, we have to specify a username and password for Grafana in our request according to the regular properties. We have two ways. First, we can specify the credentials in the requests location

curl -X GET --location 'http://USER:PASSWORD@GRAFANA_HOST:GRAFANA_PORT/healthz'

The second way is using the –user flag in the HTTP request.

curl -X GET --location 'http://GRAFANA_HOST:GRAFANA_PORT/healthz' --user 'USER:PASSWORD'

For this specific HTTP request we actually don’t need to specify username and password, but this is just an example of how to use the credentials inside the request.

Dashboards

When it comes to what actions we can perform on our dashboards, we mostly speak about the /api/dashboards API. So, what actions can we take?

Let’s start with the most basic action of listing all the dashboards in our environment. This action will actually use the “search” API in the following format:

curl -X GET --location 'http://USER:PASSWORD@GRAFANA_LOCATION:GRAFANA_PORT/api/search\?query\=\&' | jq -r '.[] | select(.type=="dash-db")

The /api/search?query=& will return all the dashboards and their folders, so we must use the jq tool to extract only the JSONs of type ‘dash-db’ that correspond to the dashboards themselves.

If we wish to have a copy of one of the dashboards, we can request the API of the dashboard’s unique identifier (UID) and retrieve it in a JSON format into a local file on our computer.

curl -X GET --location http://USER:PASSWORD@GRAFANA_HOST:GRAFANA_PORT/api/dashboards/uid/DASHBOARD_UID > my-dashboard.json

This is how we view dashboards, but what about importing new dashboards? Grafana’s console allows us to import dashboards we have created previously, save locally on our computers, or import official community dashboards from Grafana Labs. When using the API, all methods use the same logic. You should get a JSON file of your dashboard and then use the /api/dashboards/import API to import the dashboards into your environment. As you probably start to realize, the real problem in getting the JSON file begins when we want to import the official Grafana dashboard. Well, after a lot of digging, I found one API that comes to our aid! 

curl -u "USER:PASSWORD" http://GRAFANA_HOST:GRAFANA_PORT/api/gnet/dashboards/IMPORT_ID | jq .json > FILE.json

Here is a concrete example of how to import the official AWS RDS dashboard into a file called temp.json:

curl -u "admin:admin" http://localhost:3000/api/gnet/dashboards/707 | jq .json > temp.json

Now, after we’ve got the JSON file we can continue importing it to our Grafana environment using the ‘import’ api:

curl -X POST -k -H "Content-Type: application/json" -d "{\"dashboard\":$(cat temp.json)}" --location http://localhost:3000/api/dashboards/import --user 'admin:admin'

As you can see, instead of pasting the entire dashboard in JSON at my request, I’ve preferred to use a ‘cat’ command and specify the path to the dashboard’s location. This way the request looks less complicated and more readable.

Of course, if we can import new dashboards to our environment, we can also delete them the same way, but only using the ‘DELETE’ HTTP request.

curl -X DELETE -u 'admin:admin' http://localhost:3000/api/dashboards/uid/AWSRDSdbi

Datasources

Datasources are actually easier to create than you can imagine. When we want to create a new one, it’s simply specifying all the datasource details within the ‘datasources/api’ request, again using the –data flag like this:

curl -X POST 'http://localhost:3000/api/datasources' -u 'admin:admin' --data-binary '{"name":"prometheus","type":"prometheus","url":"http://prometheus:9090/","access":"proxy","isDefault":true}'

In the example above, we are creating a Prometheus datasource. We specify basic information about the datasource (we can also make this the default datasource), but we also need to configure the datasource URL and authentication if necessary. This specific example exposes the Prometheus service via the default port 9090 of Prometheus without requiring authentication.

Plugins

There are some Grafana panels that use plugins. Grafana API provides an API for adding plugins that are not included in the default environment. Let’s say that you want to add a clock to one of your panels but don’t have the clock plugin installed; you won’t be able to see the time. How can you resolve this issue?Using this simple request:

curl -X POST http://localhost:3000/api/plugins/grafana-clock-panel/install -u 'admin:admin'

In this example, we can see that we added the official Grafana clock plugin which is not installed by default in the official Grafana container image.

Alerts

Alerting is one of the primary reasons for using Grafana. Not only will Grafana monitor your infrastructure and services, but it can also configure alert mechanisms to notify you in case of different errors that are happening to your resources. The alerting section in Grafana opens a wide variety of new resources that we can use, such as alert-rules, contact points, and notification policies. By now, you probably understand that we can provision them all using the Grafana API, so let’s see how it is done.

For alert rules, we can provision new alerts or delete unwanted alerts using the following formats:

Get alert-rule by UID into a file:

curl -X GET -u 'admin:admin' http://localhost:3000/api/v1/provisioning/alert-rules > alerts.json

Delete alert-rules (this example is by using specific UID:

curl -s -X 'DELETE' -u admin:admin 'http://localhost:3000/api/v1/provisioning/alert-rules/f481c3ac-800c-40f1-8863-cd4aebaed16d'

Add new alert-rules:

curl -X POST -u admin:admin http://localhost:3000/api/v1/provisioning/alert-rules -d "$(cat alerts.json | jq '.[]' )"

But alert rules only monitor which thresholds have passed their limits and turn into pending state or firing state so you need to define another resource for the notification itself, contact points!

curl -X POST -u 'admin:admin' --location 'http://localhost:3000/api/v1/provisioning/contact-points' -H "Content-Type: application/json" -d '{"name":"example-cp","type":"slack","settings":{"recipient":"general","token":"example-token"},"disableResolveMessage":false}'

This is an example request to create a new contact point of type Slack. If you are getting any issues while trying to create, including errors of ‘bad request data,’ it is most likely a header problem or your –data-binary format mistake.

The last thing you should configure is a connection between the alert rules to the contact points. This is where we use notification policies. If you have ever worked with Grafana, you know by now that policies will be applied to a specific scope of alerts that you define and redirect them to a chosen contact point.

curl -X PUT -u 'admin:admin' --location 'http://localhost:3000/api/v1/provisioning/policies' -H "Content-Type: application/json" -d '{"receiver":"newcp","object_matchers":[["scope","=","tcp"]]}'

For notification policies, we can’t update each and every single one of them. Instead, we must use a PUT http request and set the entire notification policy tree.

Now we have a fully working alerting system in our Grafana environment!

Conclusion

We can use many different ways to manage our Grafana environment. Grafana API has turned out to be a useful tool for developers to edit their monitoring environment after creation time and make it as dynamic as possible. It uses the basic ‘curl’ command so it’s also available to use from many different workflows and technologies. For example, we can use it as a ‘step’ in a GitHub actions CI pipeline or a ‘null resource’ in Terraform, and hopefully, this article makes more sense in managing your monitoring tools using these simple solutions.

Ready to take your DevOps solutions to the next level?