Monitoring & Observability
alerting
Endpoint Alert

Alerting on Endpoint Availability or Request Data (HTTP Request)

Sometime you want to monitor endpoint that your application exposes, such as various health endpoints and/or the data those endpoints provide. For this, use-case we use Infinity (opens in a new tab) data source that provides simple capabilities on making requests to endpoints and parsing the JSON response data.

To create such an alert we need to

  1. First go to Alert section (opens in a new tab) in Grafana (bell icon on the left menu). Then hit the "+ New alert rule" button. create new alert

  2. Fill in the metadata of rule

    • Rule name, give it what ever you feel is descriptive
    • Folder, select based on the cluster
    • Group, you can put anything in place of a group, like project name
      Bear in mind that all alerts within the same group will be evaluated at the same time. So, if you are planing on creating more alerts for one project, we would suggest to give it the name of that project

  3. Select "Infinity" as data source. Optionally, we can also change name of expression from "A" to maybe something more descriptive

  4. The only thing we need to specify is Parser field to be "Backend". Also note, that for alerting, Infinity supports only JSON queries (opens in a new tab)

  5. Then just fill out the URL. Select method that you want (GET, POST) and fields to parse out from the JSON response based on what you want to check infinity alert example

      _Note, Grafana can only work and alert on numeric responses, therefore your endpoint has to return some sort of a number_

  6. Next, we need to add "Math" expression where we check out whether response is correct. In our example case, we want alert if response status is not 200 infinity math expression

  7. We need to define alert condition. Select the name of our "Math" expression and set check interval (Evaluate) and for how long should the alert be pending before firing (for).

  8. Lastly, we want to add some description and summary to our alert, with some custom labels, that would help us to better specify what is going on, when we receive the alert notification infinity alert details

And that is it. Now you can just <font color="white" style={{backgroundColor: '#3871dc', padding: '2px'}}>"Save and exit", and your alert should be running, and firing in case of any issues.
The default contact point is through Slack to grafana-alerting channel. If you want to receive your alerts somewhere else or through some other means, please checkout "How to add Contact Point" recipe.

Examples

Contact PointGrafana Alerting