This example requires an Fn server to be running and that you have deployed one or more functions. See the front page or any of the other examples for instructions.

The steps below assume that the Fn server is running at localhost:8080.

Examine the endpoint used to export metrics to Prometheus

The Fn server exports metrics to Prometheus using the API endpoint /metrics.

Try pointing your browser at http://localhost:8080/metrics. This will display the metrics in prometheus format.

Start Prometheus

Open a terminal window and navigate to the directory containing this example.

Examine the provised Prometheus configuration file:

cat prometheus.yml

This gives

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'fn-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's the Fn server
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'functions'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      # Specify all the Fn servers from which metrics will be scraped
      - targets: ['localhost:8080'] # Uses /metrics by default

Note the last line. This specifies the host and port of the Fn server from which metrics will be obtained. If you are running a cluster of Fn servers then you can specify them all here.

Now start Prometheus, specifying this config file:

docker run --name=prometheus -d -p 9090:9090 \
  --mount type=bind,source=`pwd`/prometheus.yml,target=/etc/prometheus/prometheus.yml \
  --add-host="localhost:`route | grep default | awk '{print $2}'`" prom/prometheus

Note: The parameter --add-host="localhost:`route | grep default | awk '{print $2}'`" means that Prometheus can use localhost to refer to the host. (The expression `route | grep default | awk '{print $2}'` returns the IP of the host).

Open a browser on Prometheus's graph tool at http://localhost:9090/graph. If you wish you can use this to view metrics and display metrics from the Fn server: see the Prometheus documentation for instructions. Alternatively continue with the next step to view a ready-made set of graphs in Grafana.

Start Grafana and load the example dashboard

Grafana provides powerful and flexible facilities to create graphs of any metric available to Prometheus. This example provides a ready-made dashboard that displays the numbers of functions that are queued, running, completed and failed.

Open a terminal window and navigate to the directory containing this example.

Start Grafana on port 3000:

docker run --name=grafana -d -p 3000:3000 \
  --add-host="localhost:`route | grep default | awk '{print $2}'`" grafana/grafana

Open a browser on Grafana at http://localhost:3000.

Create a datasource to obtain metrics from Promethesus:

Click on Add data source. In the form that opens:
Set Name to PromDS (or whatever name you choose)
Set Type to Prometheus
Set URL to http://localhost:9090
Set Access to proxy
Click Add and then Save and test

Import the example dashboard that displays metrics from the Fn server:

Click on the main menu at the top left and choose Dashboards and then Import
In the dialog that opens, click Upload .json file and specify fn_grafana_dashboard.json in this example's directory.
Specify the Prometheus data source that you just created
Click Import

You should then see the dashboard shown above. Now execute some functions and see the graphs update.

Tracing metrics

Tracing spans from the Fn server are available as Prometheus metrics. Each span has a name that describes the operation being performed (for example docker_wait_container), and its duration in seconds. Each span name is represented by a separate histogram metric, which has a name of the form fn_span_<span-name>_duration_seconds.

If the span is associated with a specific function invocation, the corresponding metric is given the labels fn_app and fn_path which are set to the application name and function path respectively.

A second example dashboard fn_grafana_dashboard2.json in this example's directory extends the first dashboard to show rate and duration data for a selection of tracing spans.