How

In order to implement the Prometheus monitoring, we need to take these steps.

Include Prometheus Java Client

We need to add these dependencies to SonarQube:

Define metrics using Prometheus Java Client
The list of metrics that needs to be defined is here

Defining a metric using the library is straightforward. For example, to define a metric of SUMMARY type time_to_complete_ce_task_in_seconds we need to call:

The metric automatically registers itself in the default CollectorRegistry which is good enough for us.

(Periodically) Collect data for metrics
For each metric we need to hook into different mechanisms in the SonarQube application and in the end, call the methods like Counter.inc() or Summary.observe(double) or Summary.observeDuration() to update the metrics. For some(or most) of the metrics i.e. uptime_minutes we will need to schedule a Timer that will collect the data in the interval and update Metric object from the library. We should not be doing anything collection of the data when we receive request to /metrics endpoint, the data should already be well prepared.

For DCE each node should collect metrics on its own. So there is no need to synchronize the data between the nodes.
For exposing ElasticSearch related metrics we could use this plugin https://github.com/vvanholl/elasticsearch-prometheus-exporter

Expose /api/monitoring/metrics endpoint

We need to define new endpoint api/monitoring/metrics and use Prometheus Java Client to retrieve metrics from the default CollectorRegistry and return them in the OpenMetrics format.

Api endpoint api/monitoring/metrics authentication

api/monitoring/metrics can be accessed using:

• "X-Sonar-passcode: xxxxx" header during database upgrade (Web process in Safe Mode), and when SonarQube is fully operational

• "Authorization: Bearer xxxxx" header during database upgrade (Web process in Safe Mode), and when SonarQube is fully operational

• username:password & JWT token when SonarQube is fully operational. System administrator permission is required.

X-Sonar-passcode and Bearer token can be defined in sonar.properties with the "sonar.web.systemPasscode" property.

Operate
In our Helm charts we need to provide a pod/service monitor definition in order to easily use this new endpoint 

Documentation
We need to document what metrics we are exposing and maybe short references to Prometheus documentation on how to make use of them