Prometheus connector

Default queries target prom-client conventions — the de-facto standard for Node.js/Express instrumentation. Most apps that expose /metrics via prom-client work out of the box without any source-level overrides.

Default metrics

For each synthetic metric the connector probes a list of candidate series and picks the first one that actually exists in the backend. This makes the same MCP work for prom-client apps and node_exporter hosts without per-source configuration.

Metric	First candidate (prom-client)	Fallback (node_exporter)
cpu	rate(process_cpu_seconds_total[1m]) * 100	100 - avg(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100
memory	process_resident_memory_bytes	node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
request_rate	sum(rate(http_requests_total[1m]))	— (HTTP-app concept)
error_rate	sum(rate(http_requests_total{status=~"5.."}[1m]))	—
latency_p99	histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[1m])) by (le))	—
latency_p50	same with quantile 0.50	—
latency_avg	sum(rate(_sum[1m])) / sum(rate(_count[1m]))	—

The probe queries /api/v1/series?match[]=<seriesName>{<label>="<service>"} for each candidate per service (cached 60 s), so a Prometheus that holds both prom-client apps and node_exporter hosts in the same instance still picks the right candidate per service. The selected candidate is reflected in the response's resolvedSeries field.

Dynamic label resolution

The {{selector}} placeholder is resolved at query time. The connector probes a list of labels and uses the first one that contains the requested service name as a value:

1. job
2. service
3. app
4. service_name

So query_metrics(service="my-app") issues /api/v1/label/job/values, finds my-app, then runs ... process_cpu_seconds_total{job="my-app"} ....

If no label matches, the first label in the list is used as a fallback. Override the order via PROMETHEUS_SERVICE_LABELS:

bash PROMETHEUS_SERVICE_LABELS=service,job npx @thotischner/observability-mcp

Label values are cached per-label for 60 seconds.

Per-instance breakdown (groupBy)

When a service is scraped on multiple targets (dev + prod, k8s replicas, etc.), the default queries collapse all of them into one number. Pass groupBy to break the result down by any label:

query_metrics(service="api", metric="cpu", groupBy="instance")

The connector swaps the synthetic-metric template for the groupedQuery variant — e.g. sum(rate(...)) becomes sum by(instance) (rate(...)), and histogram_quantile(... by (le)) becomes ... by (le, instance). The response shape becomes:

json { "metric": "cpu", "groupBy": "instance", "groups": [ { "key": "prod-vm-1:9100", "values": [...], "summary": {...} }, { "key": "dev-vm-1:9100", "values": [...], "summary": {...} } ], "values": [...], // first group's series, kept for back-compat "summary": {...} }

If only one group exists, the groups array is omitted and the result keeps the single-series shape.

When you call query_metrics without groupBy and the underlying series has more than one distinct instance (or pod) value for that service, the response includes a hint field telling you the breakdown is available:

json "hint": "2 distinct instances exist for this service. Pass groupBy=\"instance\" to break the result down."

resolvedSeries and resolvedLabel

Every query_metrics response includes the actual PromQL executed and the label that was matched. When results look surprising, check these first.

json { "metric": "cpu", "values": [...], "resolvedSeries": "rate(process_cpu_seconds_total{ job=\"my-app\" }[1m]) * 100", "resolvedLabel": "job" }

Overriding a single metric

Source-level metrics entries merge with defaults by name. Pin one metric without re-listing the rest:

yaml sources: - name: prometheus type: prometheus url: http://prometheus:9090 enabled: true metrics: - name: cpu query: 'my_custom_cpu_gauge{job="{{service}}"}' unit: percent description: Project-specific CPU metric

{{service}} (literal name) and {{selector}} (full label="value" pair) are both supported in custom queries.

Compatibility with managed Prometheus

The connector works with Grafana Cloud Mimir, AWS Managed Prometheus, and Chronosphere without flags. Health checks probe /api/v1/query?query=up and service discovery falls back to /api/v1/label/job/values when /api/v1/targets is unavailable.