Prometheus receiver

Takes the samples Prometheus sends over remote_write to its port. It answers each request once every destination has its samples.

Prometheus and any agent that speaks remote_write 1.0 or 2.0 send to it. Every worker in the group listens on its port, and the sender, or a balancer in front of the workers, picks a worker for each request. Add workers to take more.

Example

Takes remote_write on port 9090, over TLS with the identity receiver-tls. Senders present the token held in the worker-host variable PROM_TOKEN.

{
  "id": "prometheus",
  "type": "prometheusReceiver",
  "port": 9090,
  "tokens": {
    "prometheus": "PROM_TOKEN"
  },
  "identity": "receiver-tls"
}

The Prometheus receiver form on the Sources page: id prometheus, port 9090, no tokens yet. — The same source in the Sources page's add dialog, before you add its token and identity.

Prometheus sends to it with a remote_write block like this:

remote_write:
  - url: https://workers.example.com:9090/api/v1/write
    authorization:
      credentials_file: /etc/prometheus/smeltr-token
    tls_config:
      ca_file: /etc/prometheus/smeltr-ca.pem

Before you start

Pick a port no other receiver in the group uses.
For native histograms, set send_native_histograms: true in the sender's remote_write settings.
Decide how senders prove who they are: a token, a client certificate, or both. See Tokens and certificates.
Put each token in a credential.
For exemplars, set send_exemplars: true in the sender's remote_write settings.
Open the port on every worker, and give senders one address that reaches them all. See Open its port.

Settings

It also takes the settings every source takes.

port integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples

909019291

tokens map default none

Each token's name to the credential holding it. A sender presents one as Authorization: Bearer <token>, and any other request is refused 401. A receiver with no token needs an identity with authorities.

Examples

prometheus = PROM_TOKEN{"agents": {"secret": "prom-agents","version": 1}}

Advanced

identity identity default plain HTTP

The connection identity it serves TLS under: its certificate and key. With authorities, a sender must present a certificate that chains to one of them.

Example

receiver-tls

maxRequestBytes integer default 10485760

The largest request body it reads, between 1024 and 67108864. A larger one is refused 413, which a sender may drop.

Examples

104857667108864

Paths it serves

POST /api/v1/write: A remote_write 1.0 or 2.0 request, as the sender sends it.
GET /health: Says whether the receiver takes requests. See Health check.

Records it gives

Each sample of a request becomes one record: a float sample, or a native histogram. A series of three samples is three records, each with the series' labels. Each field has the name and type of an OTLP metric point's field, so a pipeline reads a metric from either receiver alike. A field a sample does not have is null.

time: A datetime, from the sample's milliseconds.
name: The series' metric name, its __name__ label.
value: A variant holding the sample's double, bit for bit, so a staleness marker stays one. Read it as a number with value as f64.
attributes: The series' other labels, an object of strings, or null when the series has no label but __name__. Read one with attributes["job"] as str.
type: A native histogram's: exponentialHistogram, or histogram for one with custom buckets. Null for a float sample.
aggregationTemporality: 2, cumulative, for a counter's histogram; null for a gauge histogram.
count, sum: A histogram's count and sum, doubles.
scale, zeroCount, zeroThreshold, positive, negative: An exponential histogram's, as OpenTelemetry lays it out. positive and negative each hold offset and bucketCounts, an array of doubles.
resetHint: yes or no where the sender said whether the histogram's counter reset, and null where it did not say.
bucketCounts, explicitBounds: A histogram of custom buckets: its bucket counts and their bounds, as OpenTelemetry lays them out.
exemplars: The series' exemplars, on its last record of the request, each shaped as an OTLP point's exemplar: time, value, traceId, spanId, and its other labels in filteredAttributes. A series sent with exemplars alone is one record, its value null.
startTime: When the series started counting, which remote_write 2.0 may send with a sample or histogram. Null from 1.0, and where the sender did not say.

Each metric family's metadata is a record of its own. Its time and value are null, name is the family's, description its help and unit its unit. Its type is as an OTLP metric's: a counter is sum with isMonotonic true. A pipeline that wants samples alone drops records with no time.

Over remote_write 2.0, a family's metadata record comes once a request, ahead of the family's first series.

Classic histograms arrive as samples of their _bucket, _sum and _count series, as a classic histogram is laid out.

This pipeline keeps one metric and reads its job as text:

if (name != "http_requests_total") drop();
job = attributes["job"] as str;

Fields it gives

__source.remote holds the IP address the request came from: the balancer's, when one is in front of the workers.

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.remote`	`str`	The address of the sender that delivered the record to a receiver.

Tokens and certificates

Anyone a receiver admits can put records into the group's pipelines, so a receiver needs tokens, an identity with authorities, or both. The leader refuses a receiver with neither.

Tokens: a sender presents one as Authorization: Bearer <token>, Prometheus's authorization block. The receiver refuses any other request before reading its body. Without an identity, the token crosses the network in the clear, so use a token alone only behind a balancer that terminates TLS.
Client certificates: with an identity, the receiver serves TLS with the identity's certificate and key. If the identity has authorities, every connection, health checks included, must present a certificate that chains to one of them: Prometheus's tls_config with cert_file and key_file.

With both, a sender presents a certificate and a token.

What it answers

The receiver answers success only once every destination has the request's samples, so a sender that retries an unanswered request loses nothing. The receiver takes a request whole or refuses it whole, and each refusal's body says why.

204: Every destination has the request's samples.
503, with Retry-After: The receiver is full, blocked or stopping, or the sender took more than 30 seconds to send the whole body. A sender may send it again.
401: The request has no token, gives one other than as Bearer <token>, or gives one the receiver does not hold.
400: The body does not read as remote_write, or it holds a series or histogram remote_write does not allow, such as a label with an empty name or a histogram whose count is not what its buckets hold. The answer's body names what is wrong. A sender may drop it.
413: The body is larger than maxRequestBytes, or what it holds decompressed and decoded is more than a receiver holds at once. Send smaller requests, or raise maxRequestBytes.
415: The request is neither remote_write 1.0 nor 2.0. The answer's body names what is wrong.

Health check

GET /health on the receiver's port answers 200 while the receiver takes requests and 503 while it is full, blocked or stopping. It needs no token. Point a balancer's health check or a readiness probe at it, so a busy worker gets no requests until it can take them. Under an identity with authorities it needs a client certificate like any other request.

Senders it takes

Any sender of remote_write 1.0 or 2.0 works.

Prometheus's remote_write, in server or agent mode. Set protobuf_message: io.prometheus.write.v2.Request to send 2.0, which carries each series' metadata and start time.
Grafana Alloy's prometheus.remote_write.
VictoriaMetrics' vmagent.
The OpenTelemetry Collector's prometheusremotewrite exporter, exponential histograms included.

Grafana Alloy sends to it what its scrape components forward to prometheus.remote_write.smeltr.receiver, with PROM_TOKEN set in its environment:

prometheus.remote_write "smeltr" {
  endpoint {
    url = "https://workers.example.com:9090/api/v1/write"
    bearer_token = sys.env("PROM_TOKEN")
    tls_config {
      ca_file = "/etc/alloy/smeltr-ca.pem"
    }
  }
}

Open its port

Every worker in the group listens on the receiver's port. Open that port to senders on each worker, and give senders one address that reaches them all, such as a balancer's. On a Unix host, a request sent while a worker deploys a new configuration waits, and the new configuration answers it.

On Kubernetes, add the port to the worker pods and put a Service in front of them: Expose a receiver.
On a host or in a container, open the host's firewall or publish the container's port: Enroll a worker on a host.

Verify

Point a sender at the receiver and check in its own logs or metrics that its requests succeed. Then find its samples at the destination.

Troubleshooting

The sender cannot connect: Check that the port is open on every worker and that the sender's address reaches them. An identity with authorities refuses a connection without a certificate that chains to one of them.
Every request gets 401: Check that the sender presents Authorization: Bearer <token>, with a token one of the receiver's credentials holds.
Every request gets 415: The sender does not speak remote_write 1.0 or 2.0. The answer's body names what is wrong.
Every request gets 503: Check the source's status. A blocked receiver takes nothing until its destinations take records again. A full one takes more with more workers.
The same samples arrive twice: A sender that hears no answer may send the request again. Give it a remote_timeout longer than its destinations take to deliver.

Prometheus receiver

Example

Before you start

Settings

Advanced

Paths it serves

Records it gives

Fields it gives

Tokens and certificates

What it answers

Health check

Senders it takes

Open its port

Verify

Troubleshooting

See also