HTTP receiver

Takes what senders POST to a path on its port, and answers each request once every destination has its records.

A webhook, a log shipper or a script sends to it. Every worker of the group listens on the receiver's port, and the sender, or a balancer in front of the workers, picks one worker for each request. More workers take more.

Example

Takes what a CI system POSTs to /hooks/ci on port 8088, one record per line, over TLS with the identity receiver-tls. Senders present the token held in the worker-host variable HOOK_TOKEN.

{
  "id": "ci-hooks",
  "type": "httpReceiver",
  "port": 8088,
  "path": "/hooks/ci",
  "tokens": {
    "ci": "HOOK_TOKEN"
  },
  "identity": "receiver-tls",
  "records": "lines"
}
The HTTP receiver form on the Sources page: id ci-hooks, port 8088, path /hooks/ci, records one per line, no tokens yet.
Open complete screenshot ↗
The HTTP receiver form on the Sources page: id ci-hooks, port 8088, path /hooks/ci, records one per line, no tokens yet.
Open complete screenshot ↗
The same source in the Sources page's add dialog, before you add its token and identity.

A sender posts a file of lines like this:

curl --data-binary @build.log \
  -H "Authorization: Bearer $HOOK_TOKEN" \
  https://workers.example.com:8088/hooks/ci

Before you start

  • Pick a port no other receiver of the group listens on.
  • Decide how senders prove who they are: a token, a client certificate, or both. See Tokens and certificates.
  • Put each token in a credential.
  • Open the port on every worker of the group, and give senders one address that reaches them. See Open its port.

Settings

It also takes the settings every source takes.

port integer required

The port it listens on. Every worker of the group binds it, so no two receivers share one.

Examples
80889880

path string default /

The path senders POST to. It starts with / and holds no ?, #, :, * or space. Not /health, where it answers health checks.

Examples
/hooks/github/ingest

tokens map default none

Each token's name to the credential holding it. A sender presents one as Authorization: Bearer <token>, and any other request is refused 401. A receiver with no token needs an identity with authorities.

Examples
ci = HOOK_TOKENgithub = GITHUB_HOOK_TOKENjenkins = JENKINS_HOOK_TOKEN

records string default body

Where a request body's records are.

body
Gives the whole body as one record.
lines
Gives one record per line.
array
Gives one record per element of the JSON array at recordsAt.

recordsAt string default the body is the array

The JSON pointer to the array array reads. A body with no array there is refused 400. Only with records array.

Examples
/events/records

Advanced

identity identity default plain HTTP

The connection identity it serves TLS under: its certificate and key. With authorities, a sender must present a certificate that chains to one of them.

Example
receiver-tls

maxRequestBytes integer default 10485760

The largest request body it reads, between 1024 and 67108864. A larger one is refused 413. A body sent as gzip is held to it as it inflates.

Examples
104857667108864

Fields it gives

Each record holds its text exactly as the sender wrote it. __source.remote holds the IP address the request came from: the balancer's, when one is in front of the workers.

FieldTypeHolds
__source.namestrThe name of the source that read the record.
__source.typestrThe kind of source that read it, its type, such as kafka, s3List, httpApi or hecReceiver.
__source.remotestrThe address of the sender that delivered the record to a receiver.
__source.bytesbytesThe record's payload as the source read it, before any decoding. The source decodes _raw from it. A bytes destination writes it unless the destination names another field. Text functions refuse it; decode(__source.bytes, encoding) reads it as text.

Tokens and certificates

Whoever a receiver admits puts records into the group's pipelines, so a receiver needs tokens, an identity with authorities, or both. A receiver with neither is refused at load.

  • Tokens: a sender presents one as Authorization: Bearer <token>. Any other request is refused 401 before any of its body is read. Without an identity, the token crosses the network in the clear, which suits a receiver behind a balancer that ends TLS for it.
  • Client certificates: under an identity, the receiver serves TLS with the identity's certificate and key. When the identity has authorities, every connection must present a certificate that chains to one of them, health checks included.

With both, a sender presents a certificate and a token.

What it answers

The receiver answers 200 only once every destination has the request's records, so a sender that retries an unanswered request loses nothing.

200
Every destination has the request's records.
503 with Retry-After
The receiver is full, blocked, or stopping. It answers a new request at once, and a stopping receiver answers every request it holds. A sender may send it again.
401
The request presents none of the receiver's tokens.
400
The body does not read as records says, such as one with no array at recordsAt. It is refused whole: none of its records is taken.
413
The body, or what it inflates to, is larger than maxRequestBytes.
415
The body uses a Content-Encoding other than gzip or identity. The receiver inflates a gzip body itself.
408
The sender did not send the whole body within 30 seconds.

Health check

GET /health on the receiver's port answers 200 while the receiver takes requests and 503 while it is full, blocked or stopping. It needs no token, and says only whether the receiver takes requests. Point a balancer's health check or a readiness probe at it, so a busy worker gets no requests until it can take them. Under an identity with authorities it needs a client certificate like any other request.

Open its port

Every worker of the group listens on the receiver's port. Open that port to senders on each worker, and give senders one address that reaches all of them, such as a balancer's. On a Unix host, a request sent while a worker deploys a new configuration waits for it and is answered by the new one.

  • On Kubernetes, add the port to the worker pods and a Service in front of them: Expose a receiver.
  • On a host or in a container, open the host's firewall or publish the container's port: Enroll a worker on a host.

Verify

POST a record you can recognize, as in the example. The receiver answers 200 once every destination has it. Find it at the destination.

Troubleshooting

The sender cannot connect
Check that the port is open on every worker and that the sender's address reaches them. An identity with authorities refuses a connection with no certificate that chains to them.
Every request is answered 401
Check that the sender presents Authorization: Bearer <token> with a token one of the receiver's credentials holds.
Every request is answered 503
Check the source's status. A blocked receiver takes nothing until its destinations take records again. A full one takes more with more workers.
The same records arrive twice
A sender that hears no answer may send the request again. Give the sender a timeout longer than its destinations take to deliver.

See also