HTTP API

Asks one address with a GET, or a POST of a fixed JSON body, every poll interval and reads what it answers as records.

Many services list their events or alerts at an address that answers a GET with JSON or lines of text, and some, such as Starlink's telemetry stream, answer a POST of what to send. This source polls such an address. A service that needs more than one request and its pages, such as Okta's System Log, has a source of its own.

Example

Asks api.example.com for its events every minute. The source sends the Authorization header from the worker-host variable API_TOKEN, reads each element of the array at /data as one record, and follows the API's next links. It asks only for what happened since its last poll, through the since query parameter.

{
  "id": "events",
  "type": "httpApi",
  "url": "https://api.example.com/v1/events?limit=500",
  "credentials": {
    "Authorization": "API_TOKEN"
  },
  "records": "array",
  "recordsAt": "/data",
  "followLinks": true,
  "sinceParam": "since"
}
The HTTP API form on the Sources page: id events, the URL, records array at /data, pages follow next links, since parameter since.
Open complete screenshot ↗
The HTTP API form on the Sources page: id events, the URL, records array at /data, pages follow next links, since parameter since.
Open complete screenshot ↗
The same source in the Sources page's add dialog, before you add its credential.

Before you start

  • Find the address that lists what you want, and check that a GET to it, or a POST of the body it asks for, answers with JSON or lines of text. If the answer is a JSON object, note the JSON pointer to the array that holds the records, such as /data.
  • Find out how the API pages. This source follows a Link: rel="next" header. An API that pages another way gives one page per poll.
  • Find the query parameter that asks for what happened since a time, if the API has one. Without it, every poll reads everything again.
  • Put each secret header's whole value, such as Bearer <token>, in a credential.
  • Allow workers to reach the address.

Settings

It also takes the settings every source takes.

url string required

The whole address it asks, http:// or https://, with any fixed query parameters. It sends a credential only over https://, or to this host.

Examples
https://api.example.com/v1/events?limit=500https://siem.example.com:8443/api/alerts

method string default GET

How it asks.

GET
Asks with a GET.
POST
POSTs body on every request, for an API that is asked by what it is sent.

body json default a POST of nothing

The JSON every POST carries, sent as application/json unless headers names a Content-Type. Only with method POST.

Examples
{"batchSize": 1000,"maxLingerMs": 15000}{"query": "status:open","limit": 500}

credentials map default none

Headers whose values are secret, each name to the credential holding its whole value, such as Bearer <token>.

Examples
Authorization = API_TOKENX-Api-Key = VENDOR_API_KEY

clientId string default none

The OAuth client it signs in as, with clientSecret or privateKey.

Examples
smeltr0oa1b2c3d4

clientSecret credential default none

The credential holding the OAuth client's secret. Only with clientId.

Examples
API_CLIENT_SECRET{"secret": "api-client","version": 1}

privateKey credential default none

The credential holding the client's RSA private key, PEM PKCS#8. Replaces clientSecret.

Examples
API_PRIVATE_KEY{"secret": "api-key","version": 1}

tokenUrl string default none

Where the client's token is obtained. Replaces issuer.

Example
https://auth.example.com/oauth2/token

issuer string default none

The OpenID issuer, exactly as the provider writes it. Replaces tokenUrl.

Example
https://login.example.com/realms/acme

records string default body

Where an answer's records are.

body
Gives the whole answer as one record.
lines
Gives one record per line.
array
Gives one record per element of the JSON array at recordsAt.

recordsAt string default the answer is the array

The JSON pointer to the array array reads, such as /data or /result/items. An answer with nothing there, or null, holds no records. Only with records array.

Examples
/data/result/items

followLinks bool default false

Follows each answer's Link: rel="next" to the next page, within the same poll. A link to another host is refused, since the credentials go with every request.

sinceParam string default every poll reads everything

The query parameter that asks for what happened since a time. Each poll sends it the start of the last poll whose records every destination took, in RFC 3339 UTC, such as 2026-10-10T13:00:00.000Z.

Examples
sincestart_time

Advanced

headers map default none

Headers it sends on every request, each name to its value. Smeltr sets Content-Length, Transfer-Encoding and Host itself.

Examples
Accept = application/jsonAccept = application/jsonX-Tenant = acme

identity identity default none

The connection identity it connects under, for a private CA or a client certificate. Only on an https:// URL.

Examples
api-clientcorp-proxy

keyId string default none

The kid naming privateKey, for a client with several keys.

Example
k1

scope string default none

The scope the client's token is asked for. Only with clientId.

Example
read:events

start string default everything

What its first poll asks for.

everything
Asks without sinceParam, for everything the API gives.
new
Asks from the first poll's start. Needs sinceParam.

sinceOverlapSecs integer default 0

Seconds before the last poll's start that sinceParam asks from, for an API that records events late. What falls in the overlap is read twice. Only with sinceParam.

Examples
30300

pollIntervalSecs integer default 60

Seconds from the start of one poll to the start of the next. At least 1.

Examples
30060

Fields it gives

Each record holds its text exactly as the API wrote it. __source.url holds the address of the page the source read it from.

FieldTypeHolds
__source.namestrThe name of the source that read the record.
__source.typestrThe kind of source that read it, its type, such as kafka, s3List, httpApi or hecReceiver.
__source.urlstrThe address of the page an API source read the record from.
__source.bytesbytesThe record's payload as the source read it, before any decoding. The source decodes _raw from it. A bytes destination writes it unless the destination names another field. Text functions refuse it; decode(__source.bytes, encoding) reads it as text.

Where it starts

Without sinceParam, every poll asks for everything the API lists, so the source delivers the same records again on every poll. Leave the sinceParam parameter out of url; the source adds it whenever a poll asks from a time. A start of new needs sinceParam.

With sinceParam, the first poll asks for everything, or with start set to new, for what happens from the first poll onward. Every later poll asks from the start of the last poll every destination took, less sinceOverlapSecs. A restarted worker resumes there, so it skips nothing. An API that records events late needs an overlap, and the events in the overlap arrive twice.

How it works

One worker of the group holds the source, and the leader keeps how far it got. Each poll sends a GET, or a POST of body, to url with the committed headers and the credential headers. With followLinks, the source follows each answer's next link within the same poll, but only to the same scheme, host and port, because the credentials go with every request. A page that holds no records, or a link back to a page the poll already asked, ends the poll.

The source holds each answer whole, up to 256 MiB, and asks for the next page only once every record of the last one is handed on. It follows no redirects. Each request has 120 seconds to connect and read its whole answer. A request that runs longer, or an answer larger than 256 MiB, fails.

When the API answers 429 or 503, the source waits as long as its Retry-After asks, up to an hour. After any other failure it asks the same page again, waiting one second at first and up to five minutes.

Verify

Find a record you know the API lists at the destination within a poll interval. On the Sources page, the worker column names the worker that holds the source.

Troubleshooting

The worker log shows what the API answered.

Refused the credentials (401 or 403)
Check that the credential holds the header's whole value, such as Bearer <token>, and that the token may read the address.
Answered 301 or 302, give the address it leads to
Set url to the address the log names.
The same records arrive on every poll
Set sinceParam to the API's query parameter for a start time, or accept that each poll reads everything.
Nothing arrives
Check that records matches the answer, and with array that recordsAt points at the array. An answer with nothing at the pointer holds no records.

See also