# HTTP API

Asks one address with a GET, or a POST of a fixed JSON body, every poll interval and reads what it answers as records.

Many services list their events or alerts at an address that answers a GET with JSON or lines of text, and some, such as Starlink's telemetry stream, answer a POST of what to send. This source polls such an address. A service that needs more than one request and its pages, such as Okta's System Log, has a source of its own.

## Example

Asks `api.example.com` for its events every minute. The source sends the `Authorization` header from the worker-host variable `API_TOKEN`, reads each element of the array at `/data` as one record, and follows the API's next links. It asks only for what happened since its last poll, through the `since` query parameter.

```
{
  "id": "events",
  "type": "httpApi",
  "url": "https://api.example.com/v1/events?limit=500",
  "credentials": {
    "Authorization": "API_TOKEN"
  },
  "records": "array",
  "recordsAt": "/data",
  "followLinks": true,
  "sinceParam": "since"
}
```

![The HTTP API form on the Sources page: id events, the URL, records array at /data, pages follow next links, since parameter since.](/http-api-source-form-light.png)

The same source in the Sources page's add dialog, before you add its credential.

## Before you start

-   Find the address that lists what you want, and check that a GET to it, or a POST of the body it asks for, answers with JSON or lines of text. If the answer is a JSON object, note the [JSON pointer](https://www.rfc-editor.org/rfc/rfc6901) to the array that holds the records, such as `/data`.
-   Find out how the API pages. This source follows a `Link: rel="next"` header. An API that pages another way gives one page per poll.
-   Find the query parameter that asks for what happened since a time, if the API has one. Without it, every poll reads everything again.
-   Put each secret header's whole value, such as `Bearer <token>`, in a [credential](/docs/secrets).
-   Allow workers to reach the address.

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`url`  string required

The whole address it asks, `http://` or `https://`, with any fixed query parameters. It sends a credential only over `https://`, or to this host.

Examples:

- `https://api.example.com/v1/events?limit=500`
- `https://siem.example.com:8443/api/alerts`

`method`  string default `GET`

How it asks.

`GET`

Asks with a GET.

`POST`

POSTs `body` on every request, for an API that is asked by what it is sent.

`body`  json default a POST of nothing

The JSON every POST carries, sent as `application/json` unless `headers` names a `Content-Type`. Only with `method` `POST`.

Examples:

- `{"batchSize": 1000,"maxLingerMs": 15000}`
- `{"query": "status:open","limit": 500}`

`credentials`  map default none

Headers whose values are secret, each name to the [credential](/docs/secrets) holding its whole value, such as `Bearer <token>`.

Examples:

- `Authorization = API_TOKEN`
- `X-Api-Key = VENDOR_API_KEY`

`clientId`  string default none

The OAuth client it signs in as, with `clientSecret` or `privateKey`.

Examples:

- `smeltr`
- `0oa1b2c3d4`

`clientSecret`  credential default none

The [credential](/docs/secrets) holding the OAuth client's secret. Only with `clientId`.

Examples:

- `API_CLIENT_SECRET`
- `{"secret": "api-client","version": 1}`

`privateKey`  credential default none

The [credential](/docs/secrets) holding the client's RSA private key, PEM PKCS#8. Replaces `clientSecret`.

Examples:

- `API_PRIVATE_KEY`
- `{"secret": "api-key","version": 1}`

`tokenUrl`  string default none

Where the client's token is obtained. Replaces `issuer`.

Example:

- `https://auth.example.com/oauth2/token`

`issuer`  string default none

The OpenID issuer, exactly as the provider writes it. Replaces `tokenUrl`.

Example:

- `https://login.example.com/realms/acme`

`records`  string default `body`

Where an answer's records are.

`body`

Gives the whole answer as one record.

`lines`

Gives one record per line.

`array`

Gives one record per element of the JSON array at `recordsAt`.

`recordsAt`  string default the answer is the array

The [JSON pointer](https://www.rfc-editor.org/rfc/rfc6901) to the array `array` reads, such as `/data` or `/result/items`. An answer with nothing there, or `null`, holds no records. Only with `records` `array`.

Examples:

- `/data`
- `/result/items`

`followLinks`  bool default `false`

Follows each answer's `Link: rel="next"` to the next page, within the same poll. A link to another host is refused, since the credentials go with every request.

`sinceParam`  string default every poll reads everything

The query parameter that asks for what happened since a time. Each poll sends it the start of the last poll whose records every destination took, in RFC 3339 UTC, such as `2026-10-10T13:00:00.000Z`.

Examples:

- `since`
- `start_time`

### Advanced

`headers`  map default none

Headers it sends on every request, each name to its value. Smeltr sets `Content-Length`, `Transfer-Encoding` and `Host` itself.

Examples:

- `Accept = application/json`
- `Accept = application/json`
- `X-Tenant = acme`

`identity`  identity default none

The [connection identity](/docs/secrets#identities) it connects under, for a private CA or a client certificate. Only on an `https://` URL.

Examples:

- `api-client`
- `corp-proxy`

`keyId`  string default none

The `kid` naming `privateKey`, for a client with several keys.

Example:

- `k1`

`scope`  string default none

The scope the client's token is asked for. Only with `clientId`.

Example:

- `read:events`

`start`  string default `everything`

What its first poll asks for.

`everything`

Asks without `sinceParam`, for everything the API gives.

`new`

Asks from the first poll's start. Needs `sinceParam`.

`sinceOverlapSecs`  integer default `0`

Seconds before the last poll's start that `sinceParam` asks from, for an API that records events late. What falls in the overlap is read twice. Only with `sinceParam`.

Examples:

- `30`
- `300`

`pollIntervalSecs`  integer default `60`

Seconds from the start of one poll to the start of the next. At least 1.

Examples:

- `300`
- `60`

## Fields it gives

Each record holds its text exactly as the API wrote it. `__source.url` holds the address of the page the source read it from.

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.url` | `str` | The address of the page an API source read the record from. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## Where it starts

Without `sinceParam`, every poll asks for everything the API lists, so the source delivers the same records again on every poll. Leave the `sinceParam` parameter out of `url`; the source adds it whenever a poll asks from a time. A `start` of `new` needs `sinceParam`.

With `sinceParam`, the first poll asks for everything, or with `start` set to `new`, for what happens from the first poll onward. Every later poll asks from the start of the last poll every destination took, less `sinceOverlapSecs`. A restarted worker resumes there, so it skips nothing. An API that records events late needs an overlap, and the events in the overlap arrive twice.

## How it works

One worker of the group holds the source, and the leader keeps how far it got. Each poll sends a GET, or a POST of `body`, to `url` with the committed headers and the credential headers. With `followLinks`, the source follows each answer's next link within the same poll, but only to the same scheme, host and port, because the credentials go with every request. A page that holds no records, or a link back to a page the poll already asked, ends the poll.

The source holds each answer whole, up to 256 MiB, and asks for the next page only once every record of the last one is handed on. It follows no redirects. Each request has 120 seconds to connect and read its whole answer. A request that runs longer, or an answer larger than 256 MiB, fails.

When the API answers 429 or 503, the source waits as long as its `Retry-After` asks, up to an hour. After any other failure it asks the same page again, waiting one second at first and up to five minutes.

## Verify

Find a record you know the API lists at the destination within a poll interval. On the Sources page, the **worker** column names the worker that holds the source.

## Troubleshooting

The worker log shows what the API answered.

**Refused the credentials (401 or 403)**

Check that the credential holds the header's whole value, such as `Bearer <token>`, and that the token may read the address.

**Answered 301 or 302, give the address it leads to**

Set `url` to the address the log names.

**The same records arrive on every poll**

Set `sinceParam` to the API's query parameter for a start time, or accept that each poll reads everything.

**Nothing arrives**

Check that `records` matches the answer, and with `array` that `recordsAt` points at the array. An answer with nothing at the pointer holds no records.

## See also

-   [Sources](/docs/sources)
-   [Okta System Log source](/docs/sources/okta-logs)
-   [Credentials](/docs/secrets)