# Sources

A source reads records from a system and hands them to the routes that name it. Each kind of source has a page of its own.

## Kinds

### Pull, all workers

Every worker of the group reads it, and the queue or stream splits the work between them. More workers read more.

| Kind | type | What it does |
| --- | --- | --- |
| [Amazon S3 via SQS](/docs/sources/s3-sqs) | `s3Sqs` | Read the objects S3 announces on an SQS queue. |
| [Azure Blob via Storage queue](/docs/sources/azure-blob-queue) | `azureBlobQueue` | Read the blobs Event Grid announces on a Storage queue. |
| [Azure Event Hubs](/docs/sources/event-hubs) | `eventHubs` | Read event hubs through their Kafka endpoint. |
| [Google Cloud Storage via Pub/Sub](/docs/sources/gcs-pubsub) | `gcsPubSub` | Read the objects Cloud Storage announces on a Pub/Sub subscription. |
| [Kafka](/docs/sources/kafka) | `kafka` | Read topic streams or a range of them. |

### Pull, one worker

One worker of the group reads it at a time, and the leader keeps its progress. If that worker goes silent another takes over. More workers do not read it faster.

| Kind | type | What it does |
| --- | --- | --- |
| [Amazon S3](/docs/sources/s3-list) | `s3List` | Read the objects under a prefix of a bucket that cannot notify. |
| [Azure Blob Storage](/docs/sources/azure-blob-list) | `azureBlobList` | Read the blobs under a prefix of a container that cannot notify. |
| [Google Cloud Storage](/docs/sources/gcs-list) | `gcsList` | Read the objects under a prefix of a bucket that cannot notify. |
| [HTTP API](/docs/sources/http-api) | `httpApi` | Poll an address that answers a GET with JSON or lines of text. |
| [Okta System Log](/docs/sources/okta-logs) | `oktaLogs` | Read an Okta org’s System Log, as an API token or a service app. |
| [PostgreSQL](/docs/sources/postgres) | `postgres` | Run a query against a PostgreSQL database every poll and read its rows. |
| [MySQL and MariaDB](/docs/sources/mysql) | `mysql` | Run a query against a MySQL or MariaDB database every poll and read its rows. |
| [SFTP](/docs/sources/sftp) | `sftp` | Read the files under a directory on an SFTP server. |

### Push, all workers

Senders connect to it, and every worker of the group listens. More workers take more.

| Kind | type | What it does |
| --- | --- | --- |
| [HTTP receiver](/docs/sources/http-receiver) | `httpReceiver` | Take what senders POST to a path on a port every worker listens on. |
| [HEC receiver](/docs/sources/hec-receiver) | `hecReceiver` | Take what Splunk HTTP Event Collector clients send to a port every worker listens on. |
| [S2S receiver](/docs/sources/s2s-receiver) | `s2sReceiver` | Take what Splunk forwarders send over S2S to a port every worker listens on. |
| [OTLP receiver](/docs/sources/otlp-receiver) | `otlpReceiver` | Take the logs OpenTelemetry exporters send over OTLP/HTTP to a port every worker listens on. |
| [Syslog receiver](/docs/sources/syslog-receiver) | `syslogReceiver` | Take what syslog senders send over TCP or TLS to a port every worker listens on. |
| [Prometheus receiver](/docs/sources/prometheus-receiver) | `prometheusReceiver` | Take the samples Prometheus sends over remote\_write to a port every worker listens on. |
| [Elasticsearch receiver](/docs/sources/elastic-receiver) | `elasticReceiver` | Take what Beats, Fluent Bit and other Elasticsearch clients send to a port every worker listens on. |

### Local, each worker

Each worker reads what is on its own host.

| Kind | type | What it does |
| --- | --- | --- |
| [Smeltr logs](/docs/sources/logs) | `logs` | Collect each worker’s own log records. |
| [Smeltr metrics](/docs/sources/metrics) | `metrics` | Collect each worker’s own metrics. |

## How it works

*Diagram: From a source to its destinations, and back to the commit. The reader polls a batch from the source and tracks its position. Transform routes the batch and runs its pipelines. Egress, one per destination, writes each destination's records and retries until they land. Each destination acknowledges its records, and once every destination the batch reached has acknowledged it, the reader commits the batch's position to the source.*

A position is committed only after every destination the batch reached has acknowledged it, oldest batch first. After a restart or a retry a record can arrive twice, but it is not lost. Smeltr metrics and logs sources keep no position.

What happens to a record decides when its batch's position commits. Until it commits, a restart reads the batch again.

| The record | The position commits |
| --- | --- |
| Is acknowledged by every destination it was sent to | Once every older batch has committed too. |
| Is dropped by a step, or picked by no deliver rule | With its batch. |
| Reaches a destination that does not answer | Only once it lands. Egress retries, and a destination that stays down pauses the sources feeding it. |
| Carries an exception | By the answer to it, as below. A pipeline's exceptions are answered by its route's `exceptions` over the group's; Unmatched and Unprocessable by the group's; Refused by the destination's over the group's. |

| Answer | The position commits |
| --- | --- |
| `ignore`, `tag` | Once the destinations the record goes on to acknowledge it. |
| `drop` | With its batch. |
| `deliver` | Once every destination named acknowledges the record. |
| `block` | Not until the record goes through. A refused record is sent again until the destination takes it; any other stops its source until a new config is deployed. |

Unmatched, Unprocessable and Refused block unless an answer says otherwise, and take no `ignore` or `tag`. [Exceptions](/docs/routing/exceptions#answers) covers each answer.

## Settings

A source is one entry of the bundle's Sources array. Every kind takes these settings:

`id`  string required

The source's name, unique among sources. Routes name it, and its records carry it in `__source.name`.

Examples:

- `events-in`
- `firewall-syslog`

## Fields

Every record carries its text in `_raw`, unless its source's `encoding` is `none`. An Avro or SQL source gives its declared fields instead. Each kind of source adds fields of its own.

| Field | Type | Holds |
| --- | --- | --- |
| `_raw` | `str` | The text the source received, decoded with the source's `encoding`. A parser reads it. The destination delivers it with the record unless a step sets it to `null`. A source whose `encoding` is `none` gives no `_raw`. |

## Add a source

1.  Select the worker group and work on a branch.
2.  Open Build → Sources and choose **\+ Add source**.
3.  Choose how the group reads it (**Pull, all workers**, **Pull, one worker**, **Push, all workers** or **Local, each worker**), then its type. Give it a unique id and fill in its settings, then choose **Add**. Save the page.
4.  Connect it to a route, which sends its records through a pipeline to a [destination](/docs/destinations). [Follow the routing example](/docs/first-pipeline#route).
5.  [Review and deploy](/docs/review), then [verify it](#verify).

## Verify

1.  Check that the intended workers are online and run the deployed commit.
2.  Send a record you can recognize into the source's system.
3.  Check Status and Logs for source, routing or destination errors.
4.  Find the record at the destination and check its fields.

A preview of a saved sample checks the pipeline, not the live connection.

## Troubleshooting

**No records arrive**

Follow [No records arriving](/docs/troubleshooting#no-output) to check input, worker configuration, routing, filtering and delivery in order.

**TLS verification fails**

Check the server name and the authority in the connection identity. If the server uses a private CA, the connector must trust it.

## See also

-   [Destinations](/docs/destinations)
-   [First pipeline tutorial](/docs/first-pipeline)
-   [Routing records](/docs/routing/records)
-   [Recover and replay](/docs/delivery#recovery)
-   [Secrets & credentials](/docs/secrets)