# Smeltr metrics

Gives the worker's own metrics as records, one per series every bucket. It reads locally and needs no credentials.

## Example

Gives a record per series every 10 seconds. A shorter bucket gives records more often.

```
{
  "id": "worker-metrics",
  "type": "metrics",
  "bucketSeconds": 10
}
```

![The smeltr metrics form on the Sources page: id worker-metrics, bucket seconds 10.](/metrics-source-form-light.png)

The same source in the Sources page's add dialog, as it is added.

## Settings

It also takes the settings [every source](/docs/sources#settings) takes.

`bucketSeconds`  integer required

Seconds per bucket, at least 1. The form starts at 10.

Examples:

- `10`
- `1`
- `60`

## Fields it gives

| Field | Type | Holds |
| --- | --- | --- |
| `__source.name` | `str` | The name of the source that read the record. |
| `__source.type` | `str` | The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`. |
| `__source.topic` | `str` | The Kafka topic, or event hub, the record came from. A metrics record carries `__smeltr_metrics` and a logs record `__smeltr_logs`. |
| `__source.bytes` | `bytes` | The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text. |

## How it works

Each bucket, the source gives one record per series. `metric` holds the series' name, each label is a string field of its own, `value` holds the number, and `ts` holds the time of the bucket in Unix epoch milliseconds. Every record's `__source.topic` is `__smeltr_metrics`.

```
{
  "metric": "smeltr_rows_polled_total",
  "source": "orders-west",
  "thread": "reader-0",
  "value": 1250,
  "ts": 1791100800000
}
```

A counter's value is how much it grew since its last record, and a counter that did not grow gives no record. A gauge gives its current value every bucket. The first bucket comes at once and holds everything counted since the worker started. A bucket larger than the source's batch limits is cut, and the counts cut from it are lost.

Every record carries `thread`, the worker thread that counted it: `reader-N` for a source, `transform-N` for a pipeline thread, `egress-N` for a destination, `deploy` for the deployment as a whole, and `capture` for a sample capture. The `source`, `destination`, `route`, `pipeline` and `function` labels hold the names they have in the deployment.

| Metric | Kind | Labels | Measures |
| --- | --- | --- | --- |
| Sources |
| `smeltr_rows_polled_total` | counter | `source` | Records the source returned. |
| `smeltr_bytes_polled_total` | counter | `source` | Bytes of those records. |
| `smeltr_tick_poll_seconds_total` | counter | `source` | Seconds spent waiting on the source for records. |
| `smeltr_tick_idle_polls_total` | counter | `source` | Polls that returned no records. |
| `smeltr_commits_total` | counter | `source` | Offset commits the source accepted. |
| `smeltr_commits_dropped_total` | counter | `source` | Offset commits that failed and were let go. The next commit covers their records. |
| `smeltr_source_in_flight` | gauge | `source` | Batches read and not yet committed. |
| `smeltr_source_lag` | gauge | `source`,`topic`,`partition` | What the source has yet to finish. Kafka: records in the partition past the group's committed offset. SQS and Azure Storage Queue: messages on the queue, held ones included, as partition 0 of a topic named for the queue. |
| `smeltr_source_messages_finished_total` | counter | `source` | Queue messages the source finished with: every object they named read and committed. A queue's lag drains at this rate. |
| `smeltr_source_blocked` | gauge | `source` | 1 while the source is stopped on a record it may not pass: an exception or refusal answered `block`, or nothing could process or deliver the record. 0 once it reads again. |
| `smeltr_source_dropped_total` | counter | `source` | Records the source let go because it had no room for them. A syslog receiver over UDP counts each datagram that arrives while it is full or blocked. |
| `smeltr_source_full_total` | counter | `source` | Times a receiver had no memory left for what a sender offered: a request it answered busy, or a connection it stopped reading. Its memory, not its sender, set the pace. |
| `smeltr_source_poll_failures_total` | counter | `source` | Reads of the source that failed. The source is read again after a wait, and what it has not delivered stays uncommitted there. |
| `smeltr_source_refused_total` | counter | `source`,`reason` | Requests a receiver refused, by reason: token, unreadable, too\_large, unsupported, late, busy or other. |
| Processing |
| `smeltr_tick_batches_total` | counter | `source` | Batches run through routes and pipelines. |
| `smeltr_tick_process_seconds_total` | counter | `source` | Seconds spent running those batches. |
| `smeltr_tick_flush_seconds_total` | counter | `source` | Seconds spent handing processed batches to destinations. |
| `smeltr_batches_waiting_for_room_total` | counter | `source` | Hand-offs that waited because a destination's queue was full. |
| `smeltr_batches_poisoned_total` | counter | `source` | Batches a pipeline could not finish because it crashed on them. |
| `smeltr_batches_dlq_total` | counter | `source`,`destination` | Batches no pipeline could process, sent whole to a dead-letter destination. |
| `smeltr_route_matched_total` | counter | `source`,`route` | Records the route's condition matched. |
| `smeltr_rows_unmatched_total` | counter | `source`,`cause` | Records that met Unmatched. `cause` is `route` (no route matched) or `rule` (no delivery rule of its route took it). |
| `smeltr_rows_dropped_total` | counter | `source`,`reason` | Records let go. `reason` is `unmatched` (Unmatched answered drop), `exception` (an exception answered drop), `unprocessable` (no pipeline could process it) or `rejected` (a destination refused it for good). |
| `smeltr_rows_dlq_over_budget_total` | counter | `source` | Records sent to a dead-letter destination because one alone does not fit the memory budget. |
| Pipeline steps |
| `smeltr_function_calls_total` | counter | `pipeline`,`function` | Instructions the step ran. A step can compile to several, and one inside a loop counts on every pass. |
| `smeltr_function_seconds_total` | counter | `pipeline`,`function` | Seconds the step ran. A step inside a loop is not timed: its time goes to the loop step. |
| `smeltr_rows_dropped_total` | counter | `pipeline`,`function` | Records the step removed. |
| `smeltr_rows_errored_total` | counter | `pipeline`,`function`,`kind` | Records the step gave an exception, by its kind. |
| `smeltr_rows_in_total` | counter | `pipeline`,`function` | Records in each batch that reached a loop step, whether its condition holds for them or not. |
| `smeltr_rows_out_total` | counter | `pipeline`,`function` | Records the loop step's condition let in. |
| Destinations |
| `smeltr_rows_sent_total` | counter | `destination` | Records handed to the destination. |
| `smeltr_rows_without_doc_id_total` | counter | `destination` | Records sent to a `dedupe` or `overwrite` destination with no document ID. |
| `smeltr_rows_send_failed_total` | counter | `destination` | Records in a hand-over that failed. They are sent again. |
| `smeltr_flushes_total` | counter | `destination` | Hand-overs the destination finished. |
| `smeltr_rows_delivered_total` | counter | `destination` | Records in batches the destination acknowledged, including records it refused within them. |
| `smeltr_bytes_delivered_total` | counter | `destination` | Bytes of those records. |
| `smeltr_rows_rejected_total` | counter | `source`,`destination` | Records the destination refused for good. |
| `smeltr_rows_salvaged_total` | counter | `destination` | Refused records a dead-letter destination took. |
| `smeltr_rows_lost_total` | counter | `destination` | Refused records with nowhere left to go. |
| `smeltr_destination_healthy` | gauge | `destination` | 1 while the destination takes writes, 0 after a failed send. |
| `smeltr_destination_stalls_total` | counter | `destination` | Failed sends that marked the destination unhealthy. |
| `smeltr_destination_in_flight` | gauge | `destination` | Requests the destination may have open at once. |
| `smeltr_destination_held_rows` | gauge | `destination` | Rows the destination refused and keeps to send again. |
| `smeltr_destination_open_bytes` | gauge | `destination` | Bytes in a destination pending an upload. |
| `smeltr_destination_queue_bytes` | gauge | `destination` | Bytes waiting in the destination's queue. |
| `smeltr_destination_queue_budget_bytes` | gauge | `destination` | The most bytes the destination's queue holds. |
| `smeltr_destination_unwritten_total` | counter | `destination`,`kind` | What the store took but said it did not store as sent, in its own units: a Prometheus remote\_write 2.0 store's samples, histograms and exemplars, and documents an Elasticsearch failure store took. |
| `smeltr_destination_throttled_seconds_total` | counter | `destination` | Seconds the destination was held back because it was asked to slow down: by its brokers for Kafka, and in the backoff after a throttled answer for an HTTP destination (Elasticsearch, Splunk HEC, HTTP, Prometheus). Summed over concurrent requests. |
| `smeltr_destination_buffered_bytes` | gauge | `destination` | Bytes waiting inside the Kafka client. Kafka only. |
| `smeltr_destination_rtt_seconds` | gauge | `destination` | Mean round trip to the brokers in seconds. Kafka only. |
| Memory |
| `smeltr_memory_budget_bytes` | gauge |  | The worker's memory budget. 0 under `thread=deploy` means none. |
| `smeltr_memory_peak_bytes` | gauge |  | The most memory the worker's threads have held at once. Present only with a memory budget. |
| `smeltr_memory_splits_total` | counter | `source` | Batches split in half to fit the memory budget. |
| `smeltr_memory_abandoned_seconds_total` | counter | `source` | Seconds spent on attempts given up for memory. |
| `smeltr_destination_pool_bytes` | gauge |  | Bytes the destination queues sized to the worker's memory have borrowed above their reserves. At smeltr\_destination\_pool\_budget\_bytes, memory is what holds them back. |
| `smeltr_destination_pool_budget_bytes` | gauge |  | The most the destination queues sized to the worker's memory may borrow above their reserves, together. |
| `smeltr_receiver_pool_bytes` | gauge |  | Bytes the receivers sized to the worker's memory have borrowed from the pool they share. At smeltr\_receiver\_pool\_budget\_bytes, memory is what holds them back. |
| `smeltr_memory_allocated_bytes` | gauge |  | Bytes the worker's runtime has allocated and not freed. What its resident memory holds above this is what the allocator keeps of what was freed, and what C libraries allocate themselves. |
| `smeltr_receiver_pool_budget_bytes` | gauge |  | The most the receivers sized to the worker's memory may borrow above their reserves, together. |
| `smeltr_memory_part_bytes` | gauge | `part`,`source`,`destination` | The most one part of the deployment may hold, one sample per part: what `part` names, under the source or destination it belongs to when it belongs to one. Together they are what the worker's memory must fit. |
| `smeltr_source_held_bytes` | gauge | `source` | Bytes a source holds that its reader has not taken. |
| `smeltr_source_lane_bytes` | gauge | `source` | Bytes waiting between a source's reader and transform. |
| `smeltr_process_memory_bytes` | gauge | `process` | Bytes of memory a worker process holds: the supervisor's or the runtime's. |
| Worker |
| `smeltr_worker_threads_budget` | gauge |  | The most threads the deployment may use. |
| `smeltr_worker_slice_bytes` | gauge |  | The raw bytes of a batch a transform thread runs at once, sized to the CPU's cache. |
| `smeltr_worker_readers` | gauge |  | Reader threads, one per source. |
| `smeltr_worker_transform_threads` | gauge |  | Threads running pipelines. |
| `smeltr_worker_egress_threads` | gauge |  | Threads sending to destinations. |
| `smeltr_capture_records_total` | counter | `source` | Records a sample capture read. Its `source` label holds the source's name. |
| Logs |
| `smeltr_logs_written_total` | counter | `process`,`site` | Log events written since the process started, by the `site` that wrote them. |
| `smeltr_logs_dropped_total` | counter | `process` | Log events dropped for want of room since the process started. |
| `smeltr_logs_unread_total` | counter | `process` | Log events a full logs reader's queue refused since the process started, one count per reader. |
| `smeltr_logs_undelivered_total` | counter | `process` | Log events refused while the leader was out of contact, since the process started. |
| `smeltr_logs_unwritable_total` | counter | `process` | Log events the console would not take since the process started. |

## Verify

Wait at least one bucket, then find worker metrics at the destination. If none arrive, check the running commit and the route.

## See also

-   [Sources](/docs/sources)
-   [Smeltr logs source](/docs/sources/logs)
-   [Logs & metrics](/docs/logs-metrics)
-   [Operate the fleet](/docs/operations)