Smeltr metrics

Gives the worker's own metrics as records, one per series every bucket. It reads locally and needs no credentials.

Example

Gives a record per series every 10 seconds. A shorter bucket gives records more often.

{
  "id": "worker-metrics",
  "type": "metrics",
  "bucketSeconds": 10
}

The smeltr metrics form on the Sources page: id worker-metrics, bucket seconds 10. — The same source in the Sources page's add dialog, as it is added.

Settings

It also takes the settings every source takes.

bucketSeconds integer required

Seconds per bucket, at least 1. The form starts at 10.

Examples

10160

Fields it gives

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.topic`	`str`	The Kafka topic, or event hub, the record came from. A metrics record carries `__smeltr_metrics` and a logs record `__smeltr_logs`.
`__source.bytes`	`bytes`	The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text.

How it works

Each bucket, the source gives one record per series. metric holds the series' name, each label is a string field of its own, value holds the number, and ts holds the time of the bucket in Unix epoch milliseconds. Every record's __source.topic is __smeltr_metrics.

{
  "metric": "smeltr_rows_polled_total",
  "source": "orders-west",
  "thread": "reader-0",
  "value": 1250,
  "ts": 1791100800000
}

A counter's value is how much it grew since its last record, and a counter that did not grow gives no record. A gauge gives its current value every bucket. The first bucket comes at once and holds everything counted since the worker started. A bucket larger than the source's batch limits is cut, and the counts cut from it are lost.

Every record carries thread, the worker thread that counted it: reader-N for a source, transform-N for a pipeline thread, egress-N for a destination, deploy for the deployment as a whole, and capture for a sample capture. The source, destination, route, pipeline and function labels hold the names they have in the deployment.

Metric	Kind	Labels	Measures
Sources
`smeltr_rows_polled_total`	counter	`source`	Records the source returned.
`smeltr_bytes_polled_total`	counter	`source`	Bytes of those records.
`smeltr_tick_poll_seconds_total`	counter	`source`	Seconds spent waiting on the source for records.
`smeltr_tick_idle_polls_total`	counter	`source`	Polls that returned no records.
`smeltr_commits_total`	counter	`source`	Offset commits the source accepted.
`smeltr_commits_dropped_total`	counter	`source`	Offset commits that failed and were let go. The next commit covers their records.
`smeltr_source_in_flight`	gauge	`source`	Batches read and not yet committed.
`smeltr_source_lag`	gauge	`source`,`topic`,`partition`	What the source has yet to finish. Kafka: records in the partition past the group's committed offset. SQS and Azure Storage Queue: messages on the queue, held ones included, as partition 0 of a topic named for the queue.
`smeltr_source_messages_finished_total`	counter	`source`	Queue messages the source finished with: every object they named read and committed. A queue's lag drains at this rate.
`smeltr_source_blocked`	gauge	`source`	1 while the source is stopped on a record it may not pass: an exception or refusal answered `block`, or nothing could process or deliver the record. 0 once it reads again.
`smeltr_source_dropped_total`	counter	`source`	Records the source let go because it had no room for them. A syslog receiver over UDP counts each datagram that arrives while it is full or blocked.
`smeltr_source_full_total`	counter	`source`	Times a receiver had no memory left for what a sender offered: a request it answered busy, or a connection it stopped reading. Its memory, not its sender, set the pace.
`smeltr_source_poll_failures_total`	counter	`source`	Reads of the source that failed. The source is read again after a wait, and what it has not delivered stays uncommitted there.
`smeltr_source_refused_total`	counter	`source`,`reason`	Requests a receiver refused, by reason: token, unreadable, too_large, unsupported, late, busy or other.
Processing
`smeltr_tick_batches_total`	counter	`source`	Batches run through routes and pipelines.
`smeltr_tick_process_seconds_total`	counter	`source`	Seconds spent running those batches.
`smeltr_tick_flush_seconds_total`	counter	`source`	Seconds spent handing processed batches to destinations.
`smeltr_batches_waiting_for_room_total`	counter	`source`	Hand-offs that waited because a destination's queue was full.
`smeltr_batches_poisoned_total`	counter	`source`	Batches a pipeline could not finish because it crashed on them.
`smeltr_batches_dlq_total`	counter	`source`,`destination`	Batches no pipeline could process, sent whole to a dead-letter destination.
`smeltr_route_matched_total`	counter	`source`,`route`	Records the route's condition matched.
`smeltr_rows_unmatched_total`	counter	`source`,`cause`	Records that met Unmatched. `cause` is `route` (no route matched) or `rule` (no delivery rule of its route took it).
`smeltr_rows_dropped_total`	counter	`source`,`reason`	Records let go. `reason` is `unmatched` (Unmatched answered drop), `exception` (an exception answered drop), `unprocessable` (no pipeline could process it) or `rejected` (a destination refused it for good).
`smeltr_rows_dlq_over_budget_total`	counter	`source`	Records sent to a dead-letter destination because one alone does not fit the memory budget.
Pipeline steps
`smeltr_function_calls_total`	counter	`pipeline`,`function`	Instructions the step ran. A step can compile to several, and one inside a loop counts on every pass.
`smeltr_function_seconds_total`	counter	`pipeline`,`function`	Seconds the step ran. A step inside a loop is not timed: its time goes to the loop step.
`smeltr_rows_dropped_total`	counter	`pipeline`,`function`	Records the step removed.
`smeltr_rows_errored_total`	counter	`pipeline`,`function`,`kind`	Records the step gave an exception, by its kind.
`smeltr_rows_in_total`	counter	`pipeline`,`function`	Records in each batch that reached a loop step, whether its condition holds for them or not.
`smeltr_rows_out_total`	counter	`pipeline`,`function`	Records the loop step's condition let in.
Destinations
`smeltr_rows_sent_total`	counter	`destination`	Records handed to the destination.
`smeltr_rows_without_doc_id_total`	counter	`destination`	Records sent to a `dedupe` or `overwrite` destination with no document ID.
`smeltr_rows_send_failed_total`	counter	`destination`	Records in a hand-over that failed. They are sent again.
`smeltr_flushes_total`	counter	`destination`	Hand-overs the destination finished.
`smeltr_rows_delivered_total`	counter	`destination`	Records in batches the destination acknowledged, including records it refused within them.
`smeltr_bytes_delivered_total`	counter	`destination`	Bytes of those records.
`smeltr_rows_rejected_total`	counter	`source`,`destination`	Records the destination refused for good.
`smeltr_rows_salvaged_total`	counter	`destination`	Refused records a dead-letter destination took.
`smeltr_rows_lost_total`	counter	`destination`	Refused records with nowhere left to go.
`smeltr_destination_healthy`	gauge	`destination`	1 while the destination takes writes, 0 after a failed send.
`smeltr_destination_stalls_total`	counter	`destination`	Failed sends that marked the destination unhealthy.
`smeltr_destination_in_flight`	gauge	`destination`	Requests the destination may have open at once.
`smeltr_destination_held_rows`	gauge	`destination`	Rows the destination refused and keeps to send again.
`smeltr_destination_open_bytes`	gauge	`destination`	Bytes in a destination pending an upload.
`smeltr_destination_queue_bytes`	gauge	`destination`	Bytes waiting in the destination's queue.
`smeltr_destination_queue_budget_bytes`	gauge	`destination`	The most bytes the destination's queue holds.
`smeltr_destination_unwritten_total`	counter	`destination`,`kind`	What the store took but said it did not store as sent, in its own units: a Prometheus remote_write 2.0 store's samples, histograms and exemplars, and documents an Elasticsearch failure store took.
`smeltr_destination_throttled_seconds_total`	counter	`destination`	Seconds the destination was held back because it was asked to slow down: by its brokers for Kafka, and in the backoff after a throttled answer for an HTTP destination (Elasticsearch, Splunk HEC, HTTP, Prometheus). Summed over concurrent requests.
`smeltr_destination_buffered_bytes`	gauge	`destination`	Bytes waiting inside the Kafka client. Kafka only.
`smeltr_destination_rtt_seconds`	gauge	`destination`	Mean round trip to the brokers in seconds. Kafka only.
Memory
`smeltr_memory_budget_bytes`	gauge		The worker's memory budget. 0 under `thread=deploy` means none.
`smeltr_memory_peak_bytes`	gauge		The most memory the worker's threads have held at once. Present only with a memory budget.
`smeltr_memory_splits_total`	counter	`source`	Batches split in half to fit the memory budget.
`smeltr_memory_abandoned_seconds_total`	counter	`source`	Seconds spent on attempts given up for memory.
`smeltr_destination_pool_bytes`	gauge		Bytes the destination queues sized to the worker's memory have borrowed above their reserves. At smeltr_destination_pool_budget_bytes, memory is what holds them back.
`smeltr_destination_pool_budget_bytes`	gauge		The most the destination queues sized to the worker's memory may borrow above their reserves, together.
`smeltr_receiver_pool_bytes`	gauge		Bytes the receivers sized to the worker's memory have borrowed from the pool they share. At smeltr_receiver_pool_budget_bytes, memory is what holds them back.
`smeltr_memory_allocated_bytes`	gauge		Bytes the worker's runtime has allocated and not freed. What its resident memory holds above this is what the allocator keeps of what was freed, and what C libraries allocate themselves.
`smeltr_receiver_pool_budget_bytes`	gauge		The most the receivers sized to the worker's memory may borrow above their reserves, together.
`smeltr_memory_part_bytes`	gauge	`part`,`source`,`destination`	The most one part of the deployment may hold, one sample per part: what `part` names, under the source or destination it belongs to when it belongs to one. Together they are what the worker's memory must fit.
`smeltr_source_held_bytes`	gauge	`source`	Bytes a source holds that its reader has not taken.
`smeltr_source_lane_bytes`	gauge	`source`	Bytes waiting between a source's reader and transform.
`smeltr_process_memory_bytes`	gauge	`process`	Bytes of memory a worker process holds: the supervisor's or the runtime's.
Worker
`smeltr_worker_threads_budget`	gauge		The most threads the deployment may use.
`smeltr_worker_slice_bytes`	gauge		The raw bytes of a batch a transform thread runs at once, sized to the CPU's cache.
`smeltr_worker_readers`	gauge		Reader threads, one per source.
`smeltr_worker_transform_threads`	gauge		Threads running pipelines.
`smeltr_worker_egress_threads`	gauge		Threads sending to destinations.
`smeltr_capture_records_total`	counter	`source`	Records a sample capture read. Its `source` label holds the source's name.
Logs
`smeltr_logs_written_total`	counter	`process`,`site`	Log events written since the process started, by the `site` that wrote them.
`smeltr_logs_dropped_total`	counter	`process`	Log events dropped for want of room since the process started.
`smeltr_logs_unread_total`	counter	`process`	Log events a full logs reader's queue refused since the process started, one count per reader.
`smeltr_logs_undelivered_total`	counter	`process`	Log events refused while the leader was out of contact, since the process started.
`smeltr_logs_unwritable_total`	counter	`process`	Log events the console would not take since the process started.

Verify