Logs & metrics

Every Smeltr process keeps two accounts of itself: logs, which are timed, structured events, and metrics, which are counters and gauges. A worker's heartbeat carries both to the leader, and a bundle can route either through a pipeline to any destination.

Where logs and metrics go

Goes to	Logs	Metrics
The console	Every process writes one line per event to stderr.	—
The leader	Each worker sends its logs with its heartbeat. The leader keeps the newest 100,000 rows for the fleet.	Each worker sends its metrics with its heartbeat. The leader keeps them for 30 days, in coarser buckets as they age.
A destination	Through a Smeltr logs source.	Through a Smeltr metrics source.

The leader keeps its own logs on its own console. Where one process runs both the leader and a worker, the leader's logs reach the leader with that worker's heartbeat.

The leader's copy is its operational view. Send logs and metrics you need to keep or search elsewhere to a destination.

Read them in the GUI

Status → Logs shows the fleet's logs, newest first. Choose severities, a worker, a group, a site, or text in the message, then choose Search. A severity is not a minimum: choose error and warn to see both. Select a row's fields to open them. The filters are in the page's address, so you can share a view as a link.
Status shows Rows in and out, Errors and Furthest from normal, from the metrics the heartbeat carries. Choose the window of Rows in and out: 15m, 60m, 24h, 7d or 30d. A gap in a line is missing history, not zero.

The leader cuts a log's site, message and each text field to 1,000 characters, and keeps its first 16 fields. A search cannot find what the leader cut. A worker out of contact with the leader holds the oldest 4,096 of its own events and the newest 2,048 from the process running its pipelines.

Send them to a destination

Select the worker group and work on a branch.
Open Build → Sources, choose + Add source, choose Local, each worker, and then smeltr logs or smeltr metrics. Its settings are on the Smeltr logs and Smeltr metrics pages.
Connect it to a route, which sends its records through a pipeline to a destination. Follow the routing example.
Review and deploy.

Each source gives the logs or metrics of the worker it runs on, covering both of that worker's processes: the supervisor, which talks to the leader, and the runtime, which runs its pipelines. Add one to every worker group you want to watch. Its records carry __source.topic __smeltr_logs or __smeltr_metrics.

What metrics exist

Smeltr has 74 metrics, grouped by what they measure: sources, processing, pipeline steps, destinations, memory, the worker and its logs. Each is a counter or a gauge with labels naming its subject. The Smeltr metrics source lists every one, with its labels and what it measures, and shows the record each one makes.

Check for lost logs

Writing a log never waits. A process with no room for an event drops it and counts it. These metrics carry a process label naming the process that wrote them.

Metric	Measures
`smeltr_logs_written_total`	Log events written since the process started, by the `site` that wrote them.
`smeltr_logs_dropped_total`	Log events dropped for want of room since the process started.
`smeltr_logs_unread_total`	Log events a full logs reader's queue refused since the process started, one count per reader.
`smeltr_logs_undelivered_total`	Log events refused while the leader was out of contact, since the process started.
`smeltr_logs_unwritable_total`	Log events the console would not take since the process started.

Change log verbosity

A process collects logs at or above its log level, info unless you set it. The console, the leader and every Smeltr logs source get only what the process collects. To change it, see Change log verbosity.