Size a worker
A worker's memory is a fixed columnar budget plus a ceiling for each source and destination it runs. Its CPU is one transform thread per core. Its disk is the state directory.
Memory
Give a worker at least the sum of the parts below for the sources and destinations its group
runs, plus about 150 MiB for the process itself and the size of every lookup table the group
uses and its index, which a worker holds in memory whole. Each amount below is the default; a
source or destination with a larger fetchBytes or bufferBytes holds that
instead. Every worker in a group runs the whole group's configuration, so size each one for all of
it. The warning below counts the parts and not the lookup tables.
| Part | Default | For each | Why |
|---|---|---|---|
| Columnar budget | 1 GiB | worker running any source | Memory every transform thread shares while it runs a batch. SMELTR_MEMORY_BUDGET_BYTES changes it. A batch that needs more is split and run in parts, which costs time. |
| Lanes | 4 MiB | source, and again per transform thread | One batch per slot: about 20 MiB for one source on four threads, about 80 MiB for ten sources on sixteen. |
| Fetch queue | 64 MiB | source, plus half of it per broker in the cluster | fetchBytes. A fetch already asked of each broker still lands once the queue is full. The warning counts only the brokers named in brokers. |
| Request bodies | 128 MiB | source | Twice the receiver's inbox: bodies being read, and bodies admitted. |
| Record | 64 MiB | source | The largest record it reads from an object. |
| Answer | 256 MiB | source | The largest answer it takes. A whole answer is held while it is cut into records. |
| Result | 256 MiB | source | The largest result one poll takes. A whole result is held while it is handed out; a larger one fails the poll. |
| Buffer | 64 MiB | destination | bufferBytes. Elasticsearch, Splunk HEC, HTTP, Log Analytics workspace and Prometheus take one request's size instead when that is larger. A Kafka destination's covers its client's own queue too. |
| Buffer and open files | 512 MiB | destination | bufferBytes for rows not yet in the bucket, and as much again for the open files encoding and uploading them. |
A group on four cores reading one Kafka topic from three brokers into Elasticsearch and S3 adds up to 1780 MiB:
| Columnar budget | 1024 MiB |
| Lanes, 1 source and 4 threads | 20 MiB |
| Fetch queue, a cluster of 3 brokers | 160 MiB |
| Elasticsearch buffer | 64 MiB |
| S3 buffer and open files | 512 MiB |
| Total | 1780 MiB |
|---|
Each ceiling is a worst case, and they rarely all fill at once. A worker whose deployment adds
up to more than its memory limit still runs, but can be killed for memory under load. It logs a
warning naming each part and its share, and the warning stays on Issues while it holds. Select
the worker on Status and open Memory by part to see which part it is. Raise the
limit, or lower fetchBytes, bufferBytes or SMELTR_MEMORY_BUDGET_BYTES.
The limit a worker checks against is its container's memory limit, or the host's memory when it has none. Set a memory limit on every worker container, so the warning compares against what the worker may really use.
smeltr_memory_peak_bytes shows how much of the columnar budget batches have needed. A
peak well under the budget means the budget can come down.
CPU
Transform does most of the CPU work, and it runs one thread per core the worker may use. Readers and destinations run threads of their own that mostly wait on the network, except a receiver decompressing gzip and an object destination compressing its files. Two cores is a sensible start; add cores or workers while CPU stays high under normal load.
The cores a worker counts are its CPU limit, or the host's cores when it has none. A container
with a CPU request and no limit counts every core on its node. Set worker.runtime.thread_limit to its request, or give it a CPU limit.
Disk
The state directory holds the worker's credential, its clone of the configuration repository, the lookup tables that configuration names and the progress of the single-worker sources it holds. A new version of a lookup table, and the index built beside it, arrive before the old ones are removed, so leave room for each table and its index twice. Once this volume is full, the worker takes no new configuration, lookup table or source handoff.