Pipeline steps

A pipeline is a list of steps. Each step reads and changes the records that reach it, in order. Each kind of step has a page of its own.

Kinds

Kind	type	What it does
Call	`call`	Runs another pipeline's steps in place.
Code	`code`	Runs a program on each record.
Comment	`comment`	Holds a note. Records pass it unchanged.
Drop	`drop`	Removes each record where a condition is true.
Eval	`eval`	Sets fields to the values of expressions.
If	`if`	Sends each record through one of two lists of steps.
Lookup	`lookup`	Adds fields from the rows of a lookup table that match a value.
Parser	`parser`	Reads a text field as JSON, CSV, Key/value or Regex/Grok into fields.
Rename	`rename`	Moves fields to new names.
Unroll	`unroll`	Turns each item of an array field into its own record.

How it works

Each record goes through the steps from first to last. A step sees what the steps before it wrote. A record a Drop step removes goes no further, and an Unroll step turns one record into several.

A step's filter selects the records it processes: where the filter is false or null, the record passes the step unchanged. An If step selects records for a whole list of steps, and gives the other records a list of their own. The pipeline skips a disabled step.

A value a step cannot read or compute throws an exception on its record, such as a parse or math exception. The route's or the group's exception answers decide what happens to the record.

Settings

A step is one entry of its pipeline's steps. Every step takes an id:

id string required

The step's name, unique in its pipeline, counting the steps a Call inlines. Errors and exceptions name it.

Examples

parse-jsondrop-health-checks

Every step but If and Comment also takes these:

filter expression default every record

A condition. The step runs only on records where it is true; every other record passes unchanged. Unroll refuses one. The GUI calls it filter.

Examples

status >= 400path.startsWith("/api/")source.ip.inCidr("10.0.0.0/8") && action == "deny"

disabled bool default false

Skips the step, so every record passes it unchanged. The GUI's pause button sets it.

Add a step

Select the worker group and work on a branch.
Open the pipeline, click + step, and choose the step type.
Fill in its settings, choose a sample, and click Run now to preview the output.
Save the pipeline, then review and deploy.

Verify

Once the preview is right, choose Lock this as a test. After each save, Resources → Samples shows whether every test still gives its locked output. See Save the result as a test.

Troubleshooting

A step changes nothing: Check that it is not disabled and that its filter is true on the sample's records.
Saving refuses the pipeline: The message names the step by its id and what it refuses, such as an If condition that is not a bool.