Google Cloud Storage

Lists a prefix of one Cloud Storage bucket every poll interval, reads each new or changed object it finds, and splits it into records. Only one worker of each group reads this source at a time.

Use it for a bucket that cannot send Pub/Sub notifications, or one you may not configure. When the bucket can notify, use Google Cloud Storage via Pub/Sub: every worker of the group reads, and nothing lists the bucket.

Example

Reads each object under app/ one line per record, signed in with the worker host's Application Default Credentials, and moves each object under done/ once Smeltr delivers every record.

{
  "id": "bucket-export",
  "type": "gcsList",
  "bucket": "acme-logs",
  "prefix": "app/",
  "framing": "lines",
  "afterCommit": "move",
  "movePrefix": "done/"
}

The GCS listing form on the Sources page: id bucket-export, bucket acme-logs, prefix app/, after commit move to done/, framing lines. — The same source in the Sources page's add dialog, as it is added.

Before you start

Choose what happens to an object once Smeltr delivers it, afterCommit. keep leaves the bucket as it is. delete and move drain it, so each listing finds only what is still to read.
Give the service account it reads as a role on the bucket. keep needs to list and read objects: Storage Object Viewer. delete also needs to delete them, and move to create them as well, since a move copies the object and then deletes it: Storage Object User covers both. See Cloud Storage IAM roles.
Choose how it signs in. With no key, it uses the worker host's Application Default Credentials. Otherwise put a service account key, the whole JSON file, in a credential.

Settings

It also takes the settings every source takes.

bucket string required

The bucket's name, alone.

Examples

acme-logsacme-audit-logs-prodlogs.acme.example

prefix string default

Only objects whose name starts with it. It does not start with /. Empty is the whole bucket.

Examples

app/cloudaudit.googleapis.com/activity/logs/2026/

serviceAccountKey credential default the host's Application Default Credentials

The credential holding a service account's JSON key. The account needs to list and read the bucket, and to delete or create objects in it for delete and move.

Examples

GCS_KEY{"secret": "gcs-key","version": 1}

format string default each payload is text, in _raw

How each payload becomes the record.

avro: Reads Avro files. Fields you don't declare are kept.

fields list of fields default none

The Avro fields to read, each with a name and a type such as i64 or str. Name a nested field with dots, such as user.id. Top-level fields you don't list are kept. Only with format avro.

Example

[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]

framing string default lines

How an object becomes records.

lines: Gives one record per line, or per piece lineBreaker cuts.
object: Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.
escapedLines: Gives one record per line, unescaped as a bytes object destination wrote it.

lineBreaker string default a record per line

With lines framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: ([\r\n]+)\d{4}- keeps a stack trace with the line that logged it, and </event>(\s*) cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples

([\r\n]+)\d{4}-</event>(\s*)

identity identity default none

The connection identity it connects under, for a private CA or a client certificate.

Examples

gcp-proxycorp-proxy

afterCommit string default keep

What it does with an object once every record is delivered.

keep: Leaves it and records it as read. Needs only read access, and costs a full listing per poll.
delete: Deletes it.
move: Copies it under movePrefix in place of prefix, then deletes it.

movePrefix string default none

Where move puts an object, in place of prefix. Only with move, and outside prefix. It does not start with /, and ends in / when prefix does or is empty, and only then.

Examples

done/archive/2026/processed/app/

Advanced

start string default everything

What it reads first.

everything: Reads everything its first listing finds.
new: Reads only what arrives after its first listing. Needs afterCommit keep.

pollIntervalSecs integer default 60

Seconds from the start of one listing to the start of the next. At least 1.

Examples

300303600

endpoint string default the key's own cloud

Cloud Storage's address, such as an emulator's.

Examples

http://localhost:4443https://storage-acme.p.googleapis.com

encoding string default auto

How a record's bytes become _raw's text: auto, none, or a WHATWG encoding label other than the replacement ones, such as utf-8, utf-16le or shift_jis.

auto: Detects each record's encoding.
utf-8: Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.
none: Decodes nothing: the record has no _raw, and its payload is __source.bytes alone. A pipeline reads it as text with decode(__source.bytes, encoding).

Fields it gives

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.bucket`	`str`	The bucket, or Azure container, the record's object is in.
`__source.key`	`str`	The key, or Azure blob name, of the object the record was read from.
`__source.offset`	`i64`	The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0.
`__source.bytes`	`bytes`	The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text.

How it works

Every pollIntervalSecs, counted from the start of the last listing, the source lists the objects whose names start with prefix. It reads each object the listing finds new, or changed since it last read it, and reads a changed object again whole. An empty object holds no record, and the source does not read it. The source decompresses a gzip, zstd, framed snappy or framed lz4 object first and reads any other as it is.

One worker of the group holds the source and reads it; every other worker running it reads nothing. The leader chooses the holder from the workers' heartbeats. The leader gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer, and that worker resumes from the last progress the leader kept. That worker delivers again what the previous holder delivered after that progress.

With afterCommit keep, the progress is a ledger: each object read, by name and version. The holder records an object in it once Smeltr delivers every record of the object and reports it to the leader on its heartbeats, so the next holder reads only what is not in it. An object that leaves the bucket leaves the ledger. Every listing walks the whole prefix, so a large prefix costs a full listing each poll.

With delete or move, the bucket itself says what is still to read and there is no ledger. Once Smeltr delivers every record of an object, the source deletes it, or copies it to movePrefix in place of prefix and deletes the original. An object overwritten since the source read it stays in place, and the source reads it again. It deletes or moves an empty object unread.

start everything reads every object the first listing finds. new records the first listing as read and reads only what arrives or changes after it, so it needs keep.

The source's progress belongs to its name and to its bucket and prefix. Renamed, or pointed at another bucket or prefix, it starts afresh.

Once you remove the source and deploy, the leader forgets its progress, so a rollback that brings the source back starts it afresh.

An object handed out five times and never delivered is no longer offered until it changes. With keep it is recorded in the ledger as read; with delete or move it stays in the bucket. The worker log names it. With delete or move only the holder remembers it, so a restart or a new holder hands it out again.

Verify

Upload an object with a line you can recognize under the prefix, then find that line at the destination after the next listing. On the Sources page, the source's worker column names the worker holding it, and with keep how many objects its ledger holds. With delete or move, the object leaves the prefix once it is delivered.

Troubleshooting

Nothing arrives: Check that the object's name starts with prefix, and that the poll interval has passed. With start new, the source does not read what was there at the first listing until it changes. The worker log of the holder says why a listing failed, such as a missing role.
The worker column is empty: No worker holds the source: check that a worker of its group runs the deployment that has it.
The holder was last heard some time ago: The leader has not heard from that worker. It gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer.
An object stays under the prefix: The source could not read it or could not delete it: check the role on the bucket and the worker log for the reason.

Google Cloud Storage

Example

Before you start

Settings

Advanced

Fields it gives

How it works

Verify

Troubleshooting

See also