Azure Blob Storage

Lists a prefix of one container every poll interval, reads each blob it finds new or changed, and splits it into records. Only one worker of each group reads this source at a time.

Use it where the storage account cannot notify. Where it can, Azure Blob via Storage queue reads each blob as the storage account announces it, with no listing, and any number of workers share the work.

Example

Reads each blob under app/ one line per record, as the worker host's managed identity, and moves it under done/ once Smeltr delivers it.

{
  "id": "container-export",
  "type": "azureBlobList",
  "account": "acmelogs",
  "container": "logs",
  "prefix": "app/",
  "framing": "lines",
  "afterCommit": "move",
  "movePrefix": "done/"
}

The Azure Blob listing form on the Sources page: id container-export, account acmelogs, container logs, prefix app/, after commit move to done/, framing lines. — The same source in the Sources page's add dialog, as it is added.

Before you start

Choose what it does with a blob once Smeltr delivers it, afterCommit. That sets the permission it needs: keep only lists and reads; delete also deletes; move also writes the blob under movePrefix, then deletes it.
Choose how it signs in, and give that identity that permission on the container. A key, signature or client secret goes in a credential.
- the worker host's managed identity, or a service principal, given Storage Blob Data Reader on the container for keep, or Storage Blob Data Contributor for delete and move (see assign a role for data access);
- an account key; or
- a shared access signature for the blob service, allowed to list and read, and also to delete for delete, or to create, write and delete for move.

Settings

It also takes the settings every source takes.

container string required

The container's name, alone.

Examples

logsinsights-logs-auditevent

account string required

The storage account holding the container.

Examples

acmelogsacmearchive01

prefix string default

Only blobs whose name starts with it. It does not start with /. Empty is the whole container.

Examples

app/resourceId=/SUBSCRIPTIONS/logs/2026/

accessKey credential default none

The credential holding the account's access key. Set at most one of accessKey, sasToken, clientSecret and managedIdentityClientId; with none, the host's managed or workload identity is used.

Examples

AZURE_ACCESS_KEY{"secret": "azure-access-key","version": 1}

sasToken credential default none

The credential holding a shared access signature that covers the blob service.

Examples

AZURE_SAS_TOKEN{"secret": "azure-sas-token","version": 1}

tenantId string default none

A service principal's tenant. Set tenantId, clientId and clientSecret together, or none of them.

Example

00000000-0000-0000-0000-000000000000

clientId string default none

A service principal's client id.

Example

11111111-1111-1111-1111-111111111111

clientSecret credential default none

The credential holding a service principal's client secret.

Examples

AZURE_CLIENT_SECRET{"secret": "azure-client-secret","version": 1}

managedIdentityClientId string default the host's managed or workload identity

The client id of the user-assigned managed identity to sign in as. Set it when the worker's host has more than one managed identity. Left out, it signs in as the host's managed identity, or an AKS pod's workload identity.

Example

22222222-2222-2222-2222-222222222222

format string default each payload is text, in _raw

How each payload becomes the record.

avro: Reads Avro files. Fields you don't declare are kept.

fields list of fields default none

The Avro fields to read, each with a name and a type such as i64 or str. Name a nested field with dots, such as user.id. Top-level fields you don't list are kept. Only with format avro.

Example

[{"name": "user_id","type": "i64"},{"name": "user.name","type": "str"}]

framing string default lines

How an object becomes records.

lines: Gives one record per line, or per piece lineBreaker cuts.
object: Gives the whole object as one record, for a JSON document such as CloudTrail's that a pipeline unrolls.
escapedLines: Gives one record per line, unescaped as a bytes object destination wrote it.

lineBreaker string default a record per line

With lines framing, and only then: where a record ends. Its first capture group is the break and is dropped: the bytes before it end a record, line ends and all, and the bytes after it start the next. What follows the group is kept, so the pattern can say how a record starts: ([\r\n]+)\d{4}- keeps a stack trace with the line that logged it, and </event>(\s*) cuts XML events with no line ends between them. It needs a capture group and must not match the empty string.

Examples

([\r\n]+)\d{4}-</event>(\s*)

identity identity default none

The connection identity it connects under, for a private CA or a client certificate.

Examples

azure-proxycorp-proxy

afterCommit string default keep

What it does with a blob once every record is delivered.

keep: Leaves it and records it as read. Needs only read access, and costs a full listing per poll.
delete: Deletes it.
move: Copies it under movePrefix in place of prefix, then deletes it.

movePrefix string default none

Where move puts a blob, in place of prefix. Only with move, and outside prefix. It does not start with /, and ends in / when prefix does or is empty, and only then.

Examples

done/archive/2026/processed/app/

Advanced

start string default everything

What it reads first.

everything: Reads everything its first listing finds.
new: Reads only what arrives after its first listing. Needs afterCommit keep.

pollIntervalSecs integer default 60

Seconds from the start of one listing to the start of the next. At least 1.

Examples

300303600

endpoint string default the account's, in Azure's public cloud

The blob service's address, such as Azurite's or a sovereign cloud's.

Examples

http://localhost:10000/acmelogshttps://acmelogs.blob.core.usgovcloudapi.net

authEndpoint string default its cloud's own

Where a client secret is exchanged for a token. Only with clientSecret.

Examples

https://login.microsoftonline.ushttps://login.chinacloudapi.cn

encoding string default auto

How a record's bytes become _raw's text: auto, none, or a WHATWG encoding label other than the replacement ones, such as utf-8, utf-16le or shift_jis.

auto: Detects each record's encoding.
utf-8: Decodes every record as UTF-8, as any other label decodes as its own encoding. Bytes that do not decode become U+FFFD.
none: Decodes nothing: the record has no _raw, and its payload is __source.bytes alone. A pipeline reads it as text with decode(__source.bytes, encoding).

Fields it gives

__source.bucket holds the container, and __source.key the blob's name.

Field	Type	Holds
`__source.name`	`str`	The name of the source that read the record.
`__source.type`	`str`	The kind of source that read it, its `type`, such as `kafka`, `s3List`, `httpApi` or `hecReceiver`.
`__source.bucket`	`str`	The bucket, or Azure container, the record's object is in.
`__source.key`	`str`	The key, or Azure blob name, of the object the record was read from.
`__source.offset`	`i64`	The record's offset in its Kafka or Event Hubs partition, or its record number in its object, from 0.
`__source.bytes`	`bytes`	The record's payload as the source read it, before any decoding. The source decodes `_raw` from it. A `bytes` destination writes it unless the destination names another field. Text functions refuse it; `decode(__source.bytes, encoding)` reads it as text.

How it works

Every pollIntervalSecs it lists the blobs whose names start with prefix. The source reads a blob once two listings in a row find it unchanged, since an append blob grows while something writes it. It reads the blob whole: it decompresses a gzip, zstd, framed snappy or framed lz4 blob first and reads any other as it is. An empty blob holds no record, and the source does not read it.

One worker of the group reads it at a time, the one the leader holds it for; the others read nothing. The leader gives the hold to another worker after three heartbeat intervals without hearing from the holder, the holder's interval or the leader's, whichever is longer. With keep, its progress is a ledger: the blobs it has read, each with the version it read. The holder reports it on its heartbeat and the leader keeps it; the next holder fetches the ledger and reads only what is not in it. The next holder delivers again what the previous holder delivered after the last progress the leader kept. With delete or move the source keeps no progress: the next holder reads what the container still holds, so it delivers again a blob delivered but not yet deleted or moved.

afterCommit says what it does with a blob once Smeltr delivers every record.

keep leaves the blob and records it in the ledger. When a blob changes after the source reads it, the source reads it again whole. A blob that is gone from the listing leaves the ledger. Every listing lists the whole prefix, so the cost of each poll grows with what the prefix holds.
delete deletes the blob, and move copies it under movePrefix in place of prefix and deletes it. The container is then its own record of what is left to read, and the source keeps no ledger. A blob that changed after the source read it stays in place, and the source reads it again. It deletes or moves an empty blob unread.

A blob handed out five times and never delivered is no longer read until it changes. With keep it is recorded in the ledger as read; with delete or move it stays in the container. The worker log says which blob. With delete or move only the holder remembers it, so a restart or a new holder hands it out again.

start says what the first listing reads. everything reads every blob it finds. new records every blob it finds as read and reads only what arrives or changes after it; it needs afterCommit keep, since only the ledger remembers what was skipped.

The source's progress belongs to its name and to its storage account, container and prefix. Renamed, or pointed at another container or prefix, it starts afresh.

Once you remove the source and deploy, the leader forgets its progress, so a rollback that brings the source back starts it afresh.

Verify

Upload a blob with a line you can recognize under the prefix, then find that line at the destination after two listings. On the Sources page, the source's worker column names the worker that holds it, and with keep how many blobs its ledger holds. With delete or move, the blob leaves the prefix once it is delivered.

Troubleshooting

Nothing arrives: The source reads a blob only on the second listing that finds it unchanged, so wait two pollIntervalSecs. Check that its name starts with prefix, which does not start with /, and the worker log for a listing that failed.
The worker column is empty: No worker of the group runs the source, or none has reported it to the leader. Check that the group's workers are enrolled and running the deployment.
A blob is read again: It changed after the source read it, or the source moved to another worker before its last progress reached the leader. Both deliver its records again.
Blobs stay in the container with delete or move: The identity cannot delete or write there: check its role or signature, and the worker log for the reason.

Azure Blob Storage

Example

Before you start

Settings

Advanced

Fields it gives

How it works

Verify

Troubleshooting

See also