snowflake::ingest::streaming::StreamingIngestElasticChannel Class Referencefinal
Elastic channel for Snowflake Streaming Ingest. More...
#include <snowflake/ingest/streaming/elastic_channel.h>
Public Member Functions | |
| StreamingIngestElasticChannel (const StreamingIngestElasticChannel &)=delete | |
| StreamingIngestElasticChannel & | operator= (const StreamingIngestElasticChannel &)=delete |
| StreamingIngestElasticChannel (StreamingIngestElasticChannel &&)=delete | |
| StreamingIngestElasticChannel & | operator= (StreamingIngestElasticChannel &&)=delete |
| Result< | AppendRow (std::string_view serialized_row,std::optional<AppendToken> token=std::nullopt) |
| Insert one row into the elastic channel without waiting for acknowledgement (fire-and-forget): returns as soon as the row is buffered. | |
| Result< | AppendRows (std::string_view ndjson,int64_t num_rows,std::optional<AppendToken> token=std::nullopt) |
| Insert a batch of rows into the elastic channel without waiting for acknowledgement (fire-and-forget): returns as soon as the rows are buffered. | |
| Result< | AppendRowWithWait (std::string_view serialized_row,std::optional<AppendToken> token=std::nullopt) |
| Insert one row into the elastic channel and return a future that completes when Snowflake acknowledges the row (or completes with an error if the append fails). | |
| Result< | AppendRowsWithWait (std::string_view ndjson,int64_t num_rows,std::optional<AppendToken> token=std::nullopt) |
| Insert a batch of rows into the elastic channel and return a future that completes when Snowflake acknowledges the batch (or completes with an error if the append fails). | |
| Result< | SetErrorHandler (ErrorHandler handler) |
Register a handler for appends that fail asynchronously, after AppendRow()/AppendRows() has returned. | |
| Result< | SetSuccessHandler (SuccessHandler handler) |
Register a handler for appends that are acknowledged by Snowflake, after AppendRow()/AppendRows() has returned. | |
| Result< | InitiateFlush () |
| Initiates a flush of all buffered data in this channel without waiting for completion. | |
| Result< | WaitForFlush (std::chrono::milliseconds timeout=std::chrono::milliseconds(0)) |
| Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side. | |
| Result< | GetChannelStatus () const |
| Get the channel status from Snowflake. | |
| bool | IsClosed () const |
| Check if the elastic channel is closed (because the client was closed). | |
| const std:: | channel_name () const noexcept |
| Get the channel name (always "ELASTIC"). | |
| const std:: | db_name () const noexcept |
| Get the database name. | |
| const std:: | schema_name () const noexcept |
| Get the schema name. | |
| const std:: | pipe_name () const noexcept |
| Get the pipe name. | |
Friends | |
| struct | internal::ElasticChannelAccess |
Detailed Description
Elastic channel for Snowflake Streaming Ingest.
Unlike regular channels, elastic channels have no offset token concepts and their lifecycle is tied to the client (no close method). The same instance is returned on repeated calls to StreamingIngestClient::GetElasticChannel(). Not copyable or movable.
- Warning
- Handlers must not capture the channel's
shared_ptr.
Member Function Documentation
AppendRow¶
StreamingIngestElasticChannel::AppendRow(
std::string_view serialized_row,
std::optional<AppendToken> token = std::nullopt ))
Insert one row into the elastic channel without waiting for acknowledgement (fire-and-forget): returns as soon as the row is buffered.
The append is still tracked for the error handler and the success handler — the outcome is surfaced through whichever is registered, keyed by token, just not through a future. Use AppendRowWithWait() when you need a future to await.
- Parameters
-
serialized_row data to write: a caller-serialized JSON-formatted string that maps Column Name to Column Value for the columns in the table, e.g. R"({"c1": 1, "c2": "abc"})". The string should:- hold exactly one row.
- encode BINARY values following the pipe's
BINARY_INPUT_FORMAT. - not be modified concurrently while this call is in progress.
token caller-supplied opaque append token, delivered in the ErrorDetailto a registered error handler if this append fails asynchronously, or in theSuccessDetailto a registered success handler once it is acknowledged. Iftokenisstd::nullopt, neither handler is invoked — and, with no future either (fire-and-forget), the outcome is not reported at all. The token is retained in memory until the append is acknowledged by the service, so attaching a large object raises the client's memory footprint — prefer a small id.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
AppendRows¶
StreamingIngestElasticChannel::AppendRows(
std::string_view ndjson,
int64_t num_rows,
std::optional<AppendToken> token = std::nullopt ))
Insert a batch of rows into the elastic channel without waiting for acknowledgement (fire-and-forget): returns as soon as the rows are buffered.
The append is still tracked for the error handler and the success handler — the outcome is surfaced through whichever is registered, keyed by token, just not through a future. Use AppendRowsWithWait() when you need a future to await.
- Parameters
-
ndjson data to write: a single string holding one or more rows of JSON, with rows separated by a newline character ( \n). Each row is a caller-serialized JSON object that maps Column Name to Column Value for the columns in the table. For example, for two rows:"{\"c1\":1,\"c2\":\"abc\"}\\n{\"c1\":2,\"c2\":\"def\"}". The string should:- hold exactly
num_rowsrows - encode BINARY values following the pipe's
BINARY_INPUT_FORMAT. - not be modified concurrently while this call is in progress.
num_rows number of rows in ndjson.token caller-supplied opaque append token, delivered in the ErrorDetailto a registered error handler if this append fails asynchronously, or in theSuccessDetailto a registered success handler once it is acknowledged. Iftokenisstd::nullopt, neither handler is invoked — and, with no future either (fire-and-forget), the outcome is not reported at all. The token is retained in memory until the append is acknowledged by the service, so attaching a large object raises the client's memory footprint — prefer a small id. - hold exactly
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
AppendRowsWithWait¶
StreamingIngestElasticChannel::AppendRowsWithWait(
std::string_view ndjson,
int64_t num_rows,
std::optional<AppendToken> token = std::nullopt ))
Insert a batch of rows into the elastic channel and return a future that completes when Snowflake acknowledges the batch (or completes with an error if the append fails).
A registered error handler also fires on failure, and a registered success handler on acknowledgement — dual delivery either way.
- Parameters
-
ndjson data to write; see AppendRows().num_rows number of rows in ndjson.token caller-supplied opaque append token, delivered in the ErrorDetailto a registered error handler if this append fails asynchronously, or in theSuccessDetailto a registered success handler once it is acknowledged. Iftokenisstd::nullopt, neither handler is invoked; the returned future is then the only signal. The token is retained in memory until the append is acknowledged by the service, so attaching a large object raises the client's memory footprint — prefer a small id.
- Returns
- a
std::futurethat completes with an emptyResultwhen the rows are acknowledged by Snowflake, or with the error if the append fails. Returns aStreamingIngestErrordirectly in case of error.
AppendRowWithWait¶
StreamingIngestElasticChannel::AppendRowWithWait(
std::string_view serialized_row,
std::optional<AppendToken> token = std::nullopt ))
Insert one row into the elastic channel and return a future that completes when Snowflake acknowledges the row (or completes with an error if the append fails).
A registered error handler also fires on failure, and a registered success handler on acknowledgement — dual delivery either way.
- Parameters
-
serialized_row data to write; see AppendRow().token caller-supplied opaque append token, delivered in the ErrorDetailto a registered error handler if this append fails asynchronously, or in theSuccessDetailto a registered success handler once it is acknowledged. Iftokenisstd::nullopt, neither handler is invoked; the returned future is then the only signal. The token is retained in memory until the append is acknowledged by the service, so attaching a large object raises the client's memory footprint — prefer a small id.
- Returns
- a
std::futurethat completes with an emptyResultwhen the row is acknowledged by Snowflake, or with the error if the append fails. Returns aStreamingIngestErrordirectly in case of error.
channel_name¶
|
const std::string & StreamingIngestElasticChannel::channel_name() const
|
inlinenoexcept |
Get the channel name (always "ELASTIC").
- Returns
- the channel name
db_name¶
|
const std::string & StreamingIngestElasticChannel::db_name() const
|
inlinenoexcept |
Get the database name.
- Returns
- name of the database
GetChannelStatus¶
Get the channel status from Snowflake.
- Returns
- the channel status on success, or a
StreamingIngestErroron failure.
InitiateFlush¶
Initiates a flush of all buffered data in this channel without waiting for completion.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
IsClosed¶
Check if the elastic channel is closed (because the client was closed).
- Returns
- true if the channel is closed
pipe_name¶
|
const std::string & StreamingIngestElasticChannel::pipe_name() const
|
inlinenoexcept |
Get the pipe name.
- Returns
- name of the pipe
schema_name¶
|
const std::string & StreamingIngestElasticChannel::schema_name() const
|
inlinenoexcept |
Get the schema name.
- Returns
- name of the schema
SetErrorHandler¶
Register a handler for appends that fail asynchronously, after AppendRow()/AppendRows() has returned.
The handler is invoked once per failure event with an ErrorDetail carrying the append tokens that failed together under a common error. Optional, and replaceable (last one set wins) — set it before appending for full coverage. Failed futures still complete with the error either way.
The handler is only invoked for appends made with a token; an append given std::nullopt still fails its future but is never reported to the handler.
The handler runs inline on this channel's internal acknowledgement task. It must therefore return quickly and do no heavy or thread-blocking work: no blocking I/O, no network or disk calls, no waiting on locks or futures, and no synchronous re-entry into the SDK. Do only cheap bookkeeping inline (for example, record the failed tokens) and hand any real work to your own executor or queue. Blocking delays this channel's own acknowledgements — and therefore its append and flush completion — and, because every channel's acknowledgement task shares one process-wide worker pool, enough blocked handlers can starve acknowledgements elsewhere too. The SDK logs a warning when a handler runs slowly. A thrown exception is caught and logged so one bad handler cannot poison the acknowledgement path.
- Parameters
-
handler the error handler to register. Must not be empty.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
SetSuccessHandler¶
Register a handler for appends that are acknowledged by Snowflake, after AppendRow()/AppendRows() has returned.
The handler is invoked once per successful acknowledgement batch with a SuccessDetail carrying the append tokens that were acknowledged together. Optional, and replaceable (last one set wins) — set it before appending for full coverage. Independent of the error handler: either can be registered without the other, and a given append is reported to exactly one of them. Successful futures still complete either way.
The handler runs inline on this channel's internal acknowledgement task. It must therefore return quickly and do no heavy or thread-blocking work: no blocking I/O, no network or disk calls, no waiting on locks or futures, and no synchronous re-entry into the SDK. Do only cheap bookkeeping inline (for example, record the tokens) and hand any real work to your own executor or queue. Blocking delays this channel's own acknowledgements — and therefore its append and flush completion — and, because every channel's acknowledgement task shares one process-wide worker pool, enough blocked handlers can starve acknowledgements elsewhere too. The SDK logs a warning when a handler runs slowly. A thrown exception is caught and logged so one bad handler cannot poison the acknowledgement path.
- Parameters
-
handler the success handler to register. Must not be empty.
- Returns
- an empty
Resulton success, or aStreamingIngestErroron failure.
WaitForFlush¶
Asynchronously waits for all buffered data in this channel to be flushed to the Snowflake server side.
This method triggers a flush of all pending data and waits for the flush operation to complete.
- Parameters
-
timeout The maximum time (non-negative) to wait for the flush to complete. If 0, the operation will wait indefinitely.
- Returns
- a
std::futurethat completes with an emptyResultwhen the flush is done, or withkChannelWaitForFlushTimeoutif the timeout is reached. Returns aStreamingIngestErrordirectly in case of error.