Package com.snowflake.connectors.taskreactor.worker.ingestion

Interface SimpleIngestion<I,​M,​C,​D>

  • Type Parameters:
    I - resource id class, containing properties which identify the resource in the source system
    M - resource metadata class, containing additional properties which identify the resource in the source system. The properties can be fetched automatically or calculated by the connector
    C - custom ingestion configuration class, containing custom ingestion properties
    D - destination configuration class, containing properties describing where the ingested data should be stored
    public interface SimpleIngestion<I,​M,​C,​D>
    Interface that should be implemented by users of task reactor and passed over to IngestionWorker instance.

    • Field Detail

      • log

        static final org.slf4j.Logger log
        Logger instance for use in the default methods of this interface.

    • Method Detail

      • fetchData

        default void fetchData​(SimpleIngestionWorkItem<I,​M,​C,​D> workItem,
                       String ingestionRunId)
        Fetches data from source, can be also responsible for creating necessary objects like tables. It is optional step since data can be fetched in a persistData method.
        Parameters:
        workItem - work item provided by the dispatcher.
        ingestionRunId - ingestion run id created by the worker.

      • persistData

        long persistData​(SimpleIngestionWorkItem<I,​M,​C,​D> workItem,
                 String ingestionRunId)
        Persists data in the destination. Data can be fetched here if it must be done in one transactional step.
        Parameters:
        workItem - work item provided by the dispatcher.
        ingestionRunId - ingestion run id created by the worker.
        Returns:
        number of rows persisted to the destination.

      • onSuccessfulIngestionCallback

        default void onSuccessfulIngestionCallback​(SimpleIngestionWorkItem<I,​M,​C,​D> workItem,
                                           String ingestionRunId)
        Executes callback when successful ingestion is registered.
        Parameters:
        workItem - work item provided by the dispatcher.
        ingestionRunId - ingestion run id created by the worker.

      • onFailedIngestionCallback

        default void onFailedIngestionCallback​(SimpleIngestionWorkItem<I,​M,​C,​D> workItem,
                                       String ingestionRunId)
        Executes callback when failed ingestion is registered.
        Parameters:
        workItem - work item provided by the dispatcher.
        ingestionRunId - ingestion run id created by the worker.

      • postIngestion

        default void postIngestion​(SimpleIngestionWorkItem<I,​M,​C,​D> workItem,
                           String ingestionRunId)
        Does steps after successful ingestion.
        Parameters:
        workItem - work item provided by the dispatcher.
        ingestionRunId - ingestion run id created by the worker.