Package com.snowflake.connectors.taskreactor.worker.ingestion

Interface Ingestion<STATE>

  • Type Parameters:
    STATE - Type defining the state used in the ingestion.
    public interface Ingestion<STATE>
    Interface that should be implemented by users of task reactor and passed over to IngestionWorker instance.

    • Method Detail

      • initialState

        STATE initialState​(WorkItem workItem)
        Used to initialize or load the initial state for the ingestion. State returned from this method is used in the first call of performIngestion(WorkItem, Object) method.
        Parameters:
        workItem - work item provided by the dispatcher
        Returns:
        initial ingestion state

      • preIngestion

        void preIngestion​(WorkItem workItem)
        Logic that needs to be executed before ingestion begins.
        Parameters:
        workItem - work item provided by the dispatcher

      • performIngestion

        STATE performIngestion​(WorkItem workItem,
                       STATE state)
        This is where the actual data ingestion should happen.
        Parameters:
        state - State that was returned from the previous performIngestion call, or the initial state if it's the first call of this method.
        workItem - work item provided by the dispatcher
        Returns:
        new state that will be used as a state argument in the next call of this method

      • isIngestionCompleted

        boolean isIngestionCompleted​(WorkItem workItem,
                             STATE state)
        The worker calls performIngestion(WorkItem, Object) as long as the value returned from this method is true.

        Note that performIngestion(WorkItem, Object) is called at least once.

        Parameters:
        workItem - work item provided by the dispatcher
        state - current ingestion state
        Returns:
        whether the ingestion has been completed

      • postIngestion

        void postIngestion​(WorkItem workItem,
                   STATE state)
        Logic that needs to be executed after ingestion ends.

        Note: this is NOT called when the ingestion gets cancelled.

        Parameters:
        workItem - work item provided by the dispatcher
        state - current ingestion state