Resource definition and ingestion SQL reference¶
STATE.RESOURCE_
This table is used to persist the data about configured resources. The data consists mostly of semi-structured variants.
The definition can be found in the file ingestion/resource_ingestion_definition.sql.
The table contains the following columns:
| Column name | Description |
|---|---|
id | Id of Resource Ingestion Definition. |
name | Name of the Resource Ingestion Definition that can be shown on UI. |
enabled | Information whether the ingestion is enabled. |
parent_id | Id of parent’s Resource Ingestion Definition, it allows to create resource hierarchy which can be ingested |
resource_id | Set of properties that are needed to define a resource in a specific connector. They identify a resource in a source system. They are set by a user. |
resource_metadata | Set of additional properties that describe a resource. They can be fetched automatically or calculated by a connector. Optional. |
ingestion_configurations | Set of configuration properties that describe how the resource should be ingested from the source system. Structure of this field is described in the next table. |
updated_at | UTC timestamp representing recent update. |
The ingestion_configuration property should follow the below schema:
| Field name | Description |
|---|---|
id | Id of Ingestion Configuration. Unique for given Resource Ingestion Definition |
ingestion_strategy | Strategy of given ingestion. Values: snapshot, incremental |
custom_configuration | Set of connector-specific ingestion properties |
schedule_type | Type of schedule. Values: interval, cron |
schedule_definition | String defining a schedule. e.g. 30m, 4h, 1d for interval. Cron expression in case of cron. |
destination | Set of properties that describe where ingested data for a given resource should be stored. |
Related Java objects¶
To interact with the RESOURCE_INGESTION_DEFINITION table the following Java objects are useful:
- ResourceIngestionDefinitionRepository
- ResourceIngestionDefinition
- IngestionConfiguration
- IngestionStrategy
- ScheduleType
PUBLIC.INGESTION_
File: ingestion/ingestion_definitions_view.sql
This view available to ADMIN and VIEWER users returns the data from the STATE.RESOURCE_INGESTION_DEFINITION table. The returned data is simplified and contains only some of the columns:
- id
- resource_id
- name
- enabled
STATE.INGESTION_
File: ingestion/ingestion_run.sql
This table is used to persist the data about process. It is not available to any role apart from the connector itself. It contains the following columns:
Column Type idSTRINGresource_ingestion_definition_idSTRINGingestion_configuration_idSTRINGtypeSTRINGstatusSTRINGcreated_atTIMESTAMP_NTZfinished_atTIMESTAMP_NTZupdated_atTIMESTAMP_NTZ
Related Java objects¶
The following Java classes are related to this table:
STATE.INGESTION_
File: ingestion/ingestion_run.sql
A table used to store log data about past and current ingestion triggered by the scheduler. It is not available to any role apart from the connector itself.
It contains the following columns:
Column Type idSTRINGresource_ingestion_definition_idSTRINGingestion_configuration_idSTRINGprocess_idSTRINGstarted_atTIMESTAMP_NTZcompleted_atTIMESTAMP_NTZstatusSTRINGingested_rowsNUMBERmetadataVARIANT
Related Java objects¶
The following Java classes are related to this table: