Set up the Openflow Connector for Excel¶

Prerequisites¶

1. Ensure that you have reviewed About Openflow Connector for Excel.

2. Ensure that you have set up Openflow.

Get the credentials¶

This section describes the steps to get your credentials to configure the connector. Depending on your data source,

• Get the AWS credentials

• Get the SharePoint credentials

Set up Snowflake account¶

As a Snowflake account administrator, perform the following tasks:

1. Create a new role or use an existing role.

2. Create a new Snowflake service user with the type as SERVICE.

3. Grant the Snowflake service user the role you created in the previous steps.

4. Configure with key-pair auth for the Snowflake SERVICE user from step 2.

5. Strongly recommended Configure a secrets manager supported by Openflow, for example, AWS, Azure, and Hashicorp, and store the public and private keys in the secret store.

   Note

   If you do not want to use a secrets manager, you are responsible for safeguarding the public key and private key files used for key-pair authentication according to the security policies of your organization.

   1. Once the secrets manager is configured, determine how you will authenticate to it. On AWS, it’s recommended that you use the EC2 instance role associated with Openflow so that no other secrets have to be persisted.

   2. In Openflow, configure a Parameter Provider associated with this Secrets Manager, from the hamburger menu in the upper right. Navigate to Controller Settings » Parameter Provider and fetch your parameter values.

   3. At this point all credentials can be referenced with the associated parameter paths and no sensitive values need to be persisted within Openflow.

6. If any other Snowflake users require access to the raw ingested documents and tables ingested by the connector (for example, for custom processing in Snowflake), then grant those users the role created in step 1.

7. Create a database and schema in Snowflake for the connector to store ingested data. Grant required Database privileges to the role created in the first step. Substitute the role placeholder with the actual value and use the following SQL commands:

   CREATE DATABASE excel_destination_db;
   CREATE SCHEMA excel_destination_db.excel_destination_schema;
   GRANT USAGE ON DATABASE excel_destination_db TO ROLE <excel_connector_role>;
   GRANT USAGE ON SCHEMA excel_destination_db.excel_destination_schema TO ROLE <excel_connector_role>;
   GRANT CREATE TABLE ON SCHEMA excel_destination_db.excel_destination_schema TO ROLE <excel_connector_role>;
   

   Copy

8. Create a warehouse which will be used by the connector or use an existing one. Start with the smallest warehouse size, then experiment with size depending on the number of tables being replicated, and the amount of data transferred. Large table numbers typically scale better with multi-cluster warehouses, rather than larger warehouse sizes.

9. Ensure that the user with role used by the connector has the required privileges to use the warehouse. If that is not the case - grant the required privileges to the role:

   CREATE WAREHOUSE excel_connector_warehouse WITH WAREHOUSE_SIZE = 'X-Small';
   GRANT USAGE ON WAREHOUSE excel_connector_warehouse TO ROLE <excel_connector_role>;
   

   Copy

Set up the connector¶

As a data engineer, perform the following tasks to install and configure a connector:

Run the flow¶

1. Right-click on the plane and select Enable all Controller Services.

2. Right-click on the imported process group and select Start. The connector starts the data ingestion.

Generated table names¶

The connector creates destination tables named using the following template: {PREFIX}{FILENAME}_{RANGE}. The names are always double-quoted identifiers.

• {PREFIX} is replaced with the value of the Destination Table Prefix parameter, e.g. prfx_.

• {FILENAME} is replaced by the full path of ingested file, e.g. file1.xlsx or folder/file2.xlsx.

• {RANGE} is replaced by:

  • Name of the ingested sheet if the value of Ranges parameter is empty.

  • Name of the ingested sheet with ingested range as specified in the Ranges parameter, e.g. Sheet1!A1:B2.

Example of generated table names:

• "file1.xlsx_Sheet1"

• "prfx_folder/file2.xlsx_Sheet1!A1:B2"

(Optional) Reconfigure the currently running connector¶

You can reconfigure the connector parameters after the connector has already started ingesting data. If you need to change the ingested files or ranges, perform the following steps to make sure that the data is sent to Snowflake properly:

1. Stop the connector: Ensure that all Openflow processors are stopped.

2. Access configuration settings: Navigate to the connector’s configuration settings within Openflow.

3. Modify parameters: Adjust the parameters as required.

4. Start the connector: Start the connector and also ensure that all controller services have started.