Set up the Openflow Connector for Microsoft Dataverse¶
Note
The connector is subject to the Connector Terms.
This topic describes the steps to set up the Openflow Connector for Microsoft Dataverse.
Prerequisites¶
Ensure that you have reviewed Openflow Connector for Microsoft Dataverse.
Ensure that you have set up Openflow.
Get the credentials¶
As a Microsoft Dataverse administrator, perform the following steps:
Ensure you have a Dataverse Environment to work with, and you have access to that environment through https://admin.powerplatform.microsoft.com/.
Ensure that you have an application registered in portal.azure.com. This application must have access to the tenant we have our Dataverse Environment available.
Generate and store ClientID and Client Secret within that application.
Go to Power Apps Admin Center and configure your Dataverse Environment to be accessed via applications registered before. To do that, go to Settings » Users & permissions » application users. Previously created applications must be added and granted with privileges necessary to read data from Microsoft Dataverse.
Copy and save the Environment URL of the selected Dataverse Environment from https://admin.powerplatform.microsoft.com/.
Set up Snowflake account¶
As a Snowflake account administrator, perform the following tasks:
- Create a Snowflake user with the type as SERVICE.
Create a database and schema to store the replicated data, and set up privileges for the service user to create tables in destination schema by granting the USAGE and CREATE TABLE privileges.
CREATE DATABASE <destination_database>; CREATE SCHEMA <destination_database>.<destination_schema>; CREATE USER <openflow_user> TYPE=SERVICE COMMENT='Service user for automated access of Openflow'; CREATE ROLE <openflow_role>; GRANT ROLE <openflow_role> TO USER <openflow_user>; GRANT USAGE ON DATABASE <destination_database> TO ROLE <openflow_role>; GRANT USAGE ON SCHEMA <destination_database>.<destination_schema> TO ROLE <openflow_role>; GRANT CREATE TABLE ON SCHEMA <destination_database>.<destination_schema> TO ROLE <openflow_role>; CREATE WAREHOUSE <openflow_warehouse> WITH WAREHOUSE_SIZE = 'SMALL' AUTO_SUSPEND = 300 AUTO_RESUME = TRUE; GRANT USAGE, OPERATE ON WAREHOUSE <openflow_warehouse> TO ROLE <openflow_role>;
Create a pair of secure keys (public and private). Store the private key for the user in a file to supply to the connector’s configuration. Assign the public key to the Snowflake service user:
ALTER USER <openflow_user> SET RSA_PUBLIC_KEY = 'thekey';
For more information, see pair of keys.
Snowflake strongly recommends this step. Configure a secrets manager supported by Openflow, for example, AWS, Azure, and Hashicorp, and store the public and private keys in the secret store.
Note
If for any reason, you do not wish to use a secrets manager, then you are responsible for safeguarding the public key and private key files used for key-pair authentication according to the security policies of your organization.
Once the secrets manager is configured, determine how you will authenticate to it. On AWS, it’s recommended that you the EC2 instance role associated with Openflow as this way no other secrets have to be persisted.
In Openflow, configure a Parameter Provider associated with this Secrets Manager, from the hamburger menu in the upper right. Navigate to Controller Settings » Parameter Provider and then fetch your parameter values.
At this point all credentials can be referenced with the associated parameter paths and no sensitive values need to be persisted within Openflow.
If any other Snowflake users require access to the raw ingested documents and tables ingested by the connector (for example, for custom processing in Snowflake), then grant those users the role created in step 1.
Designate a warehouse for the connector to use. Grant the USAGE privilege on the warehouse to the role created before. Start with the smallest warehouse size, then experiment with size depending on the number of tables being replicated, and the amount of data transferred. Large table numbers typically scale better with multi-cluster warehouses, rather than larger warehouse sizes.
Set up the connector¶
As a data engineer, perform the following tasks to install and configure the connector:
Install the connector¶
Navigate to the Openflow Overview page. In the Featured connectors section, select View more connectors.
On the Openflow connectors page, find the connector and select Add to runtime.
In the Select runtime dialog, select your runtime from the Available runtimes drop-down list.
Select Add.
Note
Before you install the connector, ensure that you have created a database and schema in Snowflake for the connector to store ingested data.
Authenticate to the deployment with your Snowflake account credentials and select Allow when prompted to allow the runtime application to access your Snowflake account. The connector installation process takes a few minutes to complete.
Authenticate to the runtime with your Snowflake account credentials.
The Openflow canvas appears with the connector process group added to it.
Configure the connector¶
Right-click on the imported process group and select Parameters.
Populate the required parameter values as described in Flow parameters.
Flow parameters¶
This section describes the flow parameters that you can configure based on the following parameter contexts:
Dataverse Source Parameters: Used to establish connection with Dataverse.
Dataverse Destination Parameters: Used to establish connection with Snowflake.
Dataverse Ingestion Parameters: Used to define the configuration of data downloaded from Dataverse.
Dataverse Source Parameters¶
Parameter |
Description |
---|---|
Source Dataverse Environment URL |
The main identifier of a source system to fetch data. The URL indicates a namespace where Dataverse tables exist. It also lets you create a scope parameter for OAuth. |
Source Microsoft Azure Tenant ID |
Microsoft tenant ID lets you create OAuth URLs. |
Source OAuth Client ID |
Microsoft Dataverse Web API uses OAuth authentication to secure access, and the connector uses the client credentials flow. To learn about client ID and how to find it in Microsoft Entra, see Application ID (client ID). |
Source OAuth Client Secret |
Microsoft Dataverse Web API uses OAuth authentication to secure access, and the connector uses the client credentials flow. To learn about client secret and how to find it in Microsoft Entra, see Certificates & secrets. |
Dataverse Destination Parameters¶
Parameter |
Description |
---|---|
Destination Database |
The database where data will be persisted. It must already exist in Snowflake |
Destination Schema |
The schema where data will be persisted. It must already exist in Snowflake |
Snowflake Account Identifier |
Snowflake account name formatted as [organization-name]-[account-name] where data will be persisted |
Snowflake Authentication Strategy |
Strategy of authentication to Snowflake. Possible values: SNOWFLAKE_SESSION_TOKEN - when we are running flow on SPCS, KEY_PAIR when we want to setup access using private key |
Snowflake Private Key |
The RSA private key used for authentication. The RSA key must be formatted according to PKCS8 standards and have standard PEM headers and footers. Note that either Snowflake Private Key File or Snowflake Private Key must be defined |
Snowflake Private Key File |
The file that contains the RSA Private Key used for authentication to Snowflake, formatted according to PKCS8 standards and having standard PEM headers and footers. The header line starts with |
Snowflake Private Key Password |
The password associated with the Snowflake Private Key File |
Snowflake Role |
Snowflake Role used during query execution |
Snowflake Username |
User name used to connect to Snowflake instance |
Snowflake Warehouse |
Snowflake warehouse used to run queries |
Dataverse Ingestion Parameters¶
Parameter |
Description |
---|---|
Scheduling Interval |
The processor that is fetching a list of tables to be ingested must be triggered according to a schedule. The interval is provided by the user. |
Source Tables Filter Strategy |
Strategy for filtering tables to be ingested. Can be one of REGEXP and LIST. |
Source Tables Filter Value |
Value of the tables filter. When Source Tables Filter Strategy is set to REGEXP - this is the regular expression to be matching selected tables. When LIST is provided, then it is a comma separated list of table names. |
Run the flow¶
Right-click on the plane and select Enable all Controller Services.
Right-click on the imported process group and select Start. The connector starts the data ingestion.